[vtkusers] Problem in Streaming

[David E DeMarle]

Hmmm, what happens if you take the memory limit streamer out of the
pipeline and instead call SetNumberOfPieces() directly on the writer?
My guess so far is that this writer is incompatible with the streamer
because it has the capability to stream on its own and thus prevents
the streamer from doing so.

David E DeMarle
Kitware, Inc.
R&D Engineer
28 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-371-3971 x109



On Wed, Dec 2, 2009 at 7:03 PM, Bernard Chiu <bcychiu at alumni.uwo.ca> wrote:
> Hi all,
>
> I was trying to read in a 1GB "vti" image, process it using FFT and
> write it to memory. I used streaming because the memory in my computer
> is not enough for me to do this in one block. In particular, I used
> vtkMemoryLimitImageDataStreamer to break up the image and hoped to do
> this block by block.
>
> However, vtkXMLImageDataReader still tries to allocate memory for the
> whole image volume in each streaming iteration. Thus, there is still
> not enough memory to finish the whole operation, and the following
> error message comes up eventually:
>
> ERROR: In C:\VTK504\Common\vtkDataArrayTemplate.txx, line 105
> vtkDoubleArray: Unable to allocate 93603300 elements of size 8 bytes.
>
> In particular, I found out in
> vtkXMLStructuredDataReader::ReadXMLData(), the first line
>
> this->GetOutputAsDataSet(0)->GetUpdateExtent(this->UpdateExtent);
>
> always gets the whole extent of the image.
>
> Please let me know whether there is any way to solve this problem.
>
> Thanks,
> Bernard
>
>
> The following is my code:
> (Note: ImageFFT1D is a filter derived from vtkImageFFT. It does fft in
> only one dimension instead of three in vtkImageFFT)
>
>        vtkXMLImageDataReader* vtkxmlreader = vtkXMLImageDataReader::New();
>        vtkxmlreader->SetFileName((LPCSTR) inputfile);
>        vtkxmlreader->ReleaseDataFlagOn();
>
>        ImageFFT1D* ifft = ImageFFT1D::New();
>        ifft->SetNumberOfThreads(1);
>        ifft->SetDimensionToFFT(3);
>        ifft->SetDimensionality(3);
>        ifft->ForwardFFTOff();
>        ifft->SetInputConnection(vtkxmlreader->GetOutputPort());
>        ifft->ReleaseDataFlagOn();
>
>        vtkMemoryLimitImageDataStreamer* streamer =
> vtkMemoryLimitImageDataStreamer::New();
>        streamer->SetInputConnection(ifft->GetOutputPort());
>        streamer->SetMemoryLimit(50000);
>        switch (XYZ)
>        {
>        case 0: //X -> Avoid cutting in X dim
>                streamer->GetExtentTranslator()->SetSplitModeToYSlab();
>                break;
>        case 1: case 2://Y -> Avoid cutting in Y dim
>                streamer->GetExtentTranslator()->SetSplitModeToXSlab();
>                break;
>        }
>        streamer->UpdateWholeExtent();
>
>        vtkXMLImageDataWriter* writer2 = vtkXMLImageDataWriter::New();
>        writer2->SetInputConnection(streamer->GetOutputPort());
>        writer2->SetFileName("C:\\ifft.vti");
>        writer2->Write();
>
>
> --
> -------------------------------------------
> Bernard Chiu, Ph.D.
> Senior Fellow
> Vascular Imaging Laboratory
> Department of Radiology
> University of Washington
> 815 Mercer Street
> Seattle, WA, USA, 98109
> Telephone: (206) 543-9249
> Website: bernard-chiu.com
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtkusers
>