[vtkusers] Problem in Streaming

[Bernard Chiu]

Hi David and all,

Thank you for your input. I have taken your advice to get rid of the
streamer and used writer2->SetNumberOfPieces().

However, I have another problem now. After processing the first block,
the pipeline always set the extent of the input to [0, -1, 0, -1, 0,
-1]. Ultimately, when it is run the second time,
ImageFFT1D::ThreadedExecute() fails in the line

inPtr = inData->GetScalarPointerForExtent(inExt);

because inData has an extent [0, -1, 0, -1, 0, -1].

Do you know why the extent of the input suddenly changes?

Thanks again. Attached is my code:

ImageFFT1D* ifft = ImageFFT1D::New();
ifft->SetNumberOfThreads(1);
ifft->SetDimensionToFFT(3);
ifft->SetDimensionality(3);
ifft->ForwardFFTOff();
ifft->SetInput(xmlOutput);

vtkXMLImageDataWriter* writer2 = vtkXMLImageDataWriter::New();
writer2->SetInputConnection(ifft->GetOutputPort());
writer2->SetFileName("C:\\ifft.vti");
writer2->GetExtentTranslator()->SetSplitModeToXSlab();
writer2->SetNumberOfPieces(8);
writer2->Write();

On Thu, Dec 3, 2009 at 7:46 AM, David E DeMarle
<dave.demarle at kitware.com> wrote:
> Hmmm, what happens if you take the memory limit streamer out of the
> pipeline and instead call SetNumberOfPieces() directly on the writer?
> My guess so far is that this writer is incompatible with the streamer
> because it has the capability to stream on its own and thus prevents
> the streamer from doing so.
>
> David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 28 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-371-3971 x109
>
>
>
> On Wed, Dec 2, 2009 at 7:03 PM, Bernard Chiu <bcychiu at alumni.uwo.ca> wrote:
>> Hi all,
>>
>> I was trying to read in a 1GB "vti" image, process it using FFT and
>> write it to memory. I used streaming because the memory in my computer
>> is not enough for me to do this in one block. In particular, I used
>> vtkMemoryLimitImageDataStreamer to break up the image and hoped to do
>> this block by block.
>>
>> However, vtkXMLImageDataReader still tries to allocate memory for the
>> whole image volume in each streaming iteration. Thus, there is still
>> not enough memory to finish the whole operation, and the following
>> error message comes up eventually:
>>
>> ERROR: In C:\VTK504\Common\vtkDataArrayTemplate.txx, line 105
>> vtkDoubleArray: Unable to allocate 93603300 elements of size 8 bytes.
>>
>> In particular, I found out in
>> vtkXMLStructuredDataReader::ReadXMLData(), the first line
>>
>> this->GetOutputAsDataSet(0)->GetUpdateExtent(this->UpdateExtent);
>>
>> always gets the whole extent of the image.
>>
>> Please let me know whether there is any way to solve this problem.
>>
>> Thanks,
>> Bernard
>>
>>
>> The following is my code:
>> (Note: ImageFFT1D is a filter derived from vtkImageFFT. It does fft in
>> only one dimension instead of three in vtkImageFFT)
>>
>>        vtkXMLImageDataReader* vtkxmlreader = vtkXMLImageDataReader::New();
>>        vtkxmlreader->SetFileName((LPCSTR) inputfile);
>>        vtkxmlreader->ReleaseDataFlagOn();
>>
>>        ImageFFT1D* ifft = ImageFFT1D::New();
>>        ifft->SetNumberOfThreads(1);
>>        ifft->SetDimensionToFFT(3);
>>        ifft->SetDimensionality(3);
>>        ifft->ForwardFFTOff();
>>        ifft->SetInputConnection(vtkxmlreader->GetOutputPort());
>>        ifft->ReleaseDataFlagOn();
>>
>>        vtkMemoryLimitImageDataStreamer* streamer =
>> vtkMemoryLimitImageDataStreamer::New();
>>        streamer->SetInputConnection(ifft->GetOutputPort());
>>        streamer->SetMemoryLimit(50000);
>>        switch (XYZ)
>>        {
>>        case 0: //X -> Avoid cutting in X dim
>>                streamer->GetExtentTranslator()->SetSplitModeToYSlab();
>>                break;
>>        case 1: case 2://Y -> Avoid cutting in Y dim
>>                streamer->GetExtentTranslator()->SetSplitModeToXSlab();
>>                break;
>>        }
>>        streamer->UpdateWholeExtent();
>>
>>        vtkXMLImageDataWriter* writer2 = vtkXMLImageDataWriter::New();
>>        writer2->SetInputConnection(streamer->GetOutputPort());
>>        writer2->SetFileName("C:\\ifft.vti");
>>        writer2->Write();
-------------------------------------------
Bernard Chiu, Ph.D.
Senior Fellow
Vascular Imaging Laboratory
Department of Radiology
University of Washington
815 Mercer Street
Seattle, WA, USA, 98109
Telephone: (206) 543-9249
Website: bernard-chiu.com