[vtkusers] Problem in Streaming: Lesson Learned in ReleaseData

[Bernard Chiu]

Hi all,

I have solved my problem described below. Since the mechanism of
releasing data object in streaming isn't documented anywhere to my
knowledge, I would like to share my solution.

Streaming processes image block by block. At each iteration of
processing, the executive has the option of deleting its input if the
ReleaseData of a data object is on. Since the input of the first
filter in a pipeline will be accessed again in another iteration, it
should not be deleted. In my problem below, after the first iteration
of ifft, the input data object is released and not available to
subsequent iterations.

To solve this problem, I explicitly demand that the input data object
of ifft is not deleted by adding the following two lines:

vtkInformation* inInfo = ifft->GetExecutive()->GetInputInformation(0,0);
inInfo->Set(vtkDemandDrivenPipeline::RELEASE_DATA(), 0);

On Thu, Dec 3, 2009 at 12:36 PM, Bernard Chiu <bcychiu at alumni.uwo.ca> wrote:
> Hi David and all,
>
> Thank you for your input. I have taken your advice to get rid of the
> streamer and used writer2->SetNumberOfPieces().
>
> However, I have another problem now. After processing the first block,
> the pipeline always set the extent of the input to [0, -1, 0, -1, 0,
> -1]. Ultimately, when it is run the second time,
> ImageFFT1D::ThreadedExecute() fails in the line
>
> inPtr = inData->GetScalarPointerForExtent(inExt);
>
> because inData has an extent [0, -1, 0, -1, 0, -1].
>
> Do you know why the extent of the input suddenly changes?
>
> Thanks again. Attached is my code:
>
> ImageFFT1D* ifft = ImageFFT1D::New();
> ifft->SetNumberOfThreads(1);
> ifft->SetDimensionToFFT(3);
> ifft->SetDimensionality(3);
> ifft->ForwardFFTOff();
> ifft->SetInput(xmlOutput);
>
> vtkXMLImageDataWriter* writer2 = vtkXMLImageDataWriter::New();
> writer2->SetInputConnection(ifft->GetOutputPort());
> writer2->SetFileName("C:\\ifft.vti");
> writer2->GetExtentTranslator()->SetSplitModeToXSlab();
> writer2->SetNumberOfPieces(8);
> writer2->Write();
>
> On Thu, Dec 3, 2009 at 7:46 AM, David E DeMarle
> <dave.demarle at kitware.com> wrote:
>> Hmmm, what happens if you take the memory limit streamer out of the
>> pipeline and instead call SetNumberOfPieces() directly on the writer?
>> My guess so far is that this writer is incompatible with the streamer
>> because it has the capability to stream on its own and thus prevents
>> the streamer from doing so.
>>
>> David E DeMarle
>> Kitware, Inc.
>> R&D Engineer
>> 28 Corporate Drive
>> Clifton Park, NY 12065-8662
>> Phone: 518-371-3971 x109
>>
>>
>>
>> On Wed, Dec 2, 2009 at 7:03 PM, Bernard Chiu <bcychiu at alumni.uwo.ca> wrote:
>>> Hi all,
>>>
>>> I was trying to read in a 1GB "vti" image, process it using FFT and
>>> write it to memory. I used streaming because the memory in my computer
>>> is not enough for me to do this in one block. In particular, I used
>>> vtkMemoryLimitImageDataStreamer to break up the image and hoped to do
>>> this block by block.
>>>
>>> However, vtkXMLImageDataReader still tries to allocate memory for the
>>> whole image volume in each streaming iteration. Thus, there is still
>>> not enough memory to finish the whole operation, and the following
>>> error message comes up eventually:
>>>
>>> ERROR: In C:\VTK504\Common\vtkDataArrayTemplate.txx, line 105
>>> vtkDoubleArray: Unable to allocate 93603300 elements of size 8 bytes.
>>>
>>> In particular, I found out in
>>> vtkXMLStructuredDataReader::ReadXMLData(), the first line
>>>
>>> this->GetOutputAsDataSet(0)->GetUpdateExtent(this->UpdateExtent);
>>>
>>> always gets the whole extent of the image.
>>>
>>> Please let me know whether there is any way to solve this problem.
>>>
>>> Thanks,
>>> Bernard
>>>
>>>
>>> The following is my code:
>>> (Note: ImageFFT1D is a filter derived from vtkImageFFT. It does fft in
>>> only one dimension instead of three in vtkImageFFT)
>>>
>>>        vtkXMLImageDataReader* vtkxmlreader = vtkXMLImageDataReader::New();
>>>        vtkxmlreader->SetFileName((LPCSTR) inputfile);
>>>        vtkxmlreader->ReleaseDataFlagOn();
>>>
>>>        ImageFFT1D* ifft = ImageFFT1D::New();
>>>        ifft->SetNumberOfThreads(1);
>>>        ifft->SetDimensionToFFT(3);
>>>        ifft->SetDimensionality(3);
>>>        ifft->ForwardFFTOff();
>>>        ifft->SetInputConnection(vtkxmlreader->GetOutputPort());
>>>        ifft->ReleaseDataFlagOn();
>>>
>>>        vtkMemoryLimitImageDataStreamer* streamer =
>>> vtkMemoryLimitImageDataStreamer::New();
>>>        streamer->SetInputConnection(ifft->GetOutputPort());
>>>        streamer->SetMemoryLimit(50000);
>>>        switch (XYZ)
>>>        {
>>>        case 0: //X -> Avoid cutting in X dim
>>>                streamer->GetExtentTranslator()->SetSplitModeToYSlab();
>>>                break;
>>>        case 1: case 2://Y -> Avoid cutting in Y dim
>>>                streamer->GetExtentTranslator()->SetSplitModeToXSlab();
>>>                break;
>>>        }
>>>        streamer->UpdateWholeExtent();
>>>
>>>        vtkXMLImageDataWriter* writer2 = vtkXMLImageDataWriter::New();
>>>        writer2->SetInputConnection(streamer->GetOutputPort());
>>>        writer2->SetFileName("C:\\ifft.vti");
>>>        writer2->Write();
> -------------------------------------------
> Bernard Chiu, Ph.D.
> Senior Fellow
> Vascular Imaging Laboratory
> Department of Radiology
> University of Washington
> 815 Mercer Street
> Seattle, WA, USA, 98109
> Telephone: (206) 543-9249
> Website: bernard-chiu.com
>



-- 
-------------------------------------------
Bernard Chiu, Ph.D.
Senior Fellow
Vascular Imaging Laboratory
Department of Radiology
University of Washington
815 Mercer Street
Seattle, WA, USA, 98109
Telephone: (206) 543-9249
Website: bernard-chiu.com