[Insight-users] Handling large datasets

[Luis Ibanez]

Hi John,

1) The option of having a reader with multiple
    outputs seems to be hard to implement.

    I would rather have N instantiations of the
    ImageFileReader. Anyways they may be
    conflicting regions of the image requested
    by every individual pipeline.

    You can actually create a "BatteryReader"
    class that dynamically creates ImageFileReaders
    on demand.  For example with a "GetNewReader()"
    method, and still feed all of them with the
    information of the same file name.


2) It is likely (quite likely) that a filter in
    the pipeline will say that for computing a
    requested region it needs the total largest
    possible region of its input image. As you
    said, this will defeat the purpose of the
    streaming.  Please look at the method

       GenerateInputRequestedRegion()

    of the process object

   http://www.itk.org/Insight/Doxygen/html/classitk_1_1ProcessObject.html

    that is one of the grandparents of all the
    filters in ITK.  For every filter that you
    plan to use in your pipeline, you must verify
    it this method has been overloaded.


     Along the same lines, you will find interesting
     the method

       GenerateOutputRequestedRegion()



3) Note that FFT is one of the classical examples
    of filters that cannot be streamed, because a
    pixel on the output image depends on *all* the
    pixels of the input image   :-/



Regards,


    Luis



-----------------------
John Biddiscombe wrote:

>>One of the difficulties at this point is
>>that many ImageIO classes do not support
>>reading regions from files. If you plan
>>to load section of 4D datasets, the first
>>thing to look at is a file format and
>>its associated ImageIO class that honors
>>the ReguestedRegion.  Most of them simply
>>load the entire image.
> 
> 
> OK. I already have a mechanism for loading data and then passing the pointer
> into vtk/itk, but I missed the obvious point that it will need to honour the
> requested region...The reader handles Dicom data, but is not implemented in
> itk.
> 
> I think I see what to do... Just leave all the data on disk, modify my IO
> class so that it behaves correctly with respect to the requested region, and
> make sure the code never asks for a chunk larger than memory will handle at
> any one instant.
> 
> Ideally, I'd want to add the ability for the IO reader to handle requests
> from multiple pipelines - so that it could serve N outputs, each one
> containing a different region. (eg N slice views, or N sub volumes each
> displayed independently). What I mean is it'd be nice to have a single IO
> reader, with N outputs, each output would have the LargestPossibleRegion set
> to <very large>, but different Bufferedregions. (Then make it thread safe so
> that multiple pipelines could each fetch chunks simultaneously....hmmm)
> 
> 
>>Note that the algorithmic nature of many
>>filters do not support streaming, so you
>>have to verify filter-by-filter which ones
>>will be able to process your data by pieces.
> 
> 
> Do you mean that if I pass a piece of data through a resampler or something
> similar, it'll ignore my requested region and ask for the whole lot (thereby
> trashing the program), or do you mean, that I can't do an operation on the
> whole lot and hope that internally the pipelines will magically fix things
> for me by streaming chunks through the filter and compositing the result. I
> certainly wouldn't want to do that, but I might want to extract a sub volume
> or slice and then FFT it and not expect the pipeline to demand the
> LargestPossibleRegion.
> 
> I welcome any thoughts you may have.
> 
> thanks
> 
> JB
> 
> 
> 
> _______________________________________________
> Insight-users mailing list
> Insight-users at itk.org
> http://www.itk.org/mailman/listinfo/insight-users
>