[Insight-users] Handling large datasets

[John Biddiscombe]

> One of the difficulties at this point is
> that many ImageIO classes do not support
> reading regions from files. If you plan
> to load section of 4D datasets, the first
> thing to look at is a file format and
> its associated ImageIO class that honors
> the ReguestedRegion.  Most of them simply
> load the entire image.

OK. I already have a mechanism for loading data and then passing the pointer
into vtk/itk, but I missed the obvious point that it will need to honour the
requested region...The reader handles Dicom data, but is not implemented in
itk.

I think I see what to do... Just leave all the data on disk, modify my IO
class so that it behaves correctly with respect to the requested region, and
make sure the code never asks for a chunk larger than memory will handle at
any one instant.

Ideally, I'd want to add the ability for the IO reader to handle requests
from multiple pipelines - so that it could serve N outputs, each one
containing a different region. (eg N slice views, or N sub volumes each
displayed independently). What I mean is it'd be nice to have a single IO
reader, with N outputs, each output would have the LargestPossibleRegion set
to <very large>, but different Bufferedregions. (Then make it thread safe so
that multiple pipelines could each fetch chunks simultaneously....hmmm)

> Note that the algorithmic nature of many
> filters do not support streaming, so you
> have to verify filter-by-filter which ones
> will be able to process your data by pieces.

Do you mean that if I pass a piece of data through a resampler or something
similar, it'll ignore my requested region and ask for the whole lot (thereby
trashing the program), or do you mean, that I can't do an operation on the
whole lot and hope that internally the pipelines will magically fix things
for me by streaming chunks through the filter and compositing the result. I
certainly wouldn't want to do that, but I might want to extract a sub volume
or slice and then FFT it and not expect the pipeline to demand the
LargestPossibleRegion.

I welcome any thoughts you may have.

thanks

JB