[vtkusers] Streaming pipeline with parts of big volume in multiple views

[Elvis Stansvik]

Hi all,

As I tried to explain in a previous mail

   http://markmail.org/message/mfp2npib5nqeeuux

I have a large amount of volume data, and I need to render only a small
part of it.

I've been trying to find information on the streaming visualization
pipeline that VTK supports, but it's been a bit hard to find examples
similar what I want to do.

My questions right now are:

1) Is there some file format for volume data which is supported by VTK with
a reader that is "intelligent" enough to only read the parts of the file
that is requested by downstream consumers into main memory? That is, is
there something I could use so that my use case would work "out of the
box". If there is, I'm prepared to make sure the output from our analysis
is in this format, as it would save me from writing a custom reader.

2) If there isn't, does anyone know of an example reader somewhere that has
this intelligence? I'm prepared to write my own reader, but would really
like a good example of how such a reader can be written.

3) I assume that all parts of the pipeline I set up must be capable of
working with partial data (extents in my case) for this to work? That is, I
obviously can't have something in the pipeline which is a "bad citizen" and
spoils it, causing the pipeline to request all the data. Is there a list of
which classes that supports this? Or are all filters et.c. in VTK
well-written in the sense that they will only request the minimal amount of
data they need, based on downstream requirements?

4) If I set the volume of interest for a volume using a vtkExtractVOI, will
the information about which piece of the volume is required reach the
source (reader) at the start of the pipeline?

5) I was planning on having a pipeline that looks roughly like

    Reader ----> Extract VOI ----> Do stuff ---> View1
         \
          +------> Extract VOI ----> Do other stuff ---> View2

where each of the two views show a different part of the full volume. One
of the views is scrollable, and the idea is that when the user scrolls that
view, the VOI is updated and that branch of the pipeline is re-executed.
Only the required data should be loaded from disk into main memory when
this happens. Is this kind of setup possible? Or will I have to use two
completely separate pipelines?

6) If it turns out there are no built-in file formats where the reader
supports loading parts of the data from the file on disk based on
downstream requirements, I'm leaning towards making our own HDF5-based
format + reader (I believe the full volume will be around 50-100 GB of
data), since I think (but I'm not sure) that it's possible to only read
parts of a 3D array from a HDF5 file using h5py. Does anyone else have a
good suggestion for a file format, or have you been in a similar situation?

Very grateful for any advice.

Best regards,
Elvis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtkusers/attachments/20160207/b627e140/attachment.html>