[vtkusers] Streaming pipeline with parts of big volume in multiple views

Wed Feb 10 13:25:58 EST 2016

Hi Aashish,

2016-02-10 18:12 GMT+01:00 Aashish Chaudhary <aashish.chaudhary at kitware.com>
:

>
>
> On Sun, Feb 7, 2016 at 5:41 AM, Elvis Stansvik <
> elvis.stansvik at orexplore.com> wrote:
>
>> Hi all,
>>
>> As I tried to explain in a previous mail
>>
>>    http://markmail.org/message/mfp2npib5nqeeuux
>>
>> I have a large amount of volume data, and I need to render only a small
>> part of it.
>>
>> I've been trying to find information on the streaming visualization
>> pipeline that VTK supports, but it's been a bit hard to find examples
>> similar what I want to do.
>>
>> My questions right now are:
>>
>> 1) Is there some file format for volume data which is supported by VTK
>> with a reader that is "intelligent" enough to only read the parts of the
>> file that is requested by downstream consumers into main memory? That is,
>> is there something I could use so that my use case would work "out of the
>> box". If there is, I'm prepared to make sure the output from our analysis
>> is in this format, as it would save me from writing a custom reader.
>>
>
> AMR data structure might help (
> http://www.vtk.org/doc/nightly/html/classvtkAMRVolumeMapper.html#details
> ).
>

Hm, the "detailed description" you linked is essentially empty. I'll do
some googling, as I don't even know what AMR is. Thanks for the pointer
though!

In general I've begun to realize that most class docs in VTK are either
very brief or essentially empty. It makes it extremely hard to work with
the library as a newcomer. There are a few blog posts and wiki pages
around, but it's a mix of outdated and up-to-date info, and the almost
total lack of class docs or high-level documents is quite discouraging :(

>
>> 2) If there isn't, does anyone know of an example reader somewhere that
>> has this intelligence? I'm prepared to write my own reader, but would
>> really like a good example of how such a reader can be written.
>>
>
> I believe some of the  VTK image filter do this kind of work but I have to
> look into it.
>

Yes, I started looking into the MINC reader which we're using in a
prototype, and it seems it tries to respect the extends requested by
downstream. I'm not 100% sure though.

>
>> 3) I assume that all parts of the pipeline I set up must be capable of
>> working with partial data (extents in my case) for this to work? That is, I
>> obviously can't have something in the pipeline which is a "bad citizen" and
>> spoils it, causing the pipeline to request all the data. Is there a list of
>> which classes that supports this? Or are all filters et.c. in VTK
>> well-written in the sense that they will only request the minimal amount of
>> data they need, based on downstream requirements?
>>
>
> I believe so.
>

Sorry for bunching several questions together like I did here. I assume
your answer was to the last question?

>
>> 4) If I set the volume of interest for a volume using a vtkExtractVOI,
>> will the information about which piece of the volume is required reach the
>> source (reader) at the start of the pipeline?
>>
>
>
>> 5) I was planning on having a pipeline that looks roughly like
>>
>>     Reader ----> Extract VOI ----> Do stuff ---> View1
>>          \
>>           +------> Extract VOI ----> Do other stuff ---> View2
>>
>> where each of the two views show a different part of the full volume. One
>> of the views is scrollable, and the idea is that when the user scrolls that
>> view, the VOI is updated and that branch of the pipeline is re-executed.
>> Only the required data should be loaded from disk into main memory when
>> this happens. Is this kind of setup possible? Or will I have to use two
>> completely separate pipelines?
>>
>
> It depends how you write the code (are you setting data to mapper or you
> are connecting it to a pipeline?).
>

Hm, I'm not sure I understand. The above was a conceptual sketch of a
pipeline. I assume I would have mappers somewhere at the end of the two
branches above, around "View1" and "View2". Each of these mappers would be
connected to a branch of the pipeline.

I don't think I would "set the data" in any way, but try to use the
streaming nature of the VTK pipeline.

Thanks a lot for taking the time to answer.

Elvis

>
>> 6) If it turns out there are no built-in file formats where the reader
>> supports loading parts of the data from the file on disk based on
>> downstream requirements, I'm leaning towards making our own HDF5-based
>> format + reader (I believe the full volume will be around 50-100 GB of
>> data), since I think (but I'm not sure) that it's possible to only read
>> parts of a 3D array from a HDF5 file using h5py. Does anyone else have a
>> good suggestion for a file format, or have you been in a similar situation?
>>
>> Very grateful for any advice.
>>
>> Best regards,
>> Elvis
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the VTK FAQ at:
>> http://www.vtk.org/Wiki/VTK_FAQ
>>
>> Search the list archives at: http://markmail.org/search/?q=vtkusers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtkusers
>>
>>
>
>
> --
>
>
>
> *| Aashish Chaudhary | Technical Leader         | Kitware Inc.            *
> *| http://www.kitware.com/company/team/chaudhary.html
> <http://www.kitware.com/company/team/chaudhary.html>*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtkusers/attachments/20160210/8eb7a4bd/attachment.html>