[vtkusers] Fast File I/O for VTK

[David E DeMarle]

Sorry I can only answer one of your questions at this time.

On Thu, May 3, 2018 at 6:10 AM, Jamil Goettlich
<jamil.goettlich at gmail.com> wrote:
> Hello vtkUsers,
>
> I am planning on writing some code for visualizing time sequential data and
> figured out that it will be hard to get around streaming data (because of
> RAM limitations). I am currently using vtp files for storing. They are
> loaded into multiblocks of vtkPolyData to be visualized further on. Since
> the data can be pretty big, its taking a while to import them, hence
> streaming data is alot about fast file I/O (please correct me if I'm wrong).
> I am currently reading vtp files (only Point Data) with about six attributes
> (velocity, acceleration, pressure, ...) and with the
> vtkDataArraySelectionClass I am able to read in only selected arrays, which
> makes everything faster. I am happy with that, but as I do not know anything
> about File I/O, I guess there is alot of optimization possible.
>
> My questions are the following and I'm happy about every answer on one of
> them:
> - I realized that for reading XML Data, there is only one CPU core being
> used. Is there a (c++) implementation to read parallelly?
> - When I stored all timestep files in vtkMultiBlockData (collection of vtp)
> and imported those ones with XMLMultiblockDataReader, the import took even
> longer than manually importing each timestep into polyData and putting them
> into a multiblock afterwards. Is it possible to parallelly read files of the
> multiblock, so reading each block is distributed on different cores? (Sorry
> if the question seems dumb, I have no idea of parallellism)
> - Has anyone got experience using the HDF5 format for storing and reading
> polydata?And is it faster than reading XML data/ are multiple cores being
> used? If so, how much effort does it take to write a custom HDF5
> reader/writer (as I did not find any vtk-c++ implementation on the web)?

You could look at the source code of IO/Truchas. That reader took me
perhaps a week of effort to get to where it is now.

It makes unstructured grids of volumetric cells types. For somple
points your code will be less complex.

> - Or is there any better/ faster/ more flexible/ parallel file format than
> HDF5/vtp for this purpose?
> - Is there some good (vtk related) literature concerning that topic?
>
> I am sorry for asking many general questions at once but I thought it would
> help many users that struggle with the same problems to get an overview. I
> would be happy for any answers/experiences.
>
> Thanks alot
>
> Jamil
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the VTK FAQ at:
> http://www.vtk.org/Wiki/VTK_FAQ
>
> Search the list archives at: http://markmail.org/search/?q=vtkusers
>
> Follow this link to subscribe/unsubscribe:
> https://vtk.org/mailman/listinfo/vtkusers
>