[vtkusers] Fast File I/O for VTK

[Jamil Goettlich]

Hello vtkUsers,

I am planning on writing some code for visualizing time sequential data and
figured out that it will be hard to get around streaming data (because of
RAM limitations). I am currently using vtp files for storing. They are
loaded into multiblocks of vtkPolyData to be visualized further on. Since
the data can be pretty big, its taking a while to import them, hence
streaming data is alot about fast file I/O (please correct me if I'm wrong).
I am currently reading vtp files (only Point Data) with about six
attributes (velocity, acceleration, pressure, ...) and with the
vtkDataArraySelectionClass I am able to read in only selected arrays, which
makes everything faster. I am happy with that, but as I do not know
anything about File I/O, I guess there is alot of optimization possible.

My questions are the following and I'm happy about every answer on one of
them:
- I realized that for reading XML Data, there is only one CPU core being
used. Is there a (c++) implementation to read parallelly?
- When I stored all timestep files in vtkMultiBlockData (collection of vtp)
and imported those ones with XMLMultiblockDataReader, the import took even
longer than manually importing each timestep into polyData and putting them
into a multiblock afterwards. Is it possible to parallelly read files of
the multiblock, so reading each block is distributed on different cores?
(Sorry if the question seems dumb, I have no idea of parallellism)
- Has anyone got experience using the HDF5 format for storing and reading
polydata?And is it faster than reading XML data/ are multiple cores being
used? If so, how much effort does it take to write a custom HDF5
reader/writer (as I did not find any vtk-c++ implementation on the web)?
- Or is there any better/ faster/ more flexible/ parallel file format than
HDF5/vtp for this purpose?
- Is there some good (vtk related) literature concerning that topic?

I am sorry for asking many general questions at once but I thought it would
help many users that struggle with the same problems to get an overview. I
would be happy for any answers/experiences.

Thanks alot

Jamil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://vtk.org/pipermail/vtkusers/attachments/20180503/adecb319/attachment.html>