[vtkusers] Fast File I/O for VTK

Wed May 23 08:17:42 EDT 2018

Thanks alot, using Windows swap actually works pretty good.
I have another question concerning File I/O. I stumbled upon the file
Format ".pvtp"  looking at VTK File Formats (
https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf). I would
like to write parallel readable pvtp files by timestep. Let's say I have a
simulation with 12 timesteps and I want to distribute the reading to 4
cores, letting each processor read 3 steps.
Unfortunatly I did not find alot of information about it. There was only a
similar question (
https://www.vtk.org/pipermail/vtkusers/2009-June/052257.html) which has
seemingly not been answered and one about spatial partitioning (
https://www.vtk.org/pipermail/vtkusers/2017-January/097642.html). As I
would like to read/write parallelly over time, I am stuck at this point.
Plus, do I need to build VTK with MPI/ OpenMP packages for using those
pvtp-files?

Thanks alot in advance!

2018-05-03 15:04 GMT+02:00 Andras Lasso <lasso at queensu.ca>:

> > it will be hard to get around streaming data (because of RAM
> limitations).
>
> You may increase virtual memory size setting in your system and let the
> operating system worry about how to save/restore physical memory content
> to/from disk. You have less control over what is written to disk and when,
> but everything is done at a very low level and you don't need file parsing,
> so it may be simpler and faster than re-loading hdf files.
>
> Andras
>
> -----Original Message-----
> From: vtkusers <vtkusers-bounces at vtk.org> On Behalf Of David E DeMarle
> Sent: Thursday, May 3, 2018 8:57 AM
> To: Jamil Goettlich <jamil.goettlich at gmail.com>
> Cc: vtk <vtkusers at vtk.org>
> Subject: Re: [vtkusers] Fast File I/O for VTK
>
> Sorry I can only answer one of your questions at this time.
>
> On Thu, May 3, 2018 at 6:10 AM, Jamil Goettlich <jamil.goettlich at gmail.com>
> wrote:
> > Hello vtkUsers,
> >
> > I am planning on writing some code for visualizing time sequential
> > data and figured out that it will be hard to get around streaming data
> > (because of RAM limitations). I am currently using vtp files for
> > storing. They are loaded into multiblocks of vtkPolyData to be
> > visualized further on. Since the data can be pretty big, its taking a
> > while to import them, hence streaming data is alot about fast file I/O
> (please correct me if I'm wrong).
> > I am currently reading vtp files (only Point Data) with about six
> > attributes (velocity, acceleration, pressure, ...) and with the
> > vtkDataArraySelectionClass I am able to read in only selected arrays,
> > which makes everything faster. I am happy with that, but as I do not
> > know anything about File I/O, I guess there is alot of optimization
> possible.
> >
> > My questions are the following and I'm happy about every answer on one
> > of
> > them:
> > - I realized that for reading XML Data, there is only one CPU core
> > being used. Is there a (c++) implementation to read parallelly?
> > - When I stored all timestep files in vtkMultiBlockData (collection of
> > vtp) and imported those ones with XMLMultiblockDataReader, the import
> > took even longer than manually importing each timestep into polyData
> > and putting them into a multiblock afterwards. Is it possible to
> > parallelly read files of the multiblock, so reading each block is
> > distributed on different cores? (Sorry if the question seems dumb, I
> > have no idea of parallellism)
> > - Has anyone got experience using the HDF5 format for storing and
> > reading polydata?And is it faster than reading XML data/ are multiple
> > cores being used? If so, how much effort does it take to write a
> > custom HDF5 reader/writer (as I did not find any vtk-c++ implementation
> on the web)?
>
> You could look at the source code of IO/Truchas. That reader took me
> perhaps a week of effort to get to where it is now.
>
> It makes unstructured grids of volumetric cells types. For somple points
> your code will be less complex.
>
> > - Or is there any better/ faster/ more flexible/ parallel file format
> > than HDF5/vtp for this purpose?
> > - Is there some good (vtk related) literature concerning that topic?
> >
> > I am sorry for asking many general questions at once but I thought it
> > would help many users that struggle with the same problems to get an
> > overview. I would be happy for any answers/experiences.
> >
> > Thanks alot
> >
> > Jamil
> >
> >
> >
> > _______________________________________________
> > Powered by
> > https://na01.safelinks.protection.outlook.com/?url=www.kitware.com&dat
> > a=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd6
> > 1ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=ZlZe
> > F0bsckE9EbKlv9jspraeoJNwfP0OzeV2MTyDeyQ%3D&reserved=0
> >
> > Visit other Kitware open-source projects at
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ki
> > tware.com%2Fopensource%2Fopensource.html&data=02%7C01%7Classo%40queens
> > u.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> > 925c%7C1%7C0%7C636609490646480690&sdata=Yc6YnPgFfYMzrRIIX8FdjWNQe40UbF
> > ojCPyQ7CDzckY%3D&reserved=0
> >
> > Please keep messages on-topic and check the VTK FAQ at:
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.vt
> > k.org%2FWiki%2FVTK_FAQ&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d
> > 640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636
> > 609490646480690&sdata=w8ravzj8oob5nwoV6WNIf3ITYJdJAkDmfINUy3wzR54%3D&r
> > eserved=0
> >
> > Search the list archives at:
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmarkma
> > il.org%2Fsearch%2F%3Fq%3Dvtkusers&data=02%7C01%7Classo%40queensu.ca%7C
> > 5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C
> > 1%7C0%7C636609490646480690&sdata=hDaOmlmp%2FMsuC2E463IMIAfdHqRZRI4rtAX
> > 1x%2BuEDdo%3D&reserved=0
> >
> > Follow this link to subscribe/unsubscribe:
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.o
> > rg%2Fmailman%2Flistinfo%2Fvtkusers&data=02%7C01%7Classo%40queensu.ca%7
> > C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7
> > C1%7C0%7C636609490646480690&sdata=tZytuI3tWINvEA9vLmYV8q52awE56KpUGpaw
> > yKsn9JM%3D&reserved=0
> >
> _______________________________________________
> Powered by https://na01.safelinks.protection.outlook.com/?url=
> www.kitware.com&data=02%7C01%7Classo%40queensu.ca%
> 7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> 925c%7C1%7C0%7C636609490646480690&sdata=ZlZeF0bsckE9EbKlv9jspraeoJNwfP
> 0OzeV2MTyDeyQ%3D&reserved=0
>
> Visit other Kitware open-source projects at https://na01.safelinks.
> protection.outlook.com/?url=http%3A%2F%2Fwww.kitware.com%
> 2Fopensource%2Fopensource.html&data=02%7C01%7Classo%40queensu.ca%
> 7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> 925c%7C1%7C0%7C636609490646480690&sdata=Yc6YnPgFfYMzrRIIX8FdjWNQe40UbF
> ojCPyQ7CDzckY%3D&reserved=0
>
> Please keep messages on-topic and check the VTK FAQ at:
> https://na01.safelinks.protection.outlook.com/?url=
> http%3A%2F%2Fwww.vtk.org%2FWiki%2FVTK_FAQ&data=02%7C01%
> 7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%
> 7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=
> w8ravzj8oob5nwoV6WNIf3ITYJdJAkDmfINUy3wzR54%3D&reserved=0
>
> Search the list archives at: https://na01.safelinks.
> protection.outlook.com/?url=http%3A%2F%2Fmarkmail.org%
> 2Fsearch%2F%3Fq%3Dvtkusers&data=02%7C01%7Classo%40queensu.ca%
> 7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> 925c%7C1%7C0%7C636609490646480690&sdata=hDaOmlmp%
> 2FMsuC2E463IMIAfdHqRZRI4rtAX1x%2BuEDdo%3D&reserved=0
>
> Follow this link to subscribe/unsubscribe:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.org%
> 2Fmailman%2Flistinfo%2Fvtkusers&data=02%7C01%7Classo%40queensu.ca%
> 7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> 925c%7C1%7C0%7C636609490646480690&sdata=tZytuI3tWINvEA9vLmYV8q52awE56K
> pUGpawyKsn9JM%3D&reserved=0
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://vtk.org/pipermail/vtkusers/attachments/20180523/a5980ee6/attachment.html>