[vtkusers] Fast File I/O for VTK

[Andras Lasso]

> it will be hard to get around streaming data (because of RAM limitations).

You may increase virtual memory size setting in your system and let the operating system worry about how to save/restore physical memory content to/from disk. You have less control over what is written to disk and when, but everything is done at a very low level and you don't need file parsing, so it may be simpler and faster than re-loading hdf files.

Andras

-----Original Message-----
From: vtkusers <vtkusers-bounces at vtk.org> On Behalf Of David E DeMarle
Sent: Thursday, May 3, 2018 8:57 AM
To: Jamil Goettlich <jamil.goettlich at gmail.com>
Cc: vtk <vtkusers at vtk.org>
Subject: Re: [vtkusers] Fast File I/O for VTK

Sorry I can only answer one of your questions at this time.

On Thu, May 3, 2018 at 6:10 AM, Jamil Goettlich <jamil.goettlich at gmail.com> wrote:
> Hello vtkUsers,
>
> I am planning on writing some code for visualizing time sequential 
> data and figured out that it will be hard to get around streaming data 
> (because of RAM limitations). I am currently using vtp files for 
> storing. They are loaded into multiblocks of vtkPolyData to be 
> visualized further on. Since the data can be pretty big, its taking a 
> while to import them, hence streaming data is alot about fast file I/O (please correct me if I'm wrong).
> I am currently reading vtp files (only Point Data) with about six 
> attributes (velocity, acceleration, pressure, ...) and with the 
> vtkDataArraySelectionClass I am able to read in only selected arrays, 
> which makes everything faster. I am happy with that, but as I do not 
> know anything about File I/O, I guess there is alot of optimization possible.
>
> My questions are the following and I'm happy about every answer on one 
> of
> them:
> - I realized that for reading XML Data, there is only one CPU core 
> being used. Is there a (c++) implementation to read parallelly?
> - When I stored all timestep files in vtkMultiBlockData (collection of 
> vtp) and imported those ones with XMLMultiblockDataReader, the import 
> took even longer than manually importing each timestep into polyData 
> and putting them into a multiblock afterwards. Is it possible to 
> parallelly read files of the multiblock, so reading each block is 
> distributed on different cores? (Sorry if the question seems dumb, I 
> have no idea of parallellism)
> - Has anyone got experience using the HDF5 format for storing and 
> reading polydata?And is it faster than reading XML data/ are multiple 
> cores being used? If so, how much effort does it take to write a 
> custom HDF5 reader/writer (as I did not find any vtk-c++ implementation on the web)?

You could look at the source code of IO/Truchas. That reader took me perhaps a week of effort to get to where it is now.

It makes unstructured grids of volumetric cells types. For somple points your code will be less complex.

> - Or is there any better/ faster/ more flexible/ parallel file format 
> than HDF5/vtp for this purpose?
> - Is there some good (vtk related) literature concerning that topic?
>
> I am sorry for asking many general questions at once but I thought it 
> would help many users that struggle with the same problems to get an 
> overview. I would be happy for any answers/experiences.
>
> Thanks alot
>
> Jamil
>
>
>
> _______________________________________________
> Powered by 
> https://na01.safelinks.protection.outlook.com/?url=www.kitware.com&dat
> a=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd6
> 1ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=ZlZe
> F0bsckE9EbKlv9jspraeoJNwfP0OzeV2MTyDeyQ%3D&reserved=0
>
> Visit other Kitware open-source projects at
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ki
> tware.com%2Fopensource%2Fopensource.html&data=02%7C01%7Classo%40queens
> u.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b
> 925c%7C1%7C0%7C636609490646480690&sdata=Yc6YnPgFfYMzrRIIX8FdjWNQe40UbF
> ojCPyQ7CDzckY%3D&reserved=0
>
> Please keep messages on-topic and check the VTK FAQ at:
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.vt
> k.org%2FWiki%2FVTK_FAQ&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d
> 640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636
> 609490646480690&sdata=w8ravzj8oob5nwoV6WNIf3ITYJdJAkDmfINUy3wzR54%3D&r
> eserved=0
>
> Search the list archives at: 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmarkma
> il.org%2Fsearch%2F%3Fq%3Dvtkusers&data=02%7C01%7Classo%40queensu.ca%7C
> 5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C
> 1%7C0%7C636609490646480690&sdata=hDaOmlmp%2FMsuC2E463IMIAfdHqRZRI4rtAX
> 1x%2BuEDdo%3D&reserved=0
>
> Follow this link to subscribe/unsubscribe:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.o
> rg%2Fmailman%2Flistinfo%2Fvtkusers&data=02%7C01%7Classo%40queensu.ca%7
> C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7
> C1%7C0%7C636609490646480690&sdata=tZytuI3tWINvEA9vLmYV8q52awE56KpUGpaw
> yKsn9JM%3D&reserved=0
>
_______________________________________________
Powered by https://na01.safelinks.protection.outlook.com/?url=www.kitware.com&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=ZlZeF0bsckE9EbKlv9jspraeoJNwfP0OzeV2MTyDeyQ%3D&reserved=0

Visit other Kitware open-source projects at https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kitware.com%2Fopensource%2Fopensource.html&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=Yc6YnPgFfYMzrRIIX8FdjWNQe40UbFojCPyQ7CDzckY%3D&reserved=0

Please keep messages on-topic and check the VTK FAQ at: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.vtk.org%2FWiki%2FVTK_FAQ&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=w8ravzj8oob5nwoV6WNIf3ITYJdJAkDmfINUy3wzR54%3D&reserved=0

Search the list archives at: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmarkmail.org%2Fsearch%2F%3Fq%3Dvtkusers&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=hDaOmlmp%2FMsuC2E463IMIAfdHqRZRI4rtAX1x%2BuEDdo%3D&reserved=0

Follow this link to subscribe/unsubscribe:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvtk.org%2Fmailman%2Flistinfo%2Fvtkusers&data=02%7C01%7Classo%40queensu.ca%7C5d51857638d640abfce308d5b0f5757a%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636609490646480690&sdata=tZytuI3tWINvEA9vLmYV8q52awE56KpUGpawyKsn9JM%3D&reserved=0