[Paraview] Best file fomat for big data

[Mark Olesen]

A few months back there was a suggestion on the mailing list that you 
can apparently write a single vtk geometry and then use something like 
"map data" to load the new, updated fields (perhaps Ken posted it? - 
don't know any other details).

I agree that these vtk format limitations can be quite frustrating and 
cause quite a bit of data bloat on the disk. For these situations, I 
would normally resort to using EnSight format. This gives you full 
separation of geometry and data, with moving or non-moving geometries 
etc. EnSight does support a server-of-servers data format, which makes 
it possible to a sub-case for each process and then load them together, 
but I've never used it. Instead we have the master process write a 
single file and stream the geometry and fields information across.
The additional overhead of moving data across the network is offset by 
the fact that the filer only has a single process writing to it.

If it helps, you can take a look at the foamToEnsight utility for a 
quick overview: https://develop.openfoam.com
The harder bits are located under src/conversion/ensight/mesh/, 
src/conversion/ensight/output/, src/conversion/ensight/part/

If you find another possibility, please let us know since it will be of 
interest for other people too.

Cheers,
/mark

-- 
Dr Mark OLESEN
Principal Engineer, ESI-OpenCFD
ESI GmbH | Einsteinring 24 | 85609 Munich | GERMANY
Mob. +49 171 9710 149
www.openfoam.com | www.esi-group.com | mark.olesen at esi-group.com


On 11/30/17 11:13, as92 at yandex.ru wrote:
> Hi! I just finish my own CFD simulation on 256 processors. The program 
> made graphics output in 2000 time points. So large number of outputs I 
> need to prepare smooth animation. I use XML VTK ascii format. So I have 
> now 2000 pvtu files and 2000*256=512000 vtu files. I have problems with 
> such big number of files. Now I understand that may be it was not the 
> best idea to use VTK format.
> I have the one fixed unstructured grid with ~10^7 cells, about ten CFD 
> variables on the cells and several thousands output points. My program 
> is MPI parallel and output time points are irregular over time. So I'd 
> like to keep value of time in the files (as I know it is impossible in 
> the VTK format). And I'd like to use ParaView for the visualization.
> What is the better format for storing big amount of CFD data that I be 
> able to write in parallel and load in the ParaView?
> Thank in advance!