[Paraview] Opening PVD Files Referencing Thousands of Files

[John Biddiscombe]

Kent

>Does that fix make senses in general? What have been the experience of 
others with transient data stored in thousands of files?

I've had the same trouble with the XMLCollectionReader. When accessing
hundreds of time steps, it creates an internal reader for each step,
then on the very first pass, it tries to update information (memory may
cause a mistake here) on them all, and it suffers greatly.

I made a similar adjustment to the code to prevent it doing this, but I
suspect the class needs a fairly serious redesign. I wanted to interface
the XML pvd datasets to a Temporal Pipeline using the new
vtkTemporalDataSet and I got it working, but with much pain (and
hair/sleep loss etc)

JB

> If one has a parallel application using 512 processors to generate 100 
> time steps it seems like a good idea to use the pvd file format where 
> each processor can write one file per time step. This creates 51,201 
> files.
>
> When the pvd file is opened it seems that ParaView tries to read all 
> of the files and generate output (from vtkPVDReader) that includes the 
> contents of every file. As you might guess, this takes a very long 
> time. Is this an error or was the pvd format not designed for so many 
> files?
>
> I've altered our version to temporarily add the "restriction" that the 
> timestep must be 0 when the reader sees that there are no other 
> restrictions. As a result, when the file is first opened, it generates 
> output from only timestep 0. Later, as one animates through the time 
> steps, the animator adds a restriction for just the needed time.
>
> Does that fix make senses in general? What have been the experience of 
> others with transient data stored in thousands of files?
>
> TIA
> Kent
> Pittsburgh Supercomputing Center
>
>
> _______________________________________________
> ParaView mailing list
> ParaView at paraview.org
> http://www.paraview.org/mailman/listinfo/paraview


-- 
John Biddiscombe,                            email:biddisco @ cscs.ch
http://www.cscs.ch/about/BJohn.php
CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82