[Paraview] Need advice with parallel file format

[Mohammad Mirzadeh]

They are represented as unstructured grid. As a sample run, a 100M grid
point on 256 proc produces almost 8.5G file. We intent to push the limits
close to 1B at most at this time with # processors up to a few thousands.
However, it would be good to have something that could scale to larger
problems as well


On Sat, May 3, 2014 at 1:28 AM, Stephen Wornom <stephen.wornom at inria.fr>wrote:

> Mohammad Mirzadeh wrote:
>
>> Hi I am at a critical point in deciding I/O format for my application. So
>> far my conclusion is to use parallel HDF5 for restart files as they are
>> quite flexible and portable across systems.
>>
>> When it comes to visualization, however, i'm not quite sure. Up until now
>> I've been using pvtu along with vtu files and although they generally work
>> fine, one easily gets in trouble when running big simulations on large
>> number of processors as the number of files can easily get out of control
>> and even simplest utility commands (e.g. ls) takes minutes to finish!
>>
>> After many thinking I've come to a point to decide between two strategies:
>>
>> 1) Go with a single parallel HDF5 file that includes data for all
>> time-steps. This makes it all nice and portable except there are two
>> issues. i) It looks like doing MPI-IO might not be as efficient as separate
>> POSIX IO, especially on large number of processors. ii) ParaView does not
>> seem to be able to read HDF5 files in parallel
>>
>> 2) Go with the same pvtu+vtu strategy except take precautions to avoid
>> file explosions. I can think of two strategies here: i) use nested folders
>> to separate vtu files from pvtu and also each time step ii) create an IO
>> group communicator with much less processors that do the actual IO.
>>
>> My questions are 1) Is the second approach necessarily more efficient
>> than MPI-IO used in HDF5? and 2) Is there any plan to support parallel IO
>> for HDF5 files in paraview?
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/
>> opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.paraview.org/mailman/listinfo/paraview
>>
> Are your meshes structured or unstructured? How many vertices in your
> meshes?
>
> Stephen
>
> --
> stephen.wornom at inria.fr
> 2004 route des lucioles - BP93
> Sophia Antipolis
> 06902 CEDEX
>
> Tel: 04 92 38 50 54
> Fax: 04 97 15 53 51
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20140505/845cbf76/attachment.html>