[Paraview] [EXTERNAL] Re: Need advice with parallel file format

[Mohammad Mirzadeh]

On Mon, May 5, 2014 at 11:53 AM, David E DeMarle
<dave.demarle at kitware.com>wrote:

>
> On Mon, May 5, 2014 at 2:41 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>>
>>> It is more likely the case that a serial reader will be read by everyone
>>> and then ParaView will strip out everything but the local chunk on each
>>> node.
>>>
>>
>> but then isn't this still a limiting factor on the size of the file? I
>> assume you would still get into trouble reading a file that is too big.
>>
>>>
>>>
> I don't think the file size itself is much of a problem these days. Even
> in 32 bit days you could do it with the right flags. Burlen of course has
> valid points for extremely large files that you might run across in HPC
> contexts.
>
>
Well unless I still don't understand the loading process, I'm thinking that
if a process needs to load the whole file into memory and then decide which
part its going to keep, it is going to be limited by the file size. For
instance in my case the whole vis file is around 8.5 GB and each process
normally has access to 2 GB of data (16 process/node -- 32 GB/node).

 The better readers will have each node read only its own portion from the
>>> filesystem. Ideally each node's portion is located in its own file because
>>> having lots of nodes read from the same big file can be problematic. I
>>> don't think we have any parallel-hdf5 readers currently which efficiently
>>> read in parallel from a single hdf5 file.
>>>
>>> Hum... that's what I was afraid of. So I assume that currently every
>> processor reads the whole HDF5 file and deletes the unwanted parts?
>>
>
> Nope. In the case of XDMF every processor reads the whole XML file, and
> then reads only the parts of the hdf5 file (or better yet files) that they
> are responsible for.
>
>
Ok this sounds promising then. Just to be sure though, is the reading
process essentially serial for HDF5? i.e. no MPI-IO is used for reading the
file?

>
>>> David E DeMarle
>>> Kitware, Inc.
>>> R&D Engineer
>>> 21 Corporate Drive
>>> Clifton Park, NY 12065-8662
>>> Phone: 518-881-4909
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20140505/cb97b60d/attachment-0001.html>