[Paraview] Parallel file formats (again)...

Fri Jun 13 13:55:22 EDT 2008

It seems that XDMF is the unique way to go if we'd like to get parallel 
data into ParaView using HDF5. No problem since XDMF can be seen as a 
HDF5 extension. In fact, I thought about doing exactly what you cited -- 
use the HDF5 API and write simple XML/XDMF files using Fortran, no 
matter if my data is heavy or light.

The problem with XDMF is the lack of information. The first link pointed 
by Google is always down (www.arl.hpc.mil/ice/) and the "official" wiki 
site doesn't offer so much. I already got something on doing Fortran 
talk with XDMF (we can also write interfaces from the Fortran side to 
talk with C-like routine names) but, now, I'd like to go a bit further 
and use parallelism but there's no examples covering the subject using 
Fortran. My chance is debugging C++ examples and try to make some 
correlation.

Regarding row-major order and non 0 started arrays I can't say anything. 
I only say that, for Fortran programmers, it's getting a bit harder to 
work without having to deal with C++ and all that OOP stuffs. In this 
sense, I love Metis, so powerful, so easy, so fast, so simple and 
everything written in C ANSI. Just minor efforts to get it working with 
Fortran. As we say in Brazil, sometimes people like to kill cockroaches 
using bazookas instead of flip-flops... for writing files*my guess* is 
that straight C would do the job nicely

Dominik, the problem with Fortran is that everybody associates it with 
77 (just that old programming language). Maybe, they should change the 
name of the language from Fortran 2003 to F++ ;oP

Renato.

Dominik Szczerba wrote:
> And how would he handle hard-coded row-major ordering in XDMF?
> -- Dominik
>
> Chris Kees wrote:
>> You might want to reconsider XDMF or something based on it. I'm not 
>> sure that XDMF is significantly harder to implement in fortran than 
>> straight HDF5. It's just a matter  of doing some additional text i/o 
>> on a relatively simple XML file. XDMF splits the data (with some 
>> redundancy) into light/meta data stored as simple XML (ascii) file 
>> and an HDF5 archive of the "heavy" data.  You can read and write the 
>> XML file directly from fortran without using the XDMF library and 
>> then use the HDF5 fortran API directly to write the heavy data.   You 
>> have the option of storing the heavy data in the XML file as text 
>> when HDF5 isn't available (or when debugging/running on small 
>> data).   To me it looks like the posts you cite are pointing in this  
>> direction though they were unhappy with some aspects of XDMF.  It's 
>> not clear to me whether it's the XDMF xml format, the documentation 
>> of that format, or the C API that needs work in order to make it 
>> more  useful.
>> Also, it sounds like you've already decided against a mixed language 
>> approach, but the the book by H. P. Langtangen "Python Scripting for 
>> Computational Science" advocates a fortran/python pairing to deal 
>> with some of your  general concerns.
>> Chris
>> *  *
>> On Jun 13, 2008, at 7:58 AM, Renato N. Elias wrote:
>>
>>> Can anyone shed some light above how is the support status for 
>>> parallel file formats in ParaView?
>>>
>>> In my lab most of the students still work with Fortran. It seems 
>>> that "the universe nowadays only speaks C++ (and Python for 
>>> scripting)" which force us to do an extensive evaluation for a good 
>>> and well supported parallel file format to invest before struggling 
>>> with all that mixed languages interface/wrapping annoyances (not 
>>> everybody working with programs are programmers, there's still some 
>>> engineers like civil, mechanical, chemical, etc... doing science...).
>>>
>>> I could say that our my concerns about choosing a file format to 
>>> sticky with is:
>>>
>>> -- Easiness for installation and use (in this sense, Ensight is 
>>> wonderful since we don't need extra libraries. It's insane when we 
>>> need to compile 50 MB of libraries to link with a 2 MB program that 
>>> uses just one routine of such library);
>>> -- Easiness for interfacing (most of the libraries nowadays is 
>>> written in C++ for C++ programmers which discourage its use by C and 
>>> Fortran programs. Ok, we can always spend some time in interfacing 
>>> it, but, a library should offer more functionality and flexibility 
>>> than annoyances)
>>> -- Portability.
>>>
>>> Some time ago there was some interesting posts from Jean Favre and 
>>> Dominic about this, which give us some overview about the subject.
>>>
>>> http://www.paraview.org/pipermail/paraview/2008-May/008070.html
>>> http://www.paraview.org/pipermail/paraview/2008-May/008071.html
>>>
>>> My 2 cents for the discussion, *from a Fortran perspective*, is:
>>>
>>> 1). ENSIGHT:
>>> 1.1. Quite simple to implement and use (no need for extra libraries 
>>> and all that stuff. Just a few Fortran statements do the job);
>>> 1.2. Implicit support for transient data and parallelism;
>>> 1.3. Depending on the number of processes we might have a huge 
>>> number of small/medium files since each point and cell data variable 
>>> is stored in one file (sometimes it can be a serious problem);
>>> 1.4. Not compressed (too bad);
>>> 1.5. Not so well supported *as a parallel format* by ParaView yet. 
>>> After the change to deal (after PV 2.2.1) with multigroup datasets 
>>> some functionalities were lost until reimplementation.
>>> 1.6. Supported by ParaView, Visit and Ensight (of course)
>>>
>>> 2). XML/VTK:
>>> 2.1. Almost impossible for a Fortran user to implement, so, we're 
>>> forced to interface with VTK in order to write something;
>>> 2.2. Time series support has been introduced in some sense ;o)
>>> 2.3. It's a bit complicated to understand. Ok, it's XML and we 
>>> should use it (and believe on it ;o) ) through some library, so, 
>>> it's not supposed to "hand-implementation";
>>> 2.4. Encoding/compression is supported (which is really good)
>>> 2.5. It should be the most well parallel file format supported by 
>>> ParaView (after EXODUS, maybe)
>>> 2.6. Only supported by VTK based softwares (ParaView, Visit, MayaVi)
>>>
>>> 3). XDMF/HDF5:
>>> 3.1. Same as 2.1, 2.2 and 2.3
>>> 3.2. The website describing the library is a bit down lately...
>>> 3.3. HDF5 seems a very promising file format. It has some 
>>> development concern about its use by other scientific languages 
>>> besides being flexible, compressed, cross platform, etc... .
>>> 3.4. From my knowledge, XDMF is supported by Ensight, ParaView and 
>>> Visit also --> not sure about how good is that support.
>>>
>>> 4). EXODUS II:
>>> 4.1. Same as 2.1 --> I already tried more than once to find 
>>> something about Exodus format. There's a good documentation in 
>>> SANDIA/SEACAS page but the library is not open source (it's a 
>>> license based distribution) which turns it a bit complicated to adopt;
>>> 4.2. Nothing to say about timea nd compression support since I never 
>>> used it;
>>> 4.3. It must be well supported by PV since it's a Sandia's format;
>>>
>>> regards
>>>
>>> Renato.
>>>
>>>
>>>
>>> _______________________________________________
>>> ParaView mailing list
>>> ParaView at paraview.org
>>> http://www.paraview.org/mailman/listinfo/paraview
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> ParaView mailing list
>> ParaView at paraview.org
>> http://www.paraview.org/mailman/listinfo/paraview
>