[Paraview] Parallel file formats (again)...

Fri Jun 13 11:22:05 EDT 2008

And how would he handle hard-coded row-major ordering in XDMF?
-- Dominik

Chris Kees wrote:
> You might want to reconsider XDMF or something based on it. I'm not sure 
> that XDMF is significantly harder to implement in fortran than straight 
> HDF5. It's just a matter  of doing some additional text i/o on a 
> relatively simple XML file. XDMF splits the data (with some redundancy) 
> into light/meta data stored as simple XML (ascii) file and an HDF5 
> archive of the "heavy" data.  You can read and write the XML file 
> directly from fortran without using the XDMF library and then use the 
> HDF5 fortran API directly to write the heavy data.   You have the option 
> of storing the heavy data in the XML file as text when HDF5 isn't 
> available (or when debugging/running on small data).   To me it looks 
> like the posts you cite are pointing in this  direction though they were 
> unhappy with some aspects of XDMF.  It's not clear to me whether it's 
> the XDMF xml format, the documentation of that format, or the C API that 
> needs work in order to make it more  useful. 
> 
> Also, it sounds like you've already decided against a mixed language 
> approach, but the the book by H. P. Langtangen "Python Scripting for 
> Computational Science" advocates a fortran/python pairing to deal with 
> some of your  general concerns. 
> 
> Chris
> *  *
> On Jun 13, 2008, at 7:58 AM, Renato N. Elias wrote:
> 
>> Can anyone shed some light above how is the support status for 
>> parallel file formats in ParaView?
>>
>> In my lab most of the students still work with Fortran. It seems that 
>> "the universe nowadays only speaks C++ (and Python for scripting)" 
>> which force us to do an extensive evaluation for a good and well 
>> supported parallel file format to invest before struggling with all 
>> that mixed languages interface/wrapping annoyances (not everybody 
>> working with programs are programmers, there's still some engineers 
>> like civil, mechanical, chemical, etc... doing science...).
>>
>> I could say that our my concerns about choosing a file format to 
>> sticky with is:
>>
>> -- Easiness for installation and use (in this sense, Ensight is 
>> wonderful since we don't need extra libraries. It's insane when we 
>> need to compile 50 MB of libraries to link with a 2 MB program that 
>> uses just one routine of such library);
>> -- Easiness for interfacing (most of the libraries nowadays is written 
>> in C++ for C++ programmers which discourage its use by C and Fortran 
>> programs. Ok, we can always spend some time in interfacing it, but, a 
>> library should offer more functionality and flexibility than annoyances)
>> -- Portability.
>>
>> Some time ago there was some interesting posts from Jean Favre and 
>> Dominic about this, which give us some overview about the subject.
>>
>> http://www.paraview.org/pipermail/paraview/2008-May/008070.html
>> http://www.paraview.org/pipermail/paraview/2008-May/008071.html
>>
>> My 2 cents for the discussion, *from a Fortran perspective*, is:
>>
>> 1). ENSIGHT:
>> 1.1. Quite simple to implement and use (no need for extra libraries 
>> and all that stuff. Just a few Fortran statements do the job);
>> 1.2. Implicit support for transient data and parallelism;
>> 1.3. Depending on the number of processes we might have a huge number 
>> of small/medium files since each point and cell data variable is 
>> stored in one file (sometimes it can be a serious problem);
>> 1.4. Not compressed (too bad);
>> 1.5. Not so well supported *as a parallel format* by ParaView yet. 
>> After the change to deal (after PV 2.2.1) with multigroup datasets 
>> some functionalities were lost until reimplementation.
>> 1.6. Supported by ParaView, Visit and Ensight (of course)
>>
>> 2). XML/VTK:
>> 2.1. Almost impossible for a Fortran user to implement, so, we're 
>> forced to interface with VTK in order to write something;
>> 2.2. Time series support has been introduced in some sense ;o)
>> 2.3. It's a bit complicated to understand. Ok, it's XML and we should 
>> use it (and believe on it ;o) ) through some library, so, it's not 
>> supposed to "hand-implementation";
>> 2.4. Encoding/compression is supported (which is really good)
>> 2.5. It should be the most well parallel file format supported by 
>> ParaView (after EXODUS, maybe)
>> 2.6. Only supported by VTK based softwares (ParaView, Visit, MayaVi)
>>
>> 3). XDMF/HDF5:
>> 3.1. Same as 2.1, 2.2 and 2.3
>> 3.2. The website describing the library is a bit down lately...
>> 3.3. HDF5 seems a very promising file format. It has some development 
>> concern about its use by other scientific languages besides being 
>> flexible, compressed, cross platform, etc... .
>> 3.4. From my knowledge, XDMF is supported by Ensight, ParaView and 
>> Visit also --> not sure about how good is that support.
>>
>> 4). EXODUS II:
>> 4.1. Same as 2.1 --> I already tried more than once to find something 
>> about Exodus format. There's a good documentation in SANDIA/SEACAS 
>> page but the library is not open source (it's a license based 
>> distribution) which turns it a bit complicated to adopt;
>> 4.2. Nothing to say about timea nd compression support since I never 
>> used it;
>> 4.3. It must be well supported by PV since it's a Sandia's format;
>>
>> regards
>>
>> Renato.
>>
>>
>>
>> _______________________________________________
>> ParaView mailing list
>> ParaView at paraview.org
>> http://www.paraview.org/mailman/listinfo/paraview
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> ParaView mailing list
> ParaView at paraview.org
> http://www.paraview.org/mailman/listinfo/paraview

-- 
Dominik Szczerba, Ph.D.
Biomedical Simulation Group
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi