[vtkusers] standard data format

[John Shalf]

Randall Jones wrote:

> Hello,
>
>   What does the visualization community think is a good, portable
> general-purpose scientific data format?

>   I am working on standardizing our scientific data format.  We need to
> handle various types of "normal" visualization data (i.e. polygons,
> grids, meshes, images, and maybe abstract data structures like octrees?)
> We plan to do a great deal of data processing and reduction with vtk.
> But we would also like a format that is common/readable in many
> visualization tools so that our outputs could be visualized easily by
> other packages.
>
> The 2 formats we have narrowed down to are netCDF and HDF.

I'd actually recommend HDF5.  the vtkHDFReader was for HDF4 and at that, it was exclusively
for structured meshes with cartesian coordinates (very limiting).  HDF5 has a cleaner API and
much more powerful data abstrations than netCDF and HDF4 as well as pretty good parallel IO
capabilities and it works on a lot of wierd and esoteric systems (particularly the DOE ASCI
machines).  Mike Folk is crafting some data standards that are very similar to what you are
describing above.  He would really like feedback/help in developing this standard
(mfolk at ncsa.uiuc.edu).  I'm sure he'd love to talk with you about this.
    http://hdf.ncsa.uiuc.edu/

Another document you should take a look at is the DOE DMF standard.  This is much more
ambitious than what I think Mike is planning on putting together and is consequently that much
more difficult to implement.  However, the DMF's fiber-bundles based data model is very
sophisticated and likely will creep into any all-inclusive data format designs.  You may even
see an implementation soon.
  http://www.llnl.gov/tid/lof/documents/pdf/234737.pdf
  http://www.ca.sandia.gov/asci-sdm/
And its worth mentioning that this DMF data model is also being implemented on top of HDF5.
(HDF5... its where you want to be)

> vtkHDFReader
>
>   In August, 1999 there were some vtk classes announced (by John
> Shalf).  I'm looking for one of these, the vtkHDFReader class.  The
> links mentioned in the message are no longer valid.  Does anyone have
> any information on this vtk class?

I moved to Berkeley.  You can find them at
    http://dit.lbl.gov/VTK
or
    http://zeus.ncsa.uiuc.edu/~jshalf/

I haven't attempted to compile or test this in a while, so it may require some tweaking.
Sorry, my job tasks have taken me in a very different direction so some of these older
projects are not very well maintained...

-john