[Paraview] advice on saving outputs

Sun May 20 16:10:23 EDT 2012

Thanks guys:

> Just this week, I wrote a Python program that uses the numpy.fromfile
function and ParaView's paraview.vtk.dataset_adapter.**numpyTovtkDataArray
and paraview.vtk.io.**vtkXMLPolyDataWriter functions to convert a large
binary file to .vtp format.  > It is reasonably fast.  Maybe you can use
something like that as a post-processing step.

> This separates the tasks of producing data and converting it to VTK
format.

This was actually my first thought but the reason I didn't go with it is
that it is dependent on the vtk API. I prefer to keep my code's dependency
minimal. Also it rewrites basically the same information redundantly in a
vtk file. Still I think that's the way to if one wants to stick with vtu
for instance if you want to benefit from .pvtu format.

> If you know the structure of the binary file, you can use xdmf format
file to read your binary data.

Actually I went ahead and learned it and now can write xdmf files to point
to my binary files. I like and seems to be quite flexible; for example I
can switch to HDF5 in the future if I want to and can keep my
xmdf writer objects (with minor modifications).

> if you grid does not change over time you can use XPath:
> "XPath, allows for elements in the XML document and the API to reference
specific elements in a document"
  > http://www.xdmf.org/index.php/XDMF_Model_and_Format

I actually did not understand when I was reading the tutorial. Can I use
this feature to include other xdmf files as well? I am thinking on how to
replicate .pvtu functionality with xdmf. Or maybe its possible to point to
xdmf files in .pvtu directly? Could you please comment on this?

Thanks,
Mohammad

On Sun, May 20, 2012 at 7:43 AM, Felipe Bordeu Weldt <
felipe.bordeu at ec-nantes.fr> wrote:

> Hi,
>
> If you know the structure of the binary file, you can use xdmf format file
> to read your binary data.
>
>
> for the binary part use something like this:
> <DataItem ItemType="Uniform"
> Format="Binary"
>  Dimensions="64 128 256"
> Seek="1234">
>  PressureFile.bin
> </DataItem>
>
> if you grid does not change over time you can use XPath:
> "XPath, allows for elements in the XML document and the API to reference
> specific elements in a document"
>       http://www.xdmf.org/index.php/XDMF_Model_and_Format
>
>
>
> I use xdmf all the time, I think is very flexible (ascii, binary, HDF5
> heavy data storage), time, grids, extra data using "Information" tag.  And
> all kinds of topologies and geometries.
>
>
> Le 20 mai 12 à 13:47, Hal Canary a écrit :
>
> On 05/19/2012 06:30 PM, Mohammad Mirzadeh wrote:
>
> Hi guys,
>
>
> I need a little bit of advice on how I'm outputting data. So far I have
>
> been using a simple legacy VTK format and dumping my grid and all
>
> sclaar/vectors into that single file. Basically I start with writing the
>
> grid and then append as many as variables that I like to the file. This
>
> has the problem that it was ascii and slow for large data sets,
>
> especially fro a parallel code where the solver was taking only 5 sec
>
> but the IO was around 20!!
>
>
> So I want to turn into binary and I have learned how to output binary
>
> vtu files. The issue is, since the cluster I'm working with has kill
>
> times (Max 24 hr) I also need to be able to save and resume my
>
> simulation. So, what I do right now is dump my variables in binary form
>
> into a single file that does not have any xml tag or anything -- just
>
> raw binary data. This is a preferred method since when I want to resume,
>
> I do not need to parse the xml code; I just read the binary file and
>
> that's it. Besides, I save on the grid information I'm writing to disk.
>
> For a 23M grid point, the grid information (which basically is a
>
> quadtree saved as a general unstructured grid with
>
> cell connectivity information etc) file is about 1GB whereas my actual
>
> data is only about 150MB. In most cases my grid does not change with
>
> time, so I really only need to save it once and that's enough.
>
>
> Now, the problem is, I want to be able to use or refer to these binary
>
> files using the xml tags. Here are the questions:
>
>
> 1) Is this supported in .vtu format? How can I refer to a binary file
>
> instead of dumping all the binary information in the file?
>
> 2) If this is not supported in .vtu, is it something xdmf can do? If so
>
> how? Can you please point me to a tutorial?
>
> 3) Is there any plan or strategy that would be better than what I have
>
> in mind right now? Please remember that retaining the raw binary format
>
> of the files is a must for me (well I do not see convincing reasons to
>
> give up on it, but if there is one, please let me know)
>
>
> Thanks
>
> Mohammad
>
>
>
>
> Just this week, I wrote a Python program that uses the numpy.fromfile
> function and ParaView's paraview.vtk.dataset_adapter.numpyTovtkDataArray
> and paraview.vtk.io.vtkXMLPolyDataWriter functions to convert a large
> binary file to .vtp format.  It is reasonably fast.  Maybe you can use
> something like that as a post-processing step.
>
> This separates the tasks of producing data and converting it to VTK format.
>
>
> --
> Hal Canary
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20120520/5d937e7f/attachment.htm>