[Paraview] compiling ParaView v4.2.0-RC1-75-gf64ee7a with -DModule_vtkIOXdmf3:BOOL=ON (UNCLASSIFIED)

Thu Oct 2 15:27:49 EDT 2014

Classification: UNCLASSIFIED
Caveats: NONE

Hi Dave,

I talked to Andrew (ARL developer working on xdmf) about the file size
issue.

I am using the ParaView-v4.2.0-source.tar.gz downloaded from ParaView
download page. Can you point me to the location of your file/code for the
reader/writer you put in?

After looking at the data file generated, he mentioned by setting
setReleaseData to true on the writer could fix the issue.

writer->getHeavyDataWriter()->setReleaseData(true);

Please see below on his comment on the matter. I have also included Andrew
in this email. If you can point me to your reader/writer code file, maybe I
can see if I can add that option to change the writer default settings.

Thanks
-simon

Classification: UNCLASSIFIED
Caveats: NONE

Here's a detailed look at the problem.

The generated Xdmf file has 31 timesteps listed in this format:

<Grid Name="Grid">
  <Time Value="0"/>
  <Geometry Type="XYZ">
    <DataItem DataType="Float" Dimensions="82724 3" Format="HDF"
Precision="4">mvf1.h5:Data7688</DataItem>
  </Geometry>
  <Topology Dimensions="49092" Type="Mixed">
    <DataItem DataType="Int" Dimensions="256032" Format="HDF"
Precision="8">mvf1.h5:Data7689</DataItem>
  </Topology>
  <Attribute Center="Node" Name="Part Index" Type="None">
    <DataItem DataType="Int" Dimensions="82724" Format="HDF"
Precision="4">mvf1.h5:Data7690</DataItem>
  </Attribute>
  <Attribute Center="Node" Name="vtkOriginalPointIds" Type="None">
    <DataItem DataType="Int" Dimensions="82724" Format="HDF"
Precision="8">mvf1.h5:Data7691</DataItem>
  </Attribute>
  <Attribute Center="Cell" Name="Material volume fraction - 1" Type="None">
    <DataItem DataType="UChar" Dimensions="49092" Format="HDF"
Precision="1">mvf1.h5:Data7692</DataItem>
  </Attribute>
  <Attribute Center="Cell" Name="vtkOriginalCellIds" Type="None">
    <DataItem DataType="Int" Dimensions="49092" Format="HDF"
Precision="8">mvf1.h5:Data7693</DataItem>
  </Attribute>
  <Attribute Center="Cell" Name="cellNormals" Type="None">
    <DataItem DataType="Float" Dimensions="49092 3" Format="HDF"
Precision="4">mvf1.h5:Data7694</DataItem>
  </Attribute>
  <Attribute Center="Cell" Name="vtkCompositeIndex" Type="None">
    <DataItem DataType="UInt" Dimensions="49092" Format="HDF"
Precision="4">mvf1.h5:Data7695</DataItem>
  </Attribute>
</Grid>

What I noticed is that the hdf5 data sets written start on dataset number
7688. Which means that there are 7687 datasets in the hdf5
file prior to the creation of the xmf file. All of these unreferenced sets
are contributing to a large hdf5 file.

I suspect that this issue may be due to the way that the writer is parsing
out the xdmf data. I presume that it is writing all data
to hdf5 each time an array is encountered, and it does make sense to do this
to save on memory.

The problem is it seems that ALL arrays are getting written to hdf5 each
time. This is probably because the XdmfHDF5Writer is on its
default settings.

The following call should help.

writer->getHeavyDataWriter()->setReleaseData(true);

This sets the writer to release the data currently contained within the
array from memory. If the XdmfWriter sees an array that has
heavy data in it, it assumes that the data has changed since the last write
and creates a new hdf5 dataset. Setting this flag causes
the heavy data writer to clear the xdmf array of data so that it will not be
written to file on a new pass.

-Andrew

-----Original Message-----
From: ParaView [mailto:paraview-bounces at paraview.org] On Behalf Of Su, Simon
M CTR USARMY ARL (US)
Sent: Wednesday, September 17, 2014 1:07 PM
To: David E DeMarle
Cc: paraview at paraview.org
Subject: Re: [Paraview] compiling ParaView v4.2.0-RC1-75-gf64ee7a with
-DModule_vtkIOXdmf3:BOOL=ON

thanks Dave.... I will pass along the info and see what they want to do...
-simon

________________________________

From: David E DeMarle [dave.demarle at kitware.com]
Sent: Wednesday, September 17, 2014 12:24 PM
To: Su, Simon M CTR USARMY ARL (US)
Cc: Ben Boeckel; paraview at paraview.org
Subject: Re: [Paraview] compiling ParaView v4.2.0-RC1-75-gf64ee7a with
-DModule_vtkIOXdmf3:BOOL=ON

I briefly looked at that a couple of weeks ago. Sorry I didn't get back to
you before now. 

My findings were:
1) xdm3writer performs the same as xdmf2writer 
2) I didn't find a compression option.

So, in other words, I don't have a quick fix and someone needs to put some
development time in to improve it.
The good news is that there is lots of room for improvement and it shouldn't
be too hard to do better on the writer end. It is very likely that the ARL
guys will have ideas of on ways to compress within libxdmf.

David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909

On Wed, Sep 17, 2014 at 10:16 AM, Su, Simon M CTR USARMY ARL (US)
<simon.m.su.ctr at mail.mil> wrote:

	thanks Dave. We will remember that when we do the compile after 4.2
release...

	I tried xdmf3 writer using the File->Save Data option in PV
v4.2.0-RC1-75-gf64ee7a and the 82M pvd file still turn into a 4.4G h5 file.
Not sure if something that I am not doing right? Is there like a compression
flag that I can turn on? thanks...

	thanks
	-simon

________________________________

	From: David E DeMarle [dave.demarle at kitware.com]
	Sent: Wednesday, September 17, 2014 11:56 AM
	To: Ben Boeckel
	Cc: Su, Simon M CTR USARMY ARL (US); paraview at paraview.org
<mailto:paraview at paraview.org> 
	Subject: Re: [Paraview] compiling ParaView v4.2.0-RC1-75-gf64ee7a
with -DModule_vtkIOXdmf3:BOOL=ON

	I had similar config problems in the DoD HPC machines I tested on.
The root cause was that the system boost was CMakefied and find boost didn't
like it but insisted on using it.

	To get around it I downloaded my own boost (no need to compile it)
and made the my PV build use it by:
	In cmakecache.txt  
	set Boost_NO_SYSTEM_PATHS:BOOL=ON
	set Boost_NO_BOOST_CMAKE:BOOL=ON

	Then it will ask for an respect your choice of boost headers to use.

	David E DeMarle
	Kitware, Inc.
	R&D Engineer
	21 Corporate Drive
	Clifton Park, NY 12065-8662
	Phone: 518-881-4909

	On Tue, Sep 16, 2014 at 12:55 PM, Ben Boeckel
<ben.boeckel at kitware.com> wrote:

		On Tue, Sep 16, 2014 at 16:22:38 +0000, Su, Simon M CTR
USARMY ARL (US) wrote:
		> I would like to test out the xdmf3 writer in ParaView 4.2
RC to see if the file
		> size is still increase 10 fold when saving pvd file to xmf
format.
		>
		> However, when I configure ParaView v4.2.0-RC1-75-gf64ee7a
with
		> -DModule_vtkIOXdmf3:BOOL=ON
		>
		> cmake -DCMAKE_BUILD_TYPE=release
-DCMAKE_INSTALL_PREFIX=/home/simonsm/tools/
		> paraview/paraview-v4.2.0-RC1-75-gf64ee7a_mesa
-DPARAVIEW_ENABLE_PYTHON:BOOL=ON
		> -DPARAVIEW_USE_MPI:BOOL=ON -DPARAVIEW_ENABLE_WEB:BOOL=OFF
		> -DModule_vtkIOXdmf3:BOOL=ON
../ParaView-v4.2.0-RC1-75-gf64ee7a
		>
		> I got whole bunch of boost related cmake error with cmake
version 3.0.0 and
		> cmake version 2.8.12.2. Some of the error messages are
included at the end of
		> this email.
		>
		> I am able to compile RC1-22-g9ca4e44 version ok but
without Module_vtkIOXdmf3
		> set to ON. Is there anything on the boost side that I am
not setting it right?
		> I am compiling on RHEL 6.5 with
boost-devel-1.41.0-18.el6.x86_64 installed.

		Looks to be a CMake-ified boost. It worked here on Fedora
with 1.54.0
		(which I don't believe is built using CMake anymore). It
seems that it
		doesn't support multiple find_package calls. Try setting
		Boost_NO_BOOST_CMAKE to ON to force the FindBoost.cmake
logic to kick
		in.

		--Ben
		_______________________________________________
		Powered by www.kitware.com

		Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
<http://www.kitware.com/opensource/opensource.html> 

		Please keep messages on-topic and check the ParaView Wiki
at: http://paraview.org/Wiki/ParaView <http://paraview.org/Wiki/ParaView> 

		Follow this link to subscribe/unsubscribe:
		http://public.kitware.com/mailman/listinfo/paraview

Classification: UNCLASSIFIED
Caveats: NONE

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 9990 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/paraview/attachments/20141002/d0993b0d/attachment.bin>