[vtkusers] Memory leak vtkEnsightGoldReader in Python
Bernhard Righolt
b.w.righolt at tudelft.nl
Wed Mar 5 04:00:22 EST 2014
Dear Bill,
With your C++ script, I am indeed facing the same issue (although the
python code runs significantly quicker for my case).
I shared a few files, and a script to create more data here:
https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz My experience is that
with dissimilar data, the memory leak seems to be larger, but with this
data (which just copies the time steps), it is still present,
Best regards,
Bernhard
On 4 March 2014 21:27, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Attached is my C++ program and CMakeLists.txt file. But I don't think
> this is a python issue...
>
> On Tue, Mar 4, 2014 at 3:10 PM, Bernhard Righolt <b.w.righolt at tudelft.nl>
> wrote:
> > Dear Bill,
> >
> > I am a C++ speaker, so I don't mind to test your code on my current large
> > dataset. Can you maybe share your code, such that I can test it? I've
> never
> > used VTK within C++, if it solves the issues I am experiencing with
> Python,
> > I am very willing to switch. I hope it is not too complicated to link the
> > libraries correctly.
> >
> > Bernhard
> >
> >
> >
> > On 4 March 2014 21:05, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> >>
> >> Bernhard,
> >>
> >> Thanks for your patience. Since I'm not a python person, I converted
> >> your program to C++. I ran valgrind and found no memory leaks.
> >>
> >> Perhaps the small dataset you provided does not exercise the code that
> >> is leaking.
> >>
> >> Can you possible place a dataset that you know has leaksat some
> accessible
> >> site?
> >>
> >> I will say, that I don't think any of the current ensight tests access
> >> multiple time steps. Your data and test script will be a good starting
> >> point for such a test.
> >>
> >> Bill
> >>
> >> On Tue, Mar 4, 2014 at 12:08 AM, Bernhard Righolt
> >> <b.w.righolt at tudelft.nl> wrote:
> >> > Therr is a python script inside the archive which is the same script
> >> > that I
> >> > used on the large case.
> >> >
> >> > Bernhard
> >> >
> >> >
> >> > On Mar 4, 2014 12:22 AM, "Bill Lorensen" <bill.lorensen at gmail.com>
> >> > wrote:
> >> >>
> >> >> This is great. Can you send a complete python script that illustrates
> >> >> the
> >> >> problem.?
> >> >>
> >> >> On Mar 3, 2014 2:47 PM, "Bernhard Righolt" <b.w.righolt at tudelft.nl>
> >> >> wrote:
> >> >> >
> >> >> > Hi Bill,
> >> >> >
> >> >> > I added a very small sample data set. I am not sure if the memory
> >> >> > problem I am encountering with my big data-set is measurable for
> this
> >> >> > one.
> >> >> >
> >> >> > Anyhow, I would like to repeat that I am using Python 2.6 and VTK
> >> >> > 5.10,
> >> >> >
> >> >> > Best regards,
> >> >> > Bernhard
> >> >> >
> >> >> >
> >> >> > On 3 March 2014 19:25, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
> >> >> >>
> >> >> >> The ensight readers could use more testing. Can you provide a
> >> >> >> "small"
> >> >> >> dataset and program that cause memory leaks?
> >> >> >>
> >> >> >> Bill
> >> >> >>
> >> >> >>
> >> >> >> On Mon, Mar 3, 2014 at 10:38 AM, Bernhard Righolt
> >> >> >> <b.w.righolt at tudelft.nl> wrote:
> >> >> >> > Dear all,
> >> >> >> >
> >> >> >> > A typical case file, looks as follows (ASCII).
> >> >> >> >
> >> >> >> > $ head -n 20 case.case
> >> >> >> > FORMAT
> >> >> >> > type: ensight gold
> >> >> >> >
> >> >> >> > GEOMETRY
> >> >> >> > model: 1 sliceCentre.000.mesh
> >> >> >> >
> >> >> >> > VARIABLE
> >> >> >> > vector per node: 1 U sliceCentre.*****.U
> >> >> >> >
> >> >> >> > TIME
> >> >> >> > time set: 1
> >> >> >> > number of steps: 11550
> >> >> >> > filename start number: 1
> >> >> >> > filename increment: 1
> >> >> >> > time values:
> >> >> >> > 0.002
> >> >> >> > 0.004
> >> >> >> > 0.006
> >> >> >> > 0.008
> >> >> >> > 0.01
> >> >> >> > $
> >> >> >> >
> >> >> >> > The mesh (snippet only), is as follows:
> >> >> >> >
> >> >> >> > $ head -n 10 sliceCentre.000.mesh
> >> >> >> > Ensight Geometry File
> >> >> >> > =====================
> >> >> >> > node id assign
> >> >> >> > element id assign
> >> >> >> > part
> >> >> >> > 1
> >> >> >> > sliceCentre.000.mesh
> >> >> >> > coordinates
> >> >> >> > 299028
> >> >> >> > 0.00000e+00
> >> >> >> > $ sed -n -e 897091,897098p sliceCentre.000.mesh
> >> >> >> > 0.00000e+00
> >> >> >> > 0.00000e+00
> >> >> >> > 0.00000e+00
> >> >> >> > tria3
> >> >> >> > 595264
> >> >> >> > 162106 162714 162799
> >> >> >> > 162799 162226 162106
> >> >> >> > 162714 162106 162021
> >> >> >> >
> >> >> >> >
> >> >> >> > And the corresponding datafile
> >> >> >> > $ head -n 10 sliceCentre.00001.U
> >> >> >> > vector
> >> >> >> > part
> >> >> >> > 1
> >> >> >> > coordinates
> >> >> >> > 5.33425e-04
> >> >> >> > 2.69515e-03
> >> >> >> > -1.65164e-03
> >> >> >> > 5.33425e-04
> >> >> >> > 1.39069e-04
> >> >> >> > 2.69515e-03
> >> >> >> >
> >> >> >> >
> >> >> >> > However, I doubt if something is wrong with my datafiles itself,
> >> >> >> > because I
> >> >> >> > can easily read them in Paraview for example, the only
> difference
> >> >> >> > is
> >> >> >> > the
> >> >> >> > large amount of data-files, e.g (124GB in this case)
> >> >> >> >
> >> >> >> > Best regards,
> >> >> >> > Bernhard
> >> >> >> >
> >> >> >> >
> >> >> >> > On 27 February 2014 22:28, Berk Geveci <berk.geveci at kitware.com
> >
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Nothing strikes me as obviously wrong in your script. Can you
> >> >> >> >> send
> >> >> >> >> me
> >> >> >> >> your case file? I'll take a look.
> >> >> >> >>
> >> >> >> >> -berk
> >> >> >> >>
> >> >> >> >> On Thu, Feb 27, 2014 at 3:20 AM, Bernhard Righolt
> >> >> >> >> <b.w.righolt at tudelft.nl> wrote:
> >> >> >> >> > Dear all,
> >> >> >> >> >
> >> >> >> >> > Currently I am using a Python script to process information
> >> >> >> >> > stored
> >> >> >> >> > in
> >> >> >> >> > the
> >> >> >> >> > EnSight Gold format. My Python (2.6) scipt uses VTK (5.10.0)
> to
> >> >> >> >> > process
> >> >> >> >> > the
> >> >> >> >> > file, where I used the vtkEnSightGoldReader for reading the
> >> >> >> >> > data,
> >> >> >> >> > and
> >> >> >> >> > loop
> >> >> >> >> > over time steps.
> >> >> >> >> >
> >> >> >> >> > In principle this works fine for smaller datasets, however,
> for
> >> >> >> >> > large
> >> >> >> >> > datasets (i.e. 11k files of 12MB), I see the memory usage
> >> >> >> >> > (recorded via
> >> >> >> >> > top)
> >> >> >> >> > increasing with time while the process is running. This
> filling
> >> >> >> >> > of
> >> >> >> >> > the
> >> >> >> >> > memory goes slow, but in some cases problems are inevitable.
> >> >> >> >> > Apparently,
> >> >> >> >> > I
> >> >> >> >> > am doing some things wrong in my script with respect to
> memory
> >> >> >> >> > management.
> >> >> >> >> >
> >> >> >> >> > I am using the following script (stripped obviously, but
> still
> >> >> >> >> > showing
> >> >> >> >> > this
> >> >> >> >> > issue)
> >> >> >> >> >
> >> >> >> >> > import vtk
> >> >> >> >> >
> >> >> >> >> > reader = vtk.vtkEnSightGoldReader()
> >> >> >> >> >
> >> >> >> >> > reader.SetCaseFileName("case.case")
> >> >> >> >> > reader.Update()
> >> >> >> >> >
> >> >> >> >> > # Get time values
> >> >> >> >> > timeset=reader.GetTimeSets()
> >> >> >> >> > time=timeset.GetItem(0)
> >> >> >> >> > timesteps=time.GetSize()
> >> >> >> >> >
> >> >> >> >> > #reader.ReleaseDataFlagOn()
> >> >> >> >> >
> >> >> >> >> > for j in range(timesteps):
> >> >> >> >> > curTime=time.GetTuple(j)[0]
> >> >> >> >> > print curTime
> >> >> >> >> > reader.SetTimeValue(curTime)
> >> >> >> >> > reader.Update()
> >> >> >> >> >
> >> >> >> >> > #reader.RemoveAllInputs()
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > I tried a few things as you can see
> >> >> >> >> > - reader.ReleaseDataFlagOn(), without success
> >> >> >> >> > - reader.RemoveAllInputs(), did not get it to run
> >> >> >> >> > - running gc.collect at the end of the loop, following
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html
> >> >> >> >> > - Using DeepCopy as suggested here:
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html
> >> >> >> >> > However, I am already experiencing this without a call to
> >> >> >> >> > GetOutput
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > My question is, how can I properly unload/replace the data
> that
> >> >> >> >> > is
> >> >> >> >> > stored in
> >> >> >> >> > the memory, instead of using more memory continuously?
> >> >> >> >> >
> >> >> >> >> > Best regards,
> >> >> >> >> > Bernhard
> >> >> >> >> >
> >> >> >> >> > _______________________________________________
> >> >> >> >> > Powered by www.kitware.com
> >> >> >> >> >
> >> >> >> >> > Visit other Kitware open-source projects at
> >> >> >> >> > http://www.kitware.com/opensource/opensource.html
> >> >> >> >> >
> >> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
> >> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
> >> >> >> >> >
> >> >> >> >> > Follow this link to subscribe/unsubscribe:
> >> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
> >> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Powered by www.kitware.com
> >> >> >> >
> >> >> >> > Visit other Kitware open-source projects at
> >> >> >> > http://www.kitware.com/opensource/opensource.html
> >> >> >> >
> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
> >> >> >> >
> >> >> >> > Follow this link to subscribe/unsubscribe:
> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Unpaid intern in BillsBasement at noware dot com
> >> >> >
> >> >> >
> >>
> >>
> >>
> >> --
> >> Unpaid intern in BillsBasement at noware dot com
> >
> >
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140305/7edf83a2/attachment.html>
More information about the vtkusers
mailing list