[vtkusers] Memory leak vtkEnsightGoldReader in Python

Bernhard Righolt b.w.righolt at tudelft.nl
Wed Mar 5 13:25:45 EST 2014 

Hi Berk,

I will try it with this patch on my dataset. I had a dataset with 11,000+
timesteps (and planning for a few more). I am not sure if it would grow
linearly or quadratically with the amount of time-steps.

I am not that experienced with VTK source cde, so before I do anything
silly, can you confirm the following:
- The patch is suitable for VTK 5.10 (or should I take a newer version?)
- I apply the patch to in my VTK source (IO/vtkEnSightReader.cxx)
- Recompile VTK

I hope I can get back to you with results of my tests as soon as possible.
Both of you already many thanks for helping me!

Best regards,
Bernhard



On 5 March 2014 18:33, Berk Geveci <berk.geveci at kitware.com> wrote:

> I went over the EnSight Gold reader carefully and there is only one
> small memory issue. Instead of a clearing a collection object every
> timestep, it keeps adding items to it. This accumulates to something
> small (7.5MB over 1000 timesteps in my example). It doesn't show up as
> a leak because the object is properly deleted at the end. Try the
> attached patch. With this change, I can see that the memory footprint
> of your Python example stays flat. I used Instruments on the Mac for
> this. It shows me all memory allocations of an applications.
>
> Best,
> -berk
>
>
> On Wed, Mar 5, 2014 at 9:32 AM, Berk Geveci <berk.geveci at kitware.com>
> wrote:
> > By the way, I have been looking into this as well. So far, I found no
> > memory issues either.
> >
> > PS: The time stepping for the EnSight reader is tested. I recently
> > added further coverage for it also.
> >
> > -berk
> >
> > On Wed, Mar 5, 2014 at 4:00 AM, Bernhard Righolt <b.w.righolt at tudelft.nl>
> wrote:
> >> Dear Bill,
> >>
> >> With your C++ script, I am indeed facing the same issue (although the
> python
> >> code runs significantly quicker for my case).
> >>
> >> I shared a few files, and a script to create more data here:
> >> https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz My experience is
> that
> >> with dissimilar data, the memory leak seems to be larger, but with this
> data
> >> (which just copies the time steps), it is still present,
> >>
> >> Best regards,
> >> Bernhard
> >>
> >>
> >> On 4 March 2014 21:27, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> >>>
> >>> Attached is my C++ program and CMakeLists.txt file. But I don't think
> >>> this is a python issue...
> >>>
> >>> On Tue, Mar 4, 2014 at 3:10 PM, Bernhard Righolt <
> b.w.righolt at tudelft.nl>
> >>> wrote:
> >>> > Dear Bill,
> >>> >
> >>> > I am a C++ speaker, so I don't mind to test your code on my current
> >>> > large
> >>> > dataset. Can you maybe share your code, such that I can test it? I've
> >>> > never
> >>> > used VTK within C++, if it solves the issues I am experiencing with
> >>> > Python,
> >>> > I am very willing to switch. I hope it is not too complicated to link
> >>> > the
> >>> > libraries correctly.
> >>> >
> >>> > Bernhard
> >>> >
> >>> >
> >>> >
> >>> > On 4 March 2014 21:05, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
> >>> >>
> >>> >> Bernhard,
> >>> >>
> >>> >> Thanks for your patience. Since I'm not a python person, I converted
> >>> >> your program to C++. I ran valgrind and found no memory leaks.
> >>> >>
> >>> >> Perhaps the small dataset you provided does not exercise the code
> that
> >>> >> is leaking.
> >>> >>
> >>> >> Can you possible place a dataset that you know has leaksat some
> >>> >> accessible
> >>> >> site?
> >>> >>
> >>> >> I will say, that I don't think any of the current ensight tests
> access
> >>> >> multiple time steps. Your data and test script will be a good
> starting
> >>> >> point for such a test.
> >>> >>
> >>> >> Bill
> >>> >>
> >>> >> On Tue, Mar 4, 2014 at 12:08 AM, Bernhard Righolt
> >>> >> <b.w.righolt at tudelft.nl> wrote:
> >>> >> > Therr is a python script inside the archive which is the same
> script
> >>> >> > that I
> >>> >> > used on the large case.
> >>> >> >
> >>> >> > Bernhard
> >>> >> >
> >>> >> >
> >>> >> > On Mar 4, 2014 12:22 AM, "Bill Lorensen" <bill.lorensen at gmail.com
> >
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> This is great. Can you send a complete python script that
> >>> >> >> illustrates
> >>> >> >> the
> >>> >> >> problem.?
> >>> >> >>
> >>> >> >> On Mar 3, 2014 2:47 PM, "Bernhard Righolt" <
> b.w.righolt at tudelft.nl>
> >>> >> >> wrote:
> >>> >> >> >
> >>> >> >> > Hi Bill,
> >>> >> >> >
> >>> >> >> > I added a very small sample data set. I am not sure if the
> memory
> >>> >> >> > problem I am encountering with my big data-set is measurable
> for
> >>> >> >> > this
> >>> >> >> > one.
> >>> >> >> >
> >>> >> >> > Anyhow, I would like to repeat that I am using Python 2.6 and
> VTK
> >>> >> >> > 5.10,
> >>> >> >> >
> >>> >> >> > Best regards,
> >>> >> >> > Bernhard
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > On 3 March 2014 19:25, Bill Lorensen <bill.lorensen at gmail.com>
> >>> >> >> > wrote:
> >>> >> >> >>
> >>> >> >> >> The ensight readers could use more testing. Can you provide a
> >>> >> >> >> "small"
> >>> >> >> >> dataset and program that cause memory leaks?
> >>> >> >> >>
> >>> >> >> >> Bill
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> On Mon, Mar 3, 2014 at 10:38 AM, Bernhard Righolt
> >>> >> >> >> <b.w.righolt at tudelft.nl> wrote:
> >>> >> >> >> > Dear all,
> >>> >> >> >> >
> >>> >> >> >> > A typical case file, looks as follows (ASCII).
> >>> >> >> >> >
> >>> >> >> >> > $ head -n 20 case.case
> >>> >> >> >> > FORMAT
> >>> >> >> >> > type: ensight gold
> >>> >> >> >> >
> >>> >> >> >> > GEOMETRY
> >>> >> >> >> > model:        1     sliceCentre.000.mesh
> >>> >> >> >> >
> >>> >> >> >> > VARIABLE
> >>> >> >> >> > vector per node:         1       U       sliceCentre.*****.U
> >>> >> >> >> >
> >>> >> >> >> > TIME
> >>> >> >> >> > time set:                      1
> >>> >> >> >> > number of steps:               11550
> >>> >> >> >> > filename start number:         1
> >>> >> >> >> > filename increment:            1
> >>> >> >> >> > time values:
> >>> >> >> >> > 0.002
> >>> >> >> >> > 0.004
> >>> >> >> >> > 0.006
> >>> >> >> >> > 0.008
> >>> >> >> >> > 0.01
> >>> >> >> >> > $
> >>> >> >> >> >
> >>> >> >> >> > The mesh (snippet only), is as follows:
> >>> >> >> >> >
> >>> >> >> >> > $ head -n 10 sliceCentre.000.mesh
> >>> >> >> >> > Ensight Geometry File
> >>> >> >> >> > =====================
> >>> >> >> >> > node id assign
> >>> >> >> >> > element id assign
> >>> >> >> >> > part
> >>> >> >> >> >          1
> >>> >> >> >> > sliceCentre.000.mesh
> >>> >> >> >> > coordinates
> >>> >> >> >> >     299028
> >>> >> >> >> >  0.00000e+00
> >>> >> >> >> > $ sed -n -e 897091,897098p sliceCentre.000.mesh
> >>> >> >> >> >  0.00000e+00
> >>> >> >> >> >  0.00000e+00
> >>> >> >> >> >  0.00000e+00
> >>> >> >> >> > tria3
> >>> >> >> >> >     595264
> >>> >> >> >> >     162106    162714    162799
> >>> >> >> >> >     162799    162226    162106
> >>> >> >> >> >     162714    162106    162021
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> > And the corresponding datafile
> >>> >> >> >> > $ head -n 10 sliceCentre.00001.U
> >>> >> >> >> > vector
> >>> >> >> >> > part
> >>> >> >> >> >          1
> >>> >> >> >> > coordinates
> >>> >> >> >> >  5.33425e-04
> >>> >> >> >> >  2.69515e-03
> >>> >> >> >> > -1.65164e-03
> >>> >> >> >> >  5.33425e-04
> >>> >> >> >> >  1.39069e-04
> >>> >> >> >> >  2.69515e-03
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> > However, I doubt if something is wrong with my datafiles
> >>> >> >> >> > itself,
> >>> >> >> >> > because I
> >>> >> >> >> > can easily read them in Paraview for example, the only
> >>> >> >> >> > difference
> >>> >> >> >> > is
> >>> >> >> >> > the
> >>> >> >> >> > large amount of data-files, e.g (124GB in this case)
> >>> >> >> >> >
> >>> >> >> >> > Best regards,
> >>> >> >> >> > Bernhard
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> > On 27 February 2014 22:28, Berk Geveci
> >>> >> >> >> > <berk.geveci at kitware.com>
> >>> >> >> >> > wrote:
> >>> >> >> >> >>
> >>> >> >> >> >> Nothing strikes me as obviously wrong in your script. Can
> you
> >>> >> >> >> >> send
> >>> >> >> >> >> me
> >>> >> >> >> >> your case file? I'll take a look.
> >>> >> >> >> >>
> >>> >> >> >> >> -berk
> >>> >> >> >> >>
> >>> >> >> >> >> On Thu, Feb 27, 2014 at 3:20 AM, Bernhard Righolt
> >>> >> >> >> >> <b.w.righolt at tudelft.nl> wrote:
> >>> >> >> >> >> > Dear all,
> >>> >> >> >> >> >
> >>> >> >> >> >> > Currently I am using a Python script to process
> information
> >>> >> >> >> >> > stored
> >>> >> >> >> >> > in
> >>> >> >> >> >> > the
> >>> >> >> >> >> > EnSight Gold format. My Python (2.6) scipt uses VTK
> (5.10.0)
> >>> >> >> >> >> > to
> >>> >> >> >> >> > process
> >>> >> >> >> >> > the
> >>> >> >> >> >> > file, where I used the vtkEnSightGoldReader for reading
> the
> >>> >> >> >> >> > data,
> >>> >> >> >> >> > and
> >>> >> >> >> >> > loop
> >>> >> >> >> >> > over time steps.
> >>> >> >> >> >> >
> >>> >> >> >> >> > In principle this works fine for smaller datasets,
> however,
> >>> >> >> >> >> > for
> >>> >> >> >> >> > large
> >>> >> >> >> >> > datasets (i.e. 11k files of 12MB), I see the memory usage
> >>> >> >> >> >> > (recorded via
> >>> >> >> >> >> > top)
> >>> >> >> >> >> > increasing with time while the process is running. This
> >>> >> >> >> >> > filling
> >>> >> >> >> >> > of
> >>> >> >> >> >> > the
> >>> >> >> >> >> > memory goes slow, but in some cases problems are
> inevitable.
> >>> >> >> >> >> > Apparently,
> >>> >> >> >> >> > I
> >>> >> >> >> >> > am doing some things wrong in my script with respect to
> >>> >> >> >> >> > memory
> >>> >> >> >> >> > management.
> >>> >> >> >> >> >
> >>> >> >> >> >> > I am using the following script (stripped obviously, but
> >>> >> >> >> >> > still
> >>> >> >> >> >> > showing
> >>> >> >> >> >> > this
> >>> >> >> >> >> > issue)
> >>> >> >> >> >> >
> >>> >> >> >> >> > import vtk
> >>> >> >> >> >> >
> >>> >> >> >> >> > reader = vtk.vtkEnSightGoldReader()
> >>> >> >> >> >> >
> >>> >> >> >> >> > reader.SetCaseFileName("case.case")
> >>> >> >> >> >> > reader.Update()
> >>> >> >> >> >> >
> >>> >> >> >> >> > # Get time values
> >>> >> >> >> >> > timeset=reader.GetTimeSets()
> >>> >> >> >> >> > time=timeset.GetItem(0)
> >>> >> >> >> >> > timesteps=time.GetSize()
> >>> >> >> >> >> >
> >>> >> >> >> >> > #reader.ReleaseDataFlagOn()
> >>> >> >> >> >> >
> >>> >> >> >> >> > for j in range(timesteps):
> >>> >> >> >> >> >     curTime=time.GetTuple(j)[0]
> >>> >> >> >> >> >     print curTime
> >>> >> >> >> >> >     reader.SetTimeValue(curTime)
> >>> >> >> >> >> >     reader.Update()
> >>> >> >> >> >> >
> >>> >> >> >> >> >     #reader.RemoveAllInputs()
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> > I tried a few things as you can see
> >>> >> >> >> >> > - reader.ReleaseDataFlagOn(), without success
> >>> >> >> >> >> > - reader.RemoveAllInputs(), did not get it to run
> >>> >> >> >> >> > - running gc.collect at the end of the loop, following
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html
> >>> >> >> >> >> > - Using DeepCopy as suggested here:
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html
> >>> >> >> >> >> > However, I am already experiencing this without a call to
> >>> >> >> >> >> > GetOutput
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> > My question is, how can I properly unload/replace the
> data
> >>> >> >> >> >> > that
> >>> >> >> >> >> > is
> >>> >> >> >> >> > stored in
> >>> >> >> >> >> > the memory, instead of using more memory continuously?
> >>> >> >> >> >> >
> >>> >> >> >> >> > Best regards,
> >>> >> >> >> >> > Bernhard
> >>> >> >> >> >> >
> >>> >> >> >> >> > _______________________________________________
> >>> >> >> >> >> > Powered by www.kitware.com
> >>> >> >> >> >> >
> >>> >> >> >> >> > Visit other Kitware open-source projects at
> >>> >> >> >> >> > http://www.kitware.com/opensource/opensource.html
> >>> >> >> >> >> >
> >>> >> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
> >>> >> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
> >>> >> >> >> >> >
> >>> >> >> >> >> > Follow this link to subscribe/unsubscribe:
> >>> >> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
> >>> >> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> > _______________________________________________
> >>> >> >> >> > Powered by www.kitware.com
> >>> >> >> >> >
> >>> >> >> >> > Visit other Kitware open-source projects at
> >>> >> >> >> > http://www.kitware.com/opensource/opensource.html
> >>> >> >> >> >
> >>> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
> >>> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
> >>> >> >> >> >
> >>> >> >> >> > Follow this link to subscribe/unsubscribe:
> >>> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
> >>> >> >> >> >
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> --
> >>> >> >> >> Unpaid intern in BillsBasement at noware dot com
> >>> >> >> >
> >>> >> >> >
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Unpaid intern in BillsBasement at noware dot com
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Unpaid intern in BillsBasement at noware dot com
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140305/ff960c7a/attachment.html>

More information about the vtkusers mailing list