[vtkusers] Memory leak vtkEnsightGoldReader in Python

Bernhard Righolt b.w.righolt at tudelft.nl
Thu Mar 13 03:34:26 EDT 2014 

Dear Berk,

Thanks again for digging into this issue. I compiled VTK 5.10.0 now with
this patch, and first tests indicate that the memory leak is indeed gone! I
started two processes with the original and patched version of VTK, where
the first one is showing the behavior described earlier, while for the
patched VTK, the memory usage does not seem to increase in time, so as far
as I can see my problem is solved. Thanks!

Bernhard



On 5 March 2014 19:25, Bernhard Righolt <b.w.righolt at tudelft.nl> wrote:

> Hi Berk,
>
> I will try it with this patch on my dataset. I had a dataset with 11,000+
> timesteps (and planning for a few more). I am not sure if it would grow
> linearly or quadratically with the amount of time-steps.
>
> I am not that experienced with VTK source cde, so before I do anything
> silly, can you confirm the following:
> - The patch is suitable for VTK 5.10 (or should I take a newer version?)
> - I apply the patch to in my VTK source (IO/vtkEnSightReader.cxx)
> - Recompile VTK
>
> I hope I can get back to you with results of my tests as soon as possible.
> Both of you already many thanks for helping me!
>
> Best regards,
> Bernhard
>
>
>
> On 5 March 2014 18:33, Berk Geveci <berk.geveci at kitware.com> wrote:
>
>> I went over the EnSight Gold reader carefully and there is only one
>> small memory issue. Instead of a clearing a collection object every
>> timestep, it keeps adding items to it. This accumulates to something
>> small (7.5MB over 1000 timesteps in my example). It doesn't show up as
>> a leak because the object is properly deleted at the end. Try the
>> attached patch. With this change, I can see that the memory footprint
>> of your Python example stays flat. I used Instruments on the Mac for
>> this. It shows me all memory allocations of an applications.
>>
>> Best,
>> -berk
>>
>>
>> On Wed, Mar 5, 2014 at 9:32 AM, Berk Geveci <berk.geveci at kitware.com>
>> wrote:
>> > By the way, I have been looking into this as well. So far, I found no
>> > memory issues either.
>> >
>> > PS: The time stepping for the EnSight reader is tested. I recently
>> > added further coverage for it also.
>> >
>> > -berk
>> >
>> > On Wed, Mar 5, 2014 at 4:00 AM, Bernhard Righolt <
>> b.w.righolt at tudelft.nl> wrote:
>> >> Dear Bill,
>> >>
>> >> With your C++ script, I am indeed facing the same issue (although the
>> python
>> >> code runs significantly quicker for my case).
>> >>
>> >> I shared a few files, and a script to create more data here:
>> >> https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz My experience
>> is that
>> >> with dissimilar data, the memory leak seems to be larger, but with
>> this data
>> >> (which just copies the time steps), it is still present,
>> >>
>> >> Best regards,
>> >> Bernhard
>> >>
>> >>
>> >> On 4 March 2014 21:27, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> >>>
>> >>> Attached is my C++ program and CMakeLists.txt file. But I don't think
>> >>> this is a python issue...
>> >>>
>> >>> On Tue, Mar 4, 2014 at 3:10 PM, Bernhard Righolt <
>> b.w.righolt at tudelft.nl>
>> >>> wrote:
>> >>> > Dear Bill,
>> >>> >
>> >>> > I am a C++ speaker, so I don't mind to test your code on my current
>> >>> > large
>> >>> > dataset. Can you maybe share your code, such that I can test it?
>> I've
>> >>> > never
>> >>> > used VTK within C++, if it solves the issues I am experiencing with
>> >>> > Python,
>> >>> > I am very willing to switch. I hope it is not too complicated to
>> link
>> >>> > the
>> >>> > libraries correctly.
>> >>> >
>> >>> > Bernhard
>> >>> >
>> >>> >
>> >>> >
>> >>> > On 4 March 2014 21:05, Bill Lorensen <bill.lorensen at gmail.com>
>> wrote:
>> >>> >>
>> >>> >> Bernhard,
>> >>> >>
>> >>> >> Thanks for your patience. Since I'm not a python person, I
>> converted
>> >>> >> your program to C++. I ran valgrind and found no memory leaks.
>> >>> >>
>> >>> >> Perhaps the small dataset you provided does not exercise the code
>> that
>> >>> >> is leaking.
>> >>> >>
>> >>> >> Can you possible place a dataset that you know has leaksat some
>> >>> >> accessible
>> >>> >> site?
>> >>> >>
>> >>> >> I will say, that I don't think any of the current ensight tests
>> access
>> >>> >> multiple time steps. Your data and test script will be a good
>> starting
>> >>> >> point for such a test.
>> >>> >>
>> >>> >> Bill
>> >>> >>
>> >>> >> On Tue, Mar 4, 2014 at 12:08 AM, Bernhard Righolt
>> >>> >> <b.w.righolt at tudelft.nl> wrote:
>> >>> >> > Therr is a python script inside the archive which is the same
>> script
>> >>> >> > that I
>> >>> >> > used on the large case.
>> >>> >> >
>> >>> >> > Bernhard
>> >>> >> >
>> >>> >> >
>> >>> >> > On Mar 4, 2014 12:22 AM, "Bill Lorensen" <
>> bill.lorensen at gmail.com>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> This is great. Can you send a complete python script that
>> >>> >> >> illustrates
>> >>> >> >> the
>> >>> >> >> problem.?
>> >>> >> >>
>> >>> >> >> On Mar 3, 2014 2:47 PM, "Bernhard Righolt" <
>> b.w.righolt at tudelft.nl>
>> >>> >> >> wrote:
>> >>> >> >> >
>> >>> >> >> > Hi Bill,
>> >>> >> >> >
>> >>> >> >> > I added a very small sample data set. I am not sure if the
>> memory
>> >>> >> >> > problem I am encountering with my big data-set is measurable
>> for
>> >>> >> >> > this
>> >>> >> >> > one.
>> >>> >> >> >
>> >>> >> >> > Anyhow, I would like to repeat that I am using Python 2.6 and
>> VTK
>> >>> >> >> > 5.10,
>> >>> >> >> >
>> >>> >> >> > Best regards,
>> >>> >> >> > Bernhard
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > On 3 March 2014 19:25, Bill Lorensen <bill.lorensen at gmail.com
>> >
>> >>> >> >> > wrote:
>> >>> >> >> >>
>> >>> >> >> >> The ensight readers could use more testing. Can you provide a
>> >>> >> >> >> "small"
>> >>> >> >> >> dataset and program that cause memory leaks?
>> >>> >> >> >>
>> >>> >> >> >> Bill
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >> On Mon, Mar 3, 2014 at 10:38 AM, Bernhard Righolt
>> >>> >> >> >> <b.w.righolt at tudelft.nl> wrote:
>> >>> >> >> >> > Dear all,
>> >>> >> >> >> >
>> >>> >> >> >> > A typical case file, looks as follows (ASCII).
>> >>> >> >> >> >
>> >>> >> >> >> > $ head -n 20 case.case
>> >>> >> >> >> > FORMAT
>> >>> >> >> >> > type: ensight gold
>> >>> >> >> >> >
>> >>> >> >> >> > GEOMETRY
>> >>> >> >> >> > model:        1     sliceCentre.000.mesh
>> >>> >> >> >> >
>> >>> >> >> >> > VARIABLE
>> >>> >> >> >> > vector per node:         1       U
>> sliceCentre.*****.U
>> >>> >> >> >> >
>> >>> >> >> >> > TIME
>> >>> >> >> >> > time set:                      1
>> >>> >> >> >> > number of steps:               11550
>> >>> >> >> >> > filename start number:         1
>> >>> >> >> >> > filename increment:            1
>> >>> >> >> >> > time values:
>> >>> >> >> >> > 0.002
>> >>> >> >> >> > 0.004
>> >>> >> >> >> > 0.006
>> >>> >> >> >> > 0.008
>> >>> >> >> >> > 0.01
>> >>> >> >> >> > $
>> >>> >> >> >> >
>> >>> >> >> >> > The mesh (snippet only), is as follows:
>> >>> >> >> >> >
>> >>> >> >> >> > $ head -n 10 sliceCentre.000.mesh
>> >>> >> >> >> > Ensight Geometry File
>> >>> >> >> >> > =====================
>> >>> >> >> >> > node id assign
>> >>> >> >> >> > element id assign
>> >>> >> >> >> > part
>> >>> >> >> >> >          1
>> >>> >> >> >> > sliceCentre.000.mesh
>> >>> >> >> >> > coordinates
>> >>> >> >> >> >     299028
>> >>> >> >> >> >  0.00000e+00
>> >>> >> >> >> > $ sed -n -e 897091,897098p sliceCentre.000.mesh
>> >>> >> >> >> >  0.00000e+00
>> >>> >> >> >> >  0.00000e+00
>> >>> >> >> >> >  0.00000e+00
>> >>> >> >> >> > tria3
>> >>> >> >> >> >     595264
>> >>> >> >> >> >     162106    162714    162799
>> >>> >> >> >> >     162799    162226    162106
>> >>> >> >> >> >     162714    162106    162021
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> > And the corresponding datafile
>> >>> >> >> >> > $ head -n 10 sliceCentre.00001.U
>> >>> >> >> >> > vector
>> >>> >> >> >> > part
>> >>> >> >> >> >          1
>> >>> >> >> >> > coordinates
>> >>> >> >> >> >  5.33425e-04
>> >>> >> >> >> >  2.69515e-03
>> >>> >> >> >> > -1.65164e-03
>> >>> >> >> >> >  5.33425e-04
>> >>> >> >> >> >  1.39069e-04
>> >>> >> >> >> >  2.69515e-03
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> > However, I doubt if something is wrong with my datafiles
>> >>> >> >> >> > itself,
>> >>> >> >> >> > because I
>> >>> >> >> >> > can easily read them in Paraview for example, the only
>> >>> >> >> >> > difference
>> >>> >> >> >> > is
>> >>> >> >> >> > the
>> >>> >> >> >> > large amount of data-files, e.g (124GB in this case)
>> >>> >> >> >> >
>> >>> >> >> >> > Best regards,
>> >>> >> >> >> > Bernhard
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> > On 27 February 2014 22:28, Berk Geveci
>> >>> >> >> >> > <berk.geveci at kitware.com>
>> >>> >> >> >> > wrote:
>> >>> >> >> >> >>
>> >>> >> >> >> >> Nothing strikes me as obviously wrong in your script. Can
>> you
>> >>> >> >> >> >> send
>> >>> >> >> >> >> me
>> >>> >> >> >> >> your case file? I'll take a look.
>> >>> >> >> >> >>
>> >>> >> >> >> >> -berk
>> >>> >> >> >> >>
>> >>> >> >> >> >> On Thu, Feb 27, 2014 at 3:20 AM, Bernhard Righolt
>> >>> >> >> >> >> <b.w.righolt at tudelft.nl> wrote:
>> >>> >> >> >> >> > Dear all,
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Currently I am using a Python script to process
>> information
>> >>> >> >> >> >> > stored
>> >>> >> >> >> >> > in
>> >>> >> >> >> >> > the
>> >>> >> >> >> >> > EnSight Gold format. My Python (2.6) scipt uses VTK
>> (5.10.0)
>> >>> >> >> >> >> > to
>> >>> >> >> >> >> > process
>> >>> >> >> >> >> > the
>> >>> >> >> >> >> > file, where I used the vtkEnSightGoldReader for reading
>> the
>> >>> >> >> >> >> > data,
>> >>> >> >> >> >> > and
>> >>> >> >> >> >> > loop
>> >>> >> >> >> >> > over time steps.
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > In principle this works fine for smaller datasets,
>> however,
>> >>> >> >> >> >> > for
>> >>> >> >> >> >> > large
>> >>> >> >> >> >> > datasets (i.e. 11k files of 12MB), I see the memory
>> usage
>> >>> >> >> >> >> > (recorded via
>> >>> >> >> >> >> > top)
>> >>> >> >> >> >> > increasing with time while the process is running. This
>> >>> >> >> >> >> > filling
>> >>> >> >> >> >> > of
>> >>> >> >> >> >> > the
>> >>> >> >> >> >> > memory goes slow, but in some cases problems are
>> inevitable.
>> >>> >> >> >> >> > Apparently,
>> >>> >> >> >> >> > I
>> >>> >> >> >> >> > am doing some things wrong in my script with respect to
>> >>> >> >> >> >> > memory
>> >>> >> >> >> >> > management.
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > I am using the following script (stripped obviously, but
>> >>> >> >> >> >> > still
>> >>> >> >> >> >> > showing
>> >>> >> >> >> >> > this
>> >>> >> >> >> >> > issue)
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > import vtk
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > reader = vtk.vtkEnSightGoldReader()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > reader.SetCaseFileName("case.case")
>> >>> >> >> >> >> > reader.Update()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > # Get time values
>> >>> >> >> >> >> > timeset=reader.GetTimeSets()
>> >>> >> >> >> >> > time=timeset.GetItem(0)
>> >>> >> >> >> >> > timesteps=time.GetSize()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > #reader.ReleaseDataFlagOn()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > for j in range(timesteps):
>> >>> >> >> >> >> >     curTime=time.GetTuple(j)[0]
>> >>> >> >> >> >> >     print curTime
>> >>> >> >> >> >> >     reader.SetTimeValue(curTime)
>> >>> >> >> >> >> >     reader.Update()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >     #reader.RemoveAllInputs()
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > I tried a few things as you can see
>> >>> >> >> >> >> > - reader.ReleaseDataFlagOn(), without success
>> >>> >> >> >> >> > - reader.RemoveAllInputs(), did not get it to run
>> >>> >> >> >> >> > - running gc.collect at the end of the loop, following
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html
>> >>> >> >> >> >> > - Using DeepCopy as suggested here:
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html
>> >>> >> >> >> >> > However, I am already experiencing this without a call
>> to
>> >>> >> >> >> >> > GetOutput
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > My question is, how can I properly unload/replace the
>> data
>> >>> >> >> >> >> > that
>> >>> >> >> >> >> > is
>> >>> >> >> >> >> > stored in
>> >>> >> >> >> >> > the memory, instead of using more memory continuously?
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Best regards,
>> >>> >> >> >> >> > Bernhard
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > _______________________________________________
>> >>> >> >> >> >> > Powered by www.kitware.com
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Visit other Kitware open-source projects at
>> >>> >> >> >> >> > http://www.kitware.com/opensource/opensource.html
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
>> >>> >> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Follow this link to subscribe/unsubscribe:
>> >>> >> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
>> >>> >> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> > _______________________________________________
>> >>> >> >> >> > Powered by www.kitware.com
>> >>> >> >> >> >
>> >>> >> >> >> > Visit other Kitware open-source projects at
>> >>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>> >>> >> >> >> >
>> >>> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
>> >>> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
>> >>> >> >> >> >
>> >>> >> >> >> > Follow this link to subscribe/unsubscribe:
>> >>> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
>> >>> >> >> >> >
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >> --
>> >>> >> >> >> Unpaid intern in BillsBasement at noware dot com
>> >>> >> >> >
>> >>> >> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Unpaid intern in BillsBasement at noware dot com
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Unpaid intern in BillsBasement at noware dot com
>> >>
>> >>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140313/7ac18770/attachment.html>

More information about the vtkusers mailing list