[vtkusers] Memory leak vtkEnsightGoldReader in Python
Berk Geveci
berk.geveci at kitware.com
Wed Mar 5 12:33:34 EST 2014
I went over the EnSight Gold reader carefully and there is only one
small memory issue. Instead of a clearing a collection object every
timestep, it keeps adding items to it. This accumulates to something
small (7.5MB over 1000 timesteps in my example). It doesn't show up as
a leak because the object is properly deleted at the end. Try the
attached patch. With this change, I can see that the memory footprint
of your Python example stays flat. I used Instruments on the Mac for
this. It shows me all memory allocations of an applications.
Best,
-berk
On Wed, Mar 5, 2014 at 9:32 AM, Berk Geveci <berk.geveci at kitware.com> wrote:
> By the way, I have been looking into this as well. So far, I found no
> memory issues either.
>
> PS: The time stepping for the EnSight reader is tested. I recently
> added further coverage for it also.
>
> -berk
>
> On Wed, Mar 5, 2014 at 4:00 AM, Bernhard Righolt <b.w.righolt at tudelft.nl> wrote:
>> Dear Bill,
>>
>> With your C++ script, I am indeed facing the same issue (although the python
>> code runs significantly quicker for my case).
>>
>> I shared a few files, and a script to create more data here:
>> https://www.dropbox.com/s/ndg6mwam2xdfmwq/data.tar.gz My experience is that
>> with dissimilar data, the memory leak seems to be larger, but with this data
>> (which just copies the time steps), it is still present,
>>
>> Best regards,
>> Bernhard
>>
>>
>> On 4 March 2014 21:27, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>
>>> Attached is my C++ program and CMakeLists.txt file. But I don't think
>>> this is a python issue...
>>>
>>> On Tue, Mar 4, 2014 at 3:10 PM, Bernhard Righolt <b.w.righolt at tudelft.nl>
>>> wrote:
>>> > Dear Bill,
>>> >
>>> > I am a C++ speaker, so I don't mind to test your code on my current
>>> > large
>>> > dataset. Can you maybe share your code, such that I can test it? I've
>>> > never
>>> > used VTK within C++, if it solves the issues I am experiencing with
>>> > Python,
>>> > I am very willing to switch. I hope it is not too complicated to link
>>> > the
>>> > libraries correctly.
>>> >
>>> > Bernhard
>>> >
>>> >
>>> >
>>> > On 4 March 2014 21:05, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> >>
>>> >> Bernhard,
>>> >>
>>> >> Thanks for your patience. Since I'm not a python person, I converted
>>> >> your program to C++. I ran valgrind and found no memory leaks.
>>> >>
>>> >> Perhaps the small dataset you provided does not exercise the code that
>>> >> is leaking.
>>> >>
>>> >> Can you possible place a dataset that you know has leaksat some
>>> >> accessible
>>> >> site?
>>> >>
>>> >> I will say, that I don't think any of the current ensight tests access
>>> >> multiple time steps. Your data and test script will be a good starting
>>> >> point for such a test.
>>> >>
>>> >> Bill
>>> >>
>>> >> On Tue, Mar 4, 2014 at 12:08 AM, Bernhard Righolt
>>> >> <b.w.righolt at tudelft.nl> wrote:
>>> >> > Therr is a python script inside the archive which is the same script
>>> >> > that I
>>> >> > used on the large case.
>>> >> >
>>> >> > Bernhard
>>> >> >
>>> >> >
>>> >> > On Mar 4, 2014 12:22 AM, "Bill Lorensen" <bill.lorensen at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> This is great. Can you send a complete python script that
>>> >> >> illustrates
>>> >> >> the
>>> >> >> problem.?
>>> >> >>
>>> >> >> On Mar 3, 2014 2:47 PM, "Bernhard Righolt" <b.w.righolt at tudelft.nl>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > Hi Bill,
>>> >> >> >
>>> >> >> > I added a very small sample data set. I am not sure if the memory
>>> >> >> > problem I am encountering with my big data-set is measurable for
>>> >> >> > this
>>> >> >> > one.
>>> >> >> >
>>> >> >> > Anyhow, I would like to repeat that I am using Python 2.6 and VTK
>>> >> >> > 5.10,
>>> >> >> >
>>> >> >> > Best regards,
>>> >> >> > Bernhard
>>> >> >> >
>>> >> >> >
>>> >> >> > On 3 March 2014 19:25, Bill Lorensen <bill.lorensen at gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> The ensight readers could use more testing. Can you provide a
>>> >> >> >> "small"
>>> >> >> >> dataset and program that cause memory leaks?
>>> >> >> >>
>>> >> >> >> Bill
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Mon, Mar 3, 2014 at 10:38 AM, Bernhard Righolt
>>> >> >> >> <b.w.righolt at tudelft.nl> wrote:
>>> >> >> >> > Dear all,
>>> >> >> >> >
>>> >> >> >> > A typical case file, looks as follows (ASCII).
>>> >> >> >> >
>>> >> >> >> > $ head -n 20 case.case
>>> >> >> >> > FORMAT
>>> >> >> >> > type: ensight gold
>>> >> >> >> >
>>> >> >> >> > GEOMETRY
>>> >> >> >> > model: 1 sliceCentre.000.mesh
>>> >> >> >> >
>>> >> >> >> > VARIABLE
>>> >> >> >> > vector per node: 1 U sliceCentre.*****.U
>>> >> >> >> >
>>> >> >> >> > TIME
>>> >> >> >> > time set: 1
>>> >> >> >> > number of steps: 11550
>>> >> >> >> > filename start number: 1
>>> >> >> >> > filename increment: 1
>>> >> >> >> > time values:
>>> >> >> >> > 0.002
>>> >> >> >> > 0.004
>>> >> >> >> > 0.006
>>> >> >> >> > 0.008
>>> >> >> >> > 0.01
>>> >> >> >> > $
>>> >> >> >> >
>>> >> >> >> > The mesh (snippet only), is as follows:
>>> >> >> >> >
>>> >> >> >> > $ head -n 10 sliceCentre.000.mesh
>>> >> >> >> > Ensight Geometry File
>>> >> >> >> > =====================
>>> >> >> >> > node id assign
>>> >> >> >> > element id assign
>>> >> >> >> > part
>>> >> >> >> > 1
>>> >> >> >> > sliceCentre.000.mesh
>>> >> >> >> > coordinates
>>> >> >> >> > 299028
>>> >> >> >> > 0.00000e+00
>>> >> >> >> > $ sed -n -e 897091,897098p sliceCentre.000.mesh
>>> >> >> >> > 0.00000e+00
>>> >> >> >> > 0.00000e+00
>>> >> >> >> > 0.00000e+00
>>> >> >> >> > tria3
>>> >> >> >> > 595264
>>> >> >> >> > 162106 162714 162799
>>> >> >> >> > 162799 162226 162106
>>> >> >> >> > 162714 162106 162021
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > And the corresponding datafile
>>> >> >> >> > $ head -n 10 sliceCentre.00001.U
>>> >> >> >> > vector
>>> >> >> >> > part
>>> >> >> >> > 1
>>> >> >> >> > coordinates
>>> >> >> >> > 5.33425e-04
>>> >> >> >> > 2.69515e-03
>>> >> >> >> > -1.65164e-03
>>> >> >> >> > 5.33425e-04
>>> >> >> >> > 1.39069e-04
>>> >> >> >> > 2.69515e-03
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > However, I doubt if something is wrong with my datafiles
>>> >> >> >> > itself,
>>> >> >> >> > because I
>>> >> >> >> > can easily read them in Paraview for example, the only
>>> >> >> >> > difference
>>> >> >> >> > is
>>> >> >> >> > the
>>> >> >> >> > large amount of data-files, e.g (124GB in this case)
>>> >> >> >> >
>>> >> >> >> > Best regards,
>>> >> >> >> > Bernhard
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On 27 February 2014 22:28, Berk Geveci
>>> >> >> >> > <berk.geveci at kitware.com>
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> Nothing strikes me as obviously wrong in your script. Can you
>>> >> >> >> >> send
>>> >> >> >> >> me
>>> >> >> >> >> your case file? I'll take a look.
>>> >> >> >> >>
>>> >> >> >> >> -berk
>>> >> >> >> >>
>>> >> >> >> >> On Thu, Feb 27, 2014 at 3:20 AM, Bernhard Righolt
>>> >> >> >> >> <b.w.righolt at tudelft.nl> wrote:
>>> >> >> >> >> > Dear all,
>>> >> >> >> >> >
>>> >> >> >> >> > Currently I am using a Python script to process information
>>> >> >> >> >> > stored
>>> >> >> >> >> > in
>>> >> >> >> >> > the
>>> >> >> >> >> > EnSight Gold format. My Python (2.6) scipt uses VTK (5.10.0)
>>> >> >> >> >> > to
>>> >> >> >> >> > process
>>> >> >> >> >> > the
>>> >> >> >> >> > file, where I used the vtkEnSightGoldReader for reading the
>>> >> >> >> >> > data,
>>> >> >> >> >> > and
>>> >> >> >> >> > loop
>>> >> >> >> >> > over time steps.
>>> >> >> >> >> >
>>> >> >> >> >> > In principle this works fine for smaller datasets, however,
>>> >> >> >> >> > for
>>> >> >> >> >> > large
>>> >> >> >> >> > datasets (i.e. 11k files of 12MB), I see the memory usage
>>> >> >> >> >> > (recorded via
>>> >> >> >> >> > top)
>>> >> >> >> >> > increasing with time while the process is running. This
>>> >> >> >> >> > filling
>>> >> >> >> >> > of
>>> >> >> >> >> > the
>>> >> >> >> >> > memory goes slow, but in some cases problems are inevitable.
>>> >> >> >> >> > Apparently,
>>> >> >> >> >> > I
>>> >> >> >> >> > am doing some things wrong in my script with respect to
>>> >> >> >> >> > memory
>>> >> >> >> >> > management.
>>> >> >> >> >> >
>>> >> >> >> >> > I am using the following script (stripped obviously, but
>>> >> >> >> >> > still
>>> >> >> >> >> > showing
>>> >> >> >> >> > this
>>> >> >> >> >> > issue)
>>> >> >> >> >> >
>>> >> >> >> >> > import vtk
>>> >> >> >> >> >
>>> >> >> >> >> > reader = vtk.vtkEnSightGoldReader()
>>> >> >> >> >> >
>>> >> >> >> >> > reader.SetCaseFileName("case.case")
>>> >> >> >> >> > reader.Update()
>>> >> >> >> >> >
>>> >> >> >> >> > # Get time values
>>> >> >> >> >> > timeset=reader.GetTimeSets()
>>> >> >> >> >> > time=timeset.GetItem(0)
>>> >> >> >> >> > timesteps=time.GetSize()
>>> >> >> >> >> >
>>> >> >> >> >> > #reader.ReleaseDataFlagOn()
>>> >> >> >> >> >
>>> >> >> >> >> > for j in range(timesteps):
>>> >> >> >> >> > curTime=time.GetTuple(j)[0]
>>> >> >> >> >> > print curTime
>>> >> >> >> >> > reader.SetTimeValue(curTime)
>>> >> >> >> >> > reader.Update()
>>> >> >> >> >> >
>>> >> >> >> >> > #reader.RemoveAllInputs()
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > I tried a few things as you can see
>>> >> >> >> >> > - reader.ReleaseDataFlagOn(), without success
>>> >> >> >> >> > - reader.RemoveAllInputs(), did not get it to run
>>> >> >> >> >> > - running gc.collect at the end of the loop, following
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > http://vtk.1045678.n5.nabble.com/VTK-memory-management-td5715661.html
>>> >> >> >> >> > - Using DeepCopy as suggested here:
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > http://vtk.1045678.n5.nabble.com/Memory-leak-in-VTK-Python-module-td1234002.html
>>> >> >> >> >> > However, I am already experiencing this without a call to
>>> >> >> >> >> > GetOutput
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > My question is, how can I properly unload/replace the data
>>> >> >> >> >> > that
>>> >> >> >> >> > is
>>> >> >> >> >> > stored in
>>> >> >> >> >> > the memory, instead of using more memory continuously?
>>> >> >> >> >> >
>>> >> >> >> >> > Best regards,
>>> >> >> >> >> > Bernhard
>>> >> >> >> >> >
>>> >> >> >> >> > _______________________________________________
>>> >> >> >> >> > Powered by www.kitware.com
>>> >> >> >> >> >
>>> >> >> >> >> > Visit other Kitware open-source projects at
>>> >> >> >> >> > http://www.kitware.com/opensource/opensource.html
>>> >> >> >> >> >
>>> >> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
>>> >> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
>>> >> >> >> >> >
>>> >> >> >> >> > Follow this link to subscribe/unsubscribe:
>>> >> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
>>> >> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > _______________________________________________
>>> >> >> >> > Powered by www.kitware.com
>>> >> >> >> >
>>> >> >> >> > Visit other Kitware open-source projects at
>>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>>> >> >> >> >
>>> >> >> >> > Please keep messages on-topic and check the VTK FAQ at:
>>> >> >> >> > http://www.vtk.org/Wiki/VTK_FAQ
>>> >> >> >> >
>>> >> >> >> > Follow this link to subscribe/unsubscribe:
>>> >> >> >> > http://www.vtk.org/mailman/listinfo/vtkusers
>>> >> >> >> >
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> --
>>> >> >> >> Unpaid intern in BillsBasement at noware dot com
>>> >> >> >
>>> >> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Unpaid intern in BillsBasement at noware dot com
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Unpaid intern in BillsBasement at noware dot com
>>
>>
-------------- next part --------------
diff --git a/IO/vtkEnSightReader.cxx b/IO/vtkEnSightReader.cxx
index 68a9b8f..7ab8ddd 100644
--- a/IO/vtkEnSightReader.cxx
+++ b/IO/vtkEnSightReader.cxx
@@ -985,6 +985,8 @@ int vtkEnSightReader::ReadCaseFileTime(char* line)
int timeSet, numTimeSteps, i, filenameNum, increment, lineRead;
float timeStep;
+ this->TimeSetFileNameNumbers->RemoveAllItems();
+
// found TIME section
int firstTimeStep = 1;
More information about the vtkusers
mailing list