[Paraview] Memory overload and I don't know what to delete (despite looking in the archives)

Tue Jan 19 17:00:48 EST 2016

Tim, 

The only thing I can recommend with your Delete() calls is to delete things in the reverse order from which they were created. So you should try doing: 

Delete(DataRepresentation1) 
Delete(RenderView1) 
Delete(AnimationScene1) 
Delete(VoltParaviewMovie) 

Other than that suggestion, I don't know if I can be much help. So if that doesn't work, somebody else will have to chime in. As I mentioned, we generally just loop over what we want to create instead. 

However, it looks like you are generating multiple movies? In that case, you wouldn't need to dump the frames and convert later -- just have your driver script loop over the movies one at a time. So long as creating a single movie works well enough, just have each call to pvpython create a single movie and loop over calls to pvpython. But that is if you want to use the other option and avoid the Delete's. 

Tim 

----- Original Message -----

From: "Tim De Coster" <tim91decoster at gmail.com> 
To: "tim gallagher" <tim.gallagher at gatech.edu> 
Cc: paraview at paraview.org 
Sent: Tuesday, January 19, 2016 4:33:49 PM 
Subject: Re: [Paraview] Memory overload and I don't know what to delete (despite looking in the archives) 

Sir, 

Thank you very much for answering so quickly. I went with the first option and tried the Delete() statements again, but still keep getting errors, be it different ones. Specifically I have put the following four statements right after Render() in my code: 

Delete(VoltParaviewMovie) 
Delete(AnimationScene1) 
Delete(RenderView1) 
Delete(DataRepresentation1) 

After executing I get the following error: 

Traceback (most recent call last): 
File "./RunParaviewMovieMaker.py", line 99, in <module> 
main() 
File "./RunParaviewMovieMaker.py", line 21, in main 
create_movie(filename3) 
File "./RunParaviewMovieMaker.py", line 91, in create_movie 
Delete(DataRepresentation1) 
File "/usr/lib/python2.7/dist-packages/paraview/simple.py", line 482, in Delete 
servermanager.UnRegister(proxy) 
File "/usr/lib/python2.7/dist-packages/paraview/servermanager.py", line 2764, in UnRegister 
raise RuntimeError, "UnRegistration error." 
RuntimeError: UnRegistration error. 
*** Error in `/usr/bin/python': corrupted double-linked list: 0x0000000004633310 *** 
Aborted (core dumped) 

Upon removing the last line, and just working with 3 Delete() statements, I also get an error, namely: 

Traceback (most recent call last): 
File "./RunParaviewMovieMaker.py", line 98, in <module> 
main() 
File "./RunParaviewMovieMaker.py", line 21, in main 
create_movie(filename3) 
File "./RunParaviewMovieMaker.py", line 44, in create_movie 
AnimationScene1 = GetAnimationScene() 
File "/usr/lib/python2.7/dist-packages/paraview/simple.py", line 794, in GetAnimationScene 
raise servermanager.MissingProxy, "Could not locate global AnimationScene." 
paraview.servermanager.MissingProxy: Could not locate global AnimationScene. 
*** Error in `/usr/bin/python': double free or corruption (!prev): 0x0000000002908e30 *** 
Aborted (core dumped) 

Do you have any idea on how to resolve these errors? I do especially not understand what the 'proxy' is doing in my errors. 
If not, I might opt for the second option with ImageMagick (although I would still prefer to resolve my current issues). 

With kind regards, 
Tim 

2016-01-19 18:28 GMT+01:00 Tim Gallagher < tim.gallagher at gatech.edu > : 

Your delete line is 

Delete(Animationscene1) 

but your variable is actually called AnimationScene1 (camel-case lettering). Python is case-sensitive, so that's why you are getting the error you are showing. 

That said, we do two things to loop over everything. We can Delete() every object, but that can be tedious in complicated pipelines. Alternatively, we have a driver script that executes pvpython on a trace script and passes in a number to indicate which file should be rendered. Our trace script then dumps a single PNG image and we stitch them together to make a movie afterwards using ImageMagick. Since each image is rendered in an entirely new instance of pvpython, there are no memory issues. An example driver script is below. We've also made our driver script parallel so it splits the range of data files into chunks for each processor, and then pvpython is called multiple times in parallel to render the files. 

Tim 

import subprocess 

start = 0 
end = 100 
stride = 1 

for n in range(start, end + 1, stride): 
print "Working on file: ", n 
subprocess.call(['/data4/pv4.1Install-OS/bin/pvpython', 'jicf_trace.py', '%i' % (n)]) 

From: "Tim De Coster" < tim91decoster at gmail.com > 
To: paraview at paraview.org 
Sent: Tuesday, January 19, 2016 10:03:01 AM 
Subject: [Paraview] Memory overload and I don't know what to delete (despite looking in the archives) 

Hi all, 

I am stuck with a problem regarding ParaView (version 4.0.1 64-bit), running on Ubuntu 14.04. It seems that when I run a python script, my memory gets overloaded. I looked already in the archives and it seems that I need to delete some data using the command Delete(). When I try this however, I get error messages saying that the things I want to delete are not possible 
( File "./RunParaviewMovieMaker.py", line 89, in create_movie 
Delete(Animationscene1) 
NameError: global name 'Animationscene1' is not defined 
*** Error in `/usr/bin/python': corrupted double-linked list: 0x0000000002b52ca0 *** 
). 
Let me first explain my problem in more detail: 

I have a lot of data that I generated with another script (about 3TB and still increasing), but now I need to convert these data into movies. When I run my loop over different folders, the first few movies are generated in less than 30s each, but then gradually the time per movie starts increasing and my memory gets overloaded. The script I am running is (which I simply got from using the trace call for python in ParaView): 

#!/usr/bin/python 

from paraview.simple import * 

def main(): 

some huge for-loop generating names of maps where the .vtk files are stored. In this for loop I use the command: 
create_movie(filename3) 

def create_movie(mapname): 

vtkfolder = '.' 
number_of_frames = 2000 

vtknames = [] 
for i in range (0, number_of_frames - 1): 
vtknames.append('{1}/{2}/VoltParaviewMovie{0:0=4d}.vtk'.format(i, vtkfolder, mapname)) 

VoltParaviewMovie = LegacyVTKReader(FileNames = vtknames) 

AnimationScene1 = GetAnimationScene() 
AnimationScene1 = GetAnimationScene() 
AnimationScene1.EndTime = 1999.0 
AnimationScene1.PlayMode = 'Snap To TimeSteps' 

AnimationScene1.EndTime = 1999.0 
AnimationScene1.PlayMode = 'Snap To TimeSteps' 

RenderView1 = GetRenderView() 
a1_Voltage_PVLookupTable = GetLookupTableForArray( "Voltage", 1, RGBPoints=[-85.52999877929688, 0.23, 0.299, 0.754, 50.0, 0.706, 0.016, 0.15], VectorMode='Magnitude', NanColor=[0.25, 0.0, 0.0], ColorSpace='Diverging', ScalarRangeInitialized=1.0, AllowDuplicateScalars=1 ) 

a1_Voltage_PiecewiseFunction = CreatePiecewiseFunction( Points=[0.0, 0.0, 0.5, 0.0, 1.0, 1.0, 0.5, 0.0] ) 

DataRepresentation1 = Show() 
DataRepresentation1.EdgeColor = [0.0, 0.0, 0.5000076295109483] 
DataRepresentation1.SelectionPointFieldDataArrayName = 'Voltage' 
DataRepresentation1.ScalarOpacityFunction = a1_Voltage_PiecewiseFunction 
DataRepresentation1.ColorArrayName = ('POINT_DATA', 'Voltage') 
DataRepresentation1.ScalarOpacityUnitDistance = 7.108580809929175 
DataRepresentation1.LookupTable = a1_Voltage_PVLookupTable 
DataRepresentation1.Representation = 'Slice' 
DataRepresentation1.ScaleFactor = 0.0 

RenderView1.CameraFocalPoint = [63.5, 63.5, 0.0] 
RenderView1.CameraPosition = [63.5, 63.5, 10000.0] 
RenderView1.InteractionMode = '2D' 
RenderView1.CenterOfRotation = [63.5, 63.5, 0.0] 

a1_Voltage_PVLookupTable.ScalarOpacityFunction = a1_Voltage_PiecewiseFunction 

RenderView1.CameraPosition = [63.5, 63.5, 346.97045256124744] 
RenderView1.CameraClippingRange = [343.50074803563496, 352.1750093496662] 
RenderView1.CameraParallelScale = 89.80256121069154 

newmapname = '' 
fragments = mapname.split('.') 
for fragment in fragments: 
newmapname +=fragment 

WriteAnimation('{0}/{1}.avi'.format(vtkfolder, newmapname), Magnification=1, Quality=2, FrameRate=15.000000) 

Render() 

print('{0} has been produced.'.format(newmapname)) 

if __name__ == '__main__': 
main() 

Would any one of you be able to point out to me what I am doing wrong? Or which objects/structures I should delete (in the archives the problem always appeared to be that a writer needed to be deleted, but I don't have such an object) such that I basically start over each time I create a new movie and hence don't use that much memory? 

With kind regards, 
Tim 
_______________________________________________ 
Powered by www.kitware.com 

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html 

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView 

Search the list archives at: http://markmail.org/search/?q=ParaView 

Follow this link to subscribe/unsubscribe: 
http://public.kitware.com/mailman/listinfo/paraview 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20160119/d16d2f53/attachment.html>