[Paraview] Memory leak with Catalyst

Tue May 24 17:25:06 EDT 2016

Guys,

The multiblock writer issue is unrelated -- as far as I can tell. Esp
since "If I take the image rendering out of my pipeline (comment out
the RegisterView call) and leave the VTK file writing, it runs fine
and doesn't report any leaks.".

Here's the MR if you're interested in checking it out in any case:
https://gitlab.kitware.com/vtk/vtk/merge_requests/1393

Utkarsh

On Mon, May 23, 2016 at 4:13 PM, Gallagher, Timothy P
<tim.gallagher at gatech.edu> wrote:
> I am using the multiblock format, so I am definitely interested in the
> solution. Hopefully Utkarsh sees this and can chime in.
>
>
> Thanks!
>
>
> Tim
>
>
>
> ________________________________
> From: Burlen Loring <burlen.loring at gmail.com>
> Sent: Monday, May 23, 2016 3:30 PM
>
> To: Gallagher, Timothy P; Andy Bauer
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Memory leak with Catalyst
>
>
> I think there is an api in the multi-block writer to disable writing
> metadata. Utkarsh knows, he's the one that figured it out. not sure how it
> applies to your use case. you may not even be using multi-block format...
>
>
> On 05/23/2016 06:57 AM, Gallagher, Timothy P wrote:
>
> Interesting -- I'd be willing to try it out if you have more details (commit
> hashtag, ticket, etc.), even if I have to figure out how to backport it from
> a newer version.
>
>
> Thanks,
>
>
> Tim
>
>
>
> ________________________________
> From: Burlen Loring <burlen.loring at gmail.com>
> Sent: Sunday, May 22, 2016 10:30 PM
> To: Gallagher, Timothy P; Andy Bauer
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Memory leak with Catalyst
>
> btw, we just had similar problem that was entirely explained by vtk writer
> attempting to gather a bunch of arrays to the root node. Utkarsh found a way
> to disable that behavior. I wonder if you are hitting the same.
>
> On 5/20/2016 4:36 PM, Gallagher, Timothy P wrote:
>
> Well... not going so well.
>
>
> If I run a small simulation with massif and the Release build of paraview, I
> don't see any growing memory in time. I also get my VTK files and images.
>
>
> If I run the same simulation linked against a Debug build with
> VTK_DEBUG_LEAKS on, there is a segfault inside one of the libvtkRendering
> libraries (it says libvtkRenderingOp but the rest of the name is cut off).
> If I take the image rendering out of my pipeline (comment out the
> RegisterView call) and leave the VTK file writing, it runs fine and doesn't
> report any leaks. I tried to get the segfault to drop a core file, but it
> won't do it.
>
>
> I'm thinking that there is an issue in the rendering part causing the leak.
> I have no idea what it could be though -- this is the 4.4.0 version, is
> anybody aware of any bug fixes between 4.4.0 and current versions that might
> be relevant?
>
>
> Thanks for the help,
>
>
> Tim
>
>
>
>
>
> ________________________________
> From: ParaView <paraview-bounces at paraview.org> on behalf of Gallagher,
> Timothy P <tim.gallagher at gatech.edu>
> Sent: Friday, May 20, 2016 4:17 PM
> To: Burlen Loring; Andy Bauer
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Memory leak with Catalyst
>
>
> Thanks for the advice Burlen -- I haven't used Massif before, so this is a
> good opportunity to give it a try.
>
>
> The initialize/finalize each time approach doesn't work either. I am calling
> the Finalize() routine on the vtkCPProcessor class but then when it tries to
> initialize again, it throws:
>
>
> Warning: In /data4/ParaView/VTK/Parallel/Core/vtkSocketController.cxx, line
> 50
> vtkSocketController (0xa4342f0): Already initialized.
>
>
> and then crashes. So, hopefully I can sort out the memory leak rather than
> trying to debug why a hacked fix doesn't work!
>
>
> Tim
>
>
>
> ________________________________
> From: Burlen Loring <burlen.loring at gmail.com>
> Sent: Friday, May 20, 2016 3:46 PM
> To: Gallagher, Timothy P; Andy Bauer
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Memory leak with Catalyst
>
>
> VTK_DEBUG_LEAKS, although will catch some serious errors, will not catch all
> kinds of leaks. For example you can have leaks where data erroneously
> accumulates with each time step, but is deleted at program termination.
> VTK_DEBUG_LEAKS will not catch that kind of error. It's probably better to
> use massif to profile your code on a small one node run. There's an
> excellent tool called massif visualizer to aid in exploring the data
> generated.
>
>
> $0.02
>
>
> On 05/20/2016 11:56 AM, Gallagher, Timothy P wrote:
>
> Hi Andy,
>
>
> Thanks for the tips. I will get working on the VTK_DEBUG_LEAKS now and see
> what it turns up.
>
>
> The initialize/finalize every time is definitely a hack and not for
> long-term use. But, sponsors want a report on Monday and we need to be able
> to visualize things for that -- the simulation is too big to write data
> files and post-process later. So I modified the code to do that for now
> until I can find a proper fix.
>
>
> I'll let you know if I get stuck with the log file.
>
>
> Thanks again,
>
>
> Tim
>
>
>
> ________________________________
> From: Andy Bauer <andy.bauer at kitware.com>
> Sent: Friday, May 20, 2016 2:39 PM
> To: Gallagher, Timothy P
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Memory leak with Catalyst
>
> Hi Tim,
>
> If you build Catalyst with VTK_DEBUG_LEAKS enabled it is pretty good at
> finding VTK objects that aren't deleted properly. You should be able to run
> this with a small amount of calls to Catalyst as well. If you try this and
> want help understanding the output (if an object like a vtkUnstructuredGrid
> is leaked it will often give a whole bunch of false leaks that the
> unstructured grid is just holding the references to), just share your output
> file and I can take a look at it to try and narrow down the culprit. That
> may be slightly easier to use than Valgrind.
>
> Beyond this, you could maybe try the same run but without doing any Catalyst
> work to see what happens then. That may be a lot of compute cycles but I'm
> not sure how else to try and bisect where the memory leak is occurring.
>
> Initializing and finalizing Catalyst every time you want output would
> probably work but I'd think there may be significant overhead doing it like
> this. Plus, it's not really solving the problem -- just avoiding it.
>
> Best,
> Andy
>
>
> On Fri, May 20, 2016 at 12:57 PM, Gallagher, Timothy P
> <tim.gallagher at gatech.edu> wrote:
>>
>> Hi,
>>
>>
>> One of our users is running a very big simulation and writing out images
>> of two slices (two different views) every 1000 iterations and writing out
>> the data for the two slices (two different data writers) as VTK files every
>> 5000 iterations. It is using Paraview 4.4.
>>
>>
>> After 21000 iterations, the simulation is killed because the memory on the
>> compute nodes fills up. I usually know how to track down memory problems in
>> our code using valgrind and related tools, but is that the right way to go
>> to try and find this problem?
>>
>>
>> Are there any tips on how to isolate where the problem may be? I don't
>> know if it is in the adapter, or in paraview itself. Has anybody encountered
>> problems with runaway memory using Catalyst in 4.4 when writing images or
>> VTK files?
>>
>>
>> I know when we use pvpython to generate images and loop over many files,
>> sometimes the memory also blows up and so we usually move the loop over the
>> files outside the pvpython script and into a driver script that executes a
>> new pvpython for each file. Is there a way to shut down/start up Catalyst
>> each time it needs to write something? Is that advisable?
>>
>>
>> Thanks,
>>
>>
>> Tim
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/paraview
>>
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
>
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>