[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

Tue Mar 20 22:18:22 EDT 2018

Totally, totally agree.  Lets go with the programmable filter solution.

From: Cory Quammen [mailto:cory.quammen at kitware.com]
Sent: Tuesday, March 20, 2018 7:27 PM
To: Bauer, Andy (External Contacts) <andy.bauer at kitware.com>
Cc: Scott, W Alan <wascott at sandia.gov>; paraview-developers at paraview.org; Patchett, John M. (LANL) <patchett at lanl.gov>
Subject: Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>> wrote:
Well, I could probably have this done by tomorrow but in the end I leave the decision on whether we wait on the release for this change to Utkarsh and Cory. I'm not sure how much of a wrench it would put into their plans.

In my judgement, it's too late to put this in 5.5. It may seem like a simple change, but experience shows time and again that simple changes do not exist :-) We do not have sufficient time to test it and get feedback prior to the release.

On the other hand, if this is really the specific use case then a Python Programmable Filter could just as easily do the trick and would be nearly negligible cost. In fact, I'll put together the a python script that does that and send it out shortly.

Sounds like a good plan.

Thanks,
Cory

On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov<mailto:wascott at sandia.gov>> wrote:
Andy,
I assume we are way too late in the release process to implement this for 5.5.0.

John P., should this be implemented early next release, so you can build an alpha/ master build?  Or will you just wait for 5.6.0, out early fall, which should make this question irrelevant?

Alan

From: Andy Bauer [mailto:andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>]
Sent: Tuesday, March 20, 2018 6:35 PM
To: Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov<mailto:boonth at lanl.gov>>
Cc: Ayachit, Utkarsh (External Contacts) <utkarsh.ayachit at kitware.com<mailto:utkarsh.ayachit at kitware.com>>; Scott, W Alan <wascott at sandia.gov<mailto:wascott at sandia.gov>>; paraview-developers at paraview.org<mailto:paraview-developers at paraview.org>; Patchett, John M. (LANL) <patchett at lanl.gov<mailto:patchett at lanl.gov>>
Subject: Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

Going back to the Merge Blocks filter, I'd say that if the input grid is a vtkUnstructuredGrid and there's only a single block on that process in this case the Merge Blocks filter should do a shallow copy and exit quickly, which it doesn't do. This isn't a silver bullet fix and probably won't matter after the composite dataset rework but for what LANL wants in this case, this would be a very simple and efficient solution.

On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <boonth at lanl.gov<mailto:boonth at lanl.gov>> wrote:
Unless I am missing something, no reason. Seems to me that's just the
way it was written. I can't see why the filter can't request 1 extra
ghost-level from its input. (cc, Berk).
Because we start with cell data, we actually require two layers of ghost cells. The ghost cells filter was developed to allocate as many layers of ghost cells as needed for a distributed dataset.
Any other ideas how to decrease the bloat of about 4 GB of data to be about
18 GB of data?
Is this for a Catalyst or pvbatch pipeline or for an interactive session?
It's for an interactive session. We have a user running out of memory trying to use a certain pipeline. More information is in the issue at https://gitlab.kitware.com/paraview/paraview/issues/18035

Boonth

On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
Why does merge blocks grow so much?  Why can’t this filter just do shallow
copies?  Is this impossible, or just not implemented?
No, merge blocks is not a shallow copy, it needs to combine mutliple
vtkDataSet's into a single vtkDataSet merging all points into a single
array, same for cells, other attribute arrays etc. Merge Blocks should
indeed be considered as a workaround for issue when handling
mutliblock datasets in the pipeline and not the go-to solution. There
is work currently underway that will revamp multiblock dataset support
in VTK/ParaView which should ultimately make Merge Blocks obsolete.

Looking at your pipeline, looks like you're doing MergeBlocks so that
you can run the Ghost Cells generator filter. What is the nature of
your data? Is it truly a multiblock or just has 1 block in it on each
rank? If its just 1 block, what is the type of the block? If it's a
unstructured grid, it will trivial to create a filter that just passes
the input block out as unstructured grid without duplicating.
Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
implemented?
It's impossible. See previous comment for justification based on how
things are currently.
(I think I know this one, but here it is).  Why does the ghost cell
generator bloat memory so much?  Can’t you just use shallow copies of the
primary data, and then add the ghost cells?
adding ghost cell etc, means adding new elements to existing arrays.
since vtk doesn't modify input arrays, when it needs to add new
elements, it has to create a deep-copy and then add new elements to
it.
Why can’t we have the cell data to point data filter be smart enough to add
ghost cells if necessary for calculations, then throw them away?
Unless I am missing something, no reason. Seems to me that's just the
way it was written. I can't see why the filter can't request 1 extra
ghost-level from its input. (cc, Berk).
Any other ideas how to decrease the bloat of about 4 GB of data to be about
18 GB of data?
Is this for a Catalyst or pvbatch pipeline or for an interactive session?

Utkarsh
_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=Paraview-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/paraview-developers

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=Paraview-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/paraview-developers

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=Paraview-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/paraview-developers

--
Cory Quammen
Staff R&D Engineer
Kitware, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180321/82d444c7/attachment-0001.html>