[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

[Andy Bauer]

Well, I could probably have this done by tomorrow but in the end I leave
the decision on whether we wait on the release for this change to Utkarsh
and Cory. I'm not sure how much of a wrench it would put into their plans.

On the other hand, if this is really the specific use case then a Python
Programmable Filter could just as easily do the trick and would be nearly
negligible cost. In fact, I'll put together the a python script that does
that and send it out shortly.

On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov> wrote:

> Andy,
>
> I assume we are way too late in the release process to implement this for
> 5.5.0.
>
>
>
> John P., should this be implemented early next release, so you can build
> an alpha/ master build?  Or will you just wait for 5.6.0, out early fall,
> which should make this question irrelevant?
>
>
>
> Alan
>
>
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Tuesday, March 20, 2018 6:35 PM
> *To:* Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov>
> *Cc:* Ayachit, Utkarsh (External Contacts) <utkarsh.ayachit at kitware.com>;
> Scott, W Alan <wascott at sandia.gov>; paraview-developers at paraview.org;
> Patchett, John M. (LANL) <patchett at lanl.gov>
> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory
> consumption on Paraview (#18035)
>
>
>
> Going back to the Merge Blocks filter, I'd say that if the input grid is a
> vtkUnstructuredGrid and there's only a single block on that process in this
> case the Merge Blocks filter should do a shallow copy and exit quickly,
> which it doesn't do. This isn't a silver bullet fix and probably won't
> matter after the composite dataset rework but for what LANL wants in this
> case, this would be a very simple and efficient solution.
>
>
>
> On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <boonth at lanl.gov>
> wrote:
>
> Unless I am missing something, no reason. Seems to me that's just the
> way it was written. I can't see why the filter can't request 1 extra
> ghost-level from its input. (cc, Berk).
>
> Because we start with cell data, we actually require two layers of ghost
> cells. The ghost cells filter was developed to allocate as many layers of
> ghost cells as needed for a distributed dataset.
>
> Any other ideas how to decrease the bloat of about 4 GB of data to be about
> 18 GB of data?
> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>
> It's for an interactive session. We have a user running out of memory
> trying to use a certain pipeline. More information is in the issue at
> https://gitlab.kitware.com/paraview/paraview/issues/18035
>
> Boonth
>
>
>
>
> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>
> Why does merge blocks grow so much?  Why can’t this filter just do shallow
> copies?  Is this impossible, or just not implemented?
>
> No, merge blocks is not a shallow copy, it needs to combine mutliple
> vtkDataSet's into a single vtkDataSet merging all points into a single
> array, same for cells, other attribute arrays etc. Merge Blocks should
> indeed be considered as a workaround for issue when handling
> mutliblock datasets in the pipeline and not the go-to solution. There
> is work currently underway that will revamp multiblock dataset support
> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>
> Looking at your pipeline, looks like you're doing MergeBlocks so that
> you can run the Ghost Cells generator filter. What is the nature of
> your data? Is it truly a multiblock or just has 1 block in it on each
> rank? If its just 1 block, what is the type of the block? If it's a
> unstructured grid, it will trivial to create a filter that just passes
> the input block out as unstructured grid without duplicating.
>
> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
> implemented?
>
> It's impossible. See previous comment for justification based on how
> things are currently.
>
> (I think I know this one, but here it is).  Why does the ghost cell
> generator bloat memory so much?  Can’t you just use shallow copies of the
> primary data, and then add the ghost cells?
>
> adding ghost cell etc, means adding new elements to existing arrays.
> since vtk doesn't modify input arrays, when it needs to add new
> elements, it has to create a deep-copy and then add new elements to
> it.
>
> Why can’t we have the cell data to point data filter be smart enough to add
> ghost cells if necessary for calculations, then throw them away?
>
> Unless I am missing something, no reason. Seems to me that's just the
> way it was written. I can't see why the filter can't request 1 extra
> ghost-level from its input. (cc, Berk).
>
> Any other ideas how to decrease the bloat of about 4 GB of data to be about
> 18 GB of data?
>
> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>
> Utkarsh
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=
> Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=
> Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180320/cd85970a/attachment.html>