[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

Wed Mar 21 17:49:59 EDT 2018

FYI: there's a merge request for doing the shallow copy in merge blocks for
the situation when there's only a single, unstructured grid --
https://gitlab.kitware.com/vtk/vtk/merge_requests/4103. Utkarsh thought
that it may be able to make it into 5.5 since it's quite a small change.

On Wed, Mar 21, 2018 at 3:15 PM, Andy Bauer <andy.bauer at kitware.com> wrote:

>
>
> On Wed, Mar 21, 2018 at 3:06 PM, Boonthanome Nouanesengsy <boonth at lanl.gov
> > wrote:
>
>> Boonth or John, would you mind shooting me an email that you got this
>> email. I'm worried that a spambot may not let this get through based on the
>> attachments.
>>
>> Yes, got the email.
>>
>> Going back to the Merge Blocks filter, I'd say that if the input grid is
>> a vtkUnstructuredGrid and there's only a single block on that process in
>> this case the Merge Blocks filter should do a shallow copy and exit
>> quickly, which it doesn't do. This isn't a silver bullet fix and probably
>> won't matter after the composite dataset rework but for what LANL wants in
>> this case, this would be a very simple and efficient solution.
>>
>> The problem is that in many use cases the number of processes used for
>> writing the data will be different from the number of processes used for
>> visualization. In this particular use case it won't help, but thanks for
>> the programmable filter. It should be helpful in the future.
>>
>
> If multiple blocks per MPI rank is the case then there's currently no way
> around the memory consumption in the merge blocks filter. In the past there
> were discussions about "cleaning out a pipeline" where the thought was that
> a user could essentially have filters that they don't care about the output
> for any more to release the data from. This would mean that if data was
> needed again from that filter (e.g. the filter after it wanted to
> re-execute) though it would have to be constructed from scratch. I think
> VisIt does this to some extent but when you're going through the data
> multiple times and requesting different things it will bog down due to all
> the reconstructing it's doing. I'd think that this would be possible in PV
> but would require a significant change and would probably be quite
> confusing to users to enable them to do things efficiently.
>
>
>
>
>>
>> Boonth
>>
>>
>>
>> On 03/20/2018 08:18 PM, Scott, W Alan wrote:
>>
>> Totally, totally agree.  Lets go with the programmable filter solution.
>>
>>
>>
>> *From:* Cory Quammen [mailto:cory.quammen at kitware.com
>> <cory.quammen at kitware.com>]
>> *Sent:* Tuesday, March 20, 2018 7:27 PM
>> *To:* Bauer, Andy (External Contacts) <andy.bauer at kitware.com>
>> <andy.bauer at kitware.com>
>> *Cc:* Scott, W Alan <wascott at sandia.gov> <wascott at sandia.gov>;
>> paraview-developers at paraview.org; Patchett, John M. (LANL)
>> <patchett at lanl.gov> <patchett at lanl.gov>
>> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
>> Memory consumption on Paraview (#18035)
>>
>>
>>
>> On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <andy.bauer at kitware.com>
>> wrote:
>>
>> Well, I could probably have this done by tomorrow but in the end I leave
>> the decision on whether we wait on the release for this change to Utkarsh
>> and Cory. I'm not sure how much of a wrench it would put into their plans.
>>
>>
>>
>> In my judgement, it's too late to put this in 5.5. It may seem like a
>> simple change, but experience shows time and again that simple changes do
>> not exist :-) We do not have sufficient time to test it and get feedback
>> prior to the release.
>>
>>
>>
>> On the other hand, if this is really the specific use case then a Python
>> Programmable Filter could just as easily do the trick and would be nearly
>> negligible cost. In fact, I'll put together the a python script that does
>> that and send it out shortly.
>>
>>
>>
>> Sounds like a good plan.
>>
>>
>>
>> Thanks,
>>
>> Cory
>>
>>
>>
>>
>>
>> On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov>
>> wrote:
>>
>> Andy,
>>
>> I assume we are way too late in the release process to implement this for
>> 5.5.0.
>>
>>
>>
>> John P., should this be implemented early next release, so you can build
>> an alpha/ master build?  Or will you just wait for 5.6.0, out early fall,
>> which should make this question irrelevant?
>>
>>
>>
>> Alan
>>
>>
>>
>> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
>> *Sent:* Tuesday, March 20, 2018 6:35 PM
>> *To:* Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov>
>> *Cc:* Ayachit, Utkarsh (External Contacts) <utkarsh.ayachit at kitware.com>;
>> Scott, W Alan <wascott at sandia.gov>; paraview-developers at paraview.org;
>> Patchett, John M. (LANL) <patchett at lanl.gov>
>> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
>> Memory consumption on Paraview (#18035)
>>
>>
>>
>> Going back to the Merge Blocks filter, I'd say that if the input grid is
>> a vtkUnstructuredGrid and there's only a single block on that process in
>> this case the Merge Blocks filter should do a shallow copy and exit
>> quickly, which it doesn't do. This isn't a silver bullet fix and probably
>> won't matter after the composite dataset rework but for what LANL wants in
>> this case, this would be a very simple and efficient solution.
>>
>>
>>
>> On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <
>> boonth at lanl.gov> wrote:
>>
>> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
>> Because we start with cell data, we actually require two layers of ghost
>> cells. The ghost cells filter was developed to allocate as many layers of
>> ghost cells as needed for a distributed dataset.
>>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>> about
>> 18 GB of data?
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
>> It's for an interactive session. We have a user running out of memory
>> trying to use a certain pipeline. More information is in the issue at
>> https://gitlab.kitware.com/paraview/paraview/issues/18035
>>
>> Boonth
>>
>>
>>
>>
>> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>>
>> Why does merge blocks grow so much?  Why can’t this filter just do shallow
>> copies?  Is this impossible, or just not implemented?
>>
>> No, merge blocks is not a shallow copy, it needs to combine mutliple
>> vtkDataSet's into a single vtkDataSet merging all points into a single
>> array, same for cells, other attribute arrays etc. Merge Blocks should
>> indeed be considered as a workaround for issue when handling
>> mutliblock datasets in the pipeline and not the go-to solution. There
>> is work currently underway that will revamp multiblock dataset support
>> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>>
>> Looking at your pipeline, looks like you're doing MergeBlocks so that
>> you can run the Ghost Cells generator filter. What is the nature of
>> your data? Is it truly a multiblock or just has 1 block in it on each
>> rank? If its just 1 block, what is the type of the block? If it's a
>> unstructured grid, it will trivial to create a filter that just passes
>> the input block out as unstructured grid without duplicating.
>>
>> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
>> implemented?
>>
>> It's impossible. See previous comment for justification based on how
>> things are currently.
>>
>> (I think I know this one, but here it is).  Why does the ghost cell
>> generator bloat memory so much?  Can’t you just use shallow copies of the
>> primary data, and then add the ghost cells?
>>
>> adding ghost cell etc, means adding new elements to existing arrays.
>> since vtk doesn't modify input arrays, when it needs to add new
>> elements, it has to create a deep-copy and then add new elements to
>> it.
>>
>> Why can’t we have the cell data to point data filter be smart enough to
>> add
>> ghost cells if necessary for calculations, then throw them away?
>>
>> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>> about
>> 18 GB of data?
>>
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
>> Utkarsh
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>>
>>
>>
>> --
>>
>> Cory Quammen
>> Staff R&D Engineer
>> Kitware, Inc.
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180321/42474ecd/attachment-0001.html>