[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

Mon Mar 26 15:25:27 EDT 2018

FYI: This has made it into the PV 5.5 release branch.

On Wed, Mar 21, 2018 at 5:49 PM, Andy Bauer <andy.bauer at kitware.com> wrote:

> FYI: there's a merge request for doing the shallow copy in merge blocks
> for the situation when there's only a single, unstructured grid --
> https://gitlab.kitware.com/vtk/vtk/merge_requests/4103. Utkarsh thought
> that it may be able to make it into 5.5 since it's quite a small change.
>
> On Wed, Mar 21, 2018 at 3:15 PM, Andy Bauer <andy.bauer at kitware.com>
> wrote:
>
>>
>>
>> On Wed, Mar 21, 2018 at 3:06 PM, Boonthanome Nouanesengsy <
>> boonth at lanl.gov> wrote:
>>
>>> Boonth or John, would you mind shooting me an email that you got this
>>> email. I'm worried that a spambot may not let this get through based on the
>>> attachments.
>>>
>>> Yes, got the email.
>>>
>>> Going back to the Merge Blocks filter, I'd say that if the input grid is
>>> a vtkUnstructuredGrid and there's only a single block on that process in
>>> this case the Merge Blocks filter should do a shallow copy and exit
>>> quickly, which it doesn't do. This isn't a silver bullet fix and probably
>>> won't matter after the composite dataset rework but for what LANL wants in
>>> this case, this would be a very simple and efficient solution.
>>>
>>> The problem is that in many use cases the number of processes used for
>>> writing the data will be different from the number of processes used for
>>> visualization. In this particular use case it won't help, but thanks for
>>> the programmable filter. It should be helpful in the future.
>>>
>>
>> If multiple blocks per MPI rank is the case then there's currently no way
>> around the memory consumption in the merge blocks filter. In the past there
>> were discussions about "cleaning out a pipeline" where the thought was that
>> a user could essentially have filters that they don't care about the output
>> for any more to release the data from. This would mean that if data was
>> needed again from that filter (e.g. the filter after it wanted to
>> re-execute) though it would have to be constructed from scratch. I think
>> VisIt does this to some extent but when you're going through the data
>> multiple times and requesting different things it will bog down due to all
>> the reconstructing it's doing. I'd think that this would be possible in PV
>> but would require a significant change and would probably be quite
>> confusing to users to enable them to do things efficiently.
>>
>>
>>
>>
>>>
>>> Boonth
>>>
>>>
>>>
>>> On 03/20/2018 08:18 PM, Scott, W Alan wrote:
>>>
>>> Totally, totally agree.  Lets go with the programmable filter solution.
>>>
>>>
>>>
>>> *From:* Cory Quammen [mailto:cory.quammen at kitware.com
>>> <cory.quammen at kitware.com>]
>>> *Sent:* Tuesday, March 20, 2018 7:27 PM
>>> *To:* Bauer, Andy (External Contacts) <andy.bauer at kitware.com>
>>> <andy.bauer at kitware.com>
>>> *Cc:* Scott, W Alan <wascott at sandia.gov> <wascott at sandia.gov>;
>>> paraview-developers at paraview.org; Patchett, John M. (LANL)
>>> <patchett at lanl.gov> <patchett at lanl.gov>
>>> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
>>> Memory consumption on Paraview (#18035)
>>>
>>>
>>>
>>> On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <andy.bauer at kitware.com>
>>> wrote:
>>>
>>> Well, I could probably have this done by tomorrow but in the end I leave
>>> the decision on whether we wait on the release for this change to Utkarsh
>>> and Cory. I'm not sure how much of a wrench it would put into their plans.
>>>
>>>
>>>
>>> In my judgement, it's too late to put this in 5.5. It may seem like a
>>> simple change, but experience shows time and again that simple changes do
>>> not exist :-) We do not have sufficient time to test it and get feedback
>>> prior to the release.
>>>
>>>
>>>
>>> On the other hand, if this is really the specific use case then a Python
>>> Programmable Filter could just as easily do the trick and would be nearly
>>> negligible cost. In fact, I'll put together the a python script that does
>>> that and send it out shortly.
>>>
>>>
>>>
>>> Sounds like a good plan.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Cory
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov>
>>> wrote:
>>>
>>> Andy,
>>>
>>> I assume we are way too late in the release process to implement this
>>> for 5.5.0.
>>>
>>>
>>>
>>> John P., should this be implemented early next release, so you can build
>>> an alpha/ master build?  Or will you just wait for 5.6.0, out early fall,
>>> which should make this question irrelevant?
>>>
>>>
>>>
>>> Alan
>>>
>>>
>>>
>>> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
>>> *Sent:* Tuesday, March 20, 2018 6:35 PM
>>> *To:* Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov>
>>> *Cc:* Ayachit, Utkarsh (External Contacts) <utkarsh.ayachit at kitware.com>;
>>> Scott, W Alan <wascott at sandia.gov>; paraview-developers at paraview.org;
>>> Patchett, John M. (LANL) <patchett at lanl.gov>
>>> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
>>> Memory consumption on Paraview (#18035)
>>>
>>>
>>>
>>> Going back to the Merge Blocks filter, I'd say that if the input grid is
>>> a vtkUnstructuredGrid and there's only a single block on that process in
>>> this case the Merge Blocks filter should do a shallow copy and exit
>>> quickly, which it doesn't do. This isn't a silver bullet fix and probably
>>> won't matter after the composite dataset rework but for what LANL wants in
>>> this case, this would be a very simple and efficient solution.
>>>
>>>
>>>
>>> On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <
>>> boonth at lanl.gov> wrote:
>>>
>>> Unless I am missing something, no reason. Seems to me that's just the
>>> way it was written. I can't see why the filter can't request 1 extra
>>> ghost-level from its input. (cc, Berk).
>>>
>>> Because we start with cell data, we actually require two layers of ghost
>>> cells. The ghost cells filter was developed to allocate as many layers of
>>> ghost cells as needed for a distributed dataset.
>>>
>>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>>> about
>>> 18 GB of data?
>>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>>
>>> It's for an interactive session. We have a user running out of memory
>>> trying to use a certain pipeline. More information is in the issue at
>>> https://gitlab.kitware.com/paraview/paraview/issues/18035
>>>
>>> Boonth
>>>
>>>
>>>
>>>
>>> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>>>
>>> Why does merge blocks grow so much?  Why can’t this filter just do
>>> shallow
>>> copies?  Is this impossible, or just not implemented?
>>>
>>> No, merge blocks is not a shallow copy, it needs to combine mutliple
>>> vtkDataSet's into a single vtkDataSet merging all points into a single
>>> array, same for cells, other attribute arrays etc. Merge Blocks should
>>> indeed be considered as a workaround for issue when handling
>>> mutliblock datasets in the pipeline and not the go-to solution. There
>>> is work currently underway that will revamp multiblock dataset support
>>> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>>>
>>> Looking at your pipeline, looks like you're doing MergeBlocks so that
>>> you can run the Ghost Cells generator filter. What is the nature of
>>> your data? Is it truly a multiblock or just has 1 block in it on each
>>> rank? If its just 1 block, what is the type of the block? If it's a
>>> unstructured grid, it will trivial to create a filter that just passes
>>> the input block out as unstructured grid without duplicating.
>>>
>>> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just
>>> not
>>> implemented?
>>>
>>> It's impossible. See previous comment for justification based on how
>>> things are currently.
>>>
>>> (I think I know this one, but here it is).  Why does the ghost cell
>>> generator bloat memory so much?  Can’t you just use shallow copies of the
>>> primary data, and then add the ghost cells?
>>>
>>> adding ghost cell etc, means adding new elements to existing arrays.
>>> since vtk doesn't modify input arrays, when it needs to add new
>>> elements, it has to create a deep-copy and then add new elements to
>>> it.
>>>
>>> Why can’t we have the cell data to point data filter be smart enough to
>>> add
>>> ghost cells if necessary for calculations, then throw them away?
>>>
>>> Unless I am missing something, no reason. Seems to me that's just the
>>> way it was written. I can't see why the filter can't request 1 extra
>>> ghost-level from its input. (cc, Berk).
>>>
>>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>>> about
>>> 18 GB of data?
>>>
>>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>>
>>> Utkarsh
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Search the list archives at: http://markmail.org/search/?q=
>>> Paraview-developers
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Search the list archives at: http://markmail.org/search/?q=
>>> Paraview-developers
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Search the list archives at: http://markmail.org/search/?q=
>>> Paraview-developers
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Cory Quammen
>>> Staff R&D Engineer
>>> Kitware, Inc.
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Search the list archives at: http://markmail.org/search/?q=Paraview-developers
>>>
>>> Follow this link to subscribe/unsubscribe:https://public.kitware.com/mailman/listinfo/paraview-developers
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180326/4ca3315e/attachment-0001.html>