[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

Tue Mar 20 21:27:18 EDT 2018

On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <andy.bauer at kitware.com> wrote:

> Well, I could probably have this done by tomorrow but in the end I leave
> the decision on whether we wait on the release for this change to Utkarsh
> and Cory. I'm not sure how much of a wrench it would put into their plans.
>

In my judgement, it's too late to put this in 5.5. It may seem like a
simple change, but experience shows time and again that simple changes do
not exist :-) We do not have sufficient time to test it and get feedback
prior to the release.

> On the other hand, if this is really the specific use case then a Python
> Programmable Filter could just as easily do the trick and would be nearly
> negligible cost. In fact, I'll put together the a python script that does
> that and send it out shortly.
>

Sounds like a good plan.

Thanks,
Cory

> On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov> wrote:
>
>> Andy,
>>
>> I assume we are way too late in the release process to implement this for
>> 5.5.0.
>>
>>
>>
>> John P., should this be implemented early next release, so you can build
>> an alpha/ master build?  Or will you just wait for 5.6.0, out early fall,
>> which should make this question irrelevant?
>>
>>
>>
>> Alan
>>
>>
>>
>> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
>> *Sent:* Tuesday, March 20, 2018 6:35 PM
>> *To:* Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov>
>> *Cc:* Ayachit, Utkarsh (External Contacts) <utkarsh.ayachit at kitware.com>;
>> Scott, W Alan <wascott at sandia.gov>; paraview-developers at paraview.org;
>> Patchett, John M. (LANL) <patchett at lanl.gov>
>> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
>> Memory consumption on Paraview (#18035)
>>
>>
>>
>> Going back to the Merge Blocks filter, I'd say that if the input grid is
>> a vtkUnstructuredGrid and there's only a single block on that process in
>> this case the Merge Blocks filter should do a shallow copy and exit
>> quickly, which it doesn't do. This isn't a silver bullet fix and probably
>> won't matter after the composite dataset rework but for what LANL wants in
>> this case, this would be a very simple and efficient solution.
>>
>>
>>
>> On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <
>> boonth at lanl.gov> wrote:
>>
>> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
>> Because we start with cell data, we actually require two layers of ghost
>> cells. The ghost cells filter was developed to allocate as many layers of
>> ghost cells as needed for a distributed dataset.
>>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>> about
>> 18 GB of data?
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
>> It's for an interactive session. We have a user running out of memory
>> trying to use a certain pipeline. More information is in the issue at
>> https://gitlab.kitware.com/paraview/paraview/issues/18035
>>
>> Boonth
>>
>>
>>
>>
>> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>>
>> Why does merge blocks grow so much?  Why can’t this filter just do shallow
>> copies?  Is this impossible, or just not implemented?
>>
>> No, merge blocks is not a shallow copy, it needs to combine mutliple
>> vtkDataSet's into a single vtkDataSet merging all points into a single
>> array, same for cells, other attribute arrays etc. Merge Blocks should
>> indeed be considered as a workaround for issue when handling
>> mutliblock datasets in the pipeline and not the go-to solution. There
>> is work currently underway that will revamp multiblock dataset support
>> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>>
>> Looking at your pipeline, looks like you're doing MergeBlocks so that
>> you can run the Ghost Cells generator filter. What is the nature of
>> your data? Is it truly a multiblock or just has 1 block in it on each
>> rank? If its just 1 block, what is the type of the block? If it's a
>> unstructured grid, it will trivial to create a filter that just passes
>> the input block out as unstructured grid without duplicating.
>>
>> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
>> implemented?
>>
>> It's impossible. See previous comment for justification based on how
>> things are currently.
>>
>> (I think I know this one, but here it is).  Why does the ghost cell
>> generator bloat memory so much?  Can’t you just use shallow copies of the
>> primary data, and then add the ghost cells?
>>
>> adding ghost cell etc, means adding new elements to existing arrays.
>> since vtk doesn't modify input arrays, when it needs to add new
>> elements, it has to create a deep-copy and then add new elements to
>> it.
>>
>> Why can’t we have the cell data to point data filter be smart enough to
>> add
>> ghost cells if necessary for calculations, then throw them away?
>>
>> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>> about
>> 18 GB of data?
>>
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
>> Utkarsh
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>>
>>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=
> Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>

-- 
Cory Quammen
Staff R&D Engineer
Kitware, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180320/fa22f9b0/attachment.html>