[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

[Andy Bauer]

Going back to the Merge Blocks filter, I'd say that if the input grid is a
vtkUnstructuredGrid and there's only a single block on that process in this
case the Merge Blocks filter should do a shallow copy and exit quickly,
which it doesn't do. This isn't a silver bullet fix and probably won't
matter after the composite dataset rework but for what LANL wants in this
case, this would be a very simple and efficient solution.

On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <boonth at lanl.gov>
wrote:

> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
> Because we start with cell data, we actually require two layers of ghost
> cells. The ghost cells filter was developed to allocate as many layers of
> ghost cells as needed for a distributed dataset.
>
> Any other ideas how to decrease the bloat of about 4 GB of data to be about
>> 18 GB of data?
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
> It's for an interactive session. We have a user running out of memory
> trying to use a certain pipeline. More information is in the issue at
> https://gitlab.kitware.com/paraview/paraview/issues/18035
>
> Boonth
>
>
>
> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>
>> Why does merge blocks grow so much?  Why can’t this filter just do shallow
>>> copies?  Is this impossible, or just not implemented?
>>>
>> No, merge blocks is not a shallow copy, it needs to combine mutliple
>> vtkDataSet's into a single vtkDataSet merging all points into a single
>> array, same for cells, other attribute arrays etc. Merge Blocks should
>> indeed be considered as a workaround for issue when handling
>> mutliblock datasets in the pipeline and not the go-to solution. There
>> is work currently underway that will revamp multiblock dataset support
>> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>>
>> Looking at your pipeline, looks like you're doing MergeBlocks so that
>> you can run the Ghost Cells generator filter. What is the nature of
>> your data? Is it truly a multiblock or just has 1 block in it on each
>> rank? If its just 1 block, what is the type of the block? If it's a
>> unstructured grid, it will trivial to create a filter that just passes
>> the input block out as unstructured grid without duplicating.
>>
>> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
>>> implemented?
>>>
>> It's impossible. See previous comment for justification based on how
>> things are currently.
>>
>> (I think I know this one, but here it is).  Why does the ghost cell
>>> generator bloat memory so much?  Can’t you just use shallow copies of the
>>> primary data, and then add the ghost cells?
>>>
>> adding ghost cell etc, means adding new elements to existing arrays.
>> since vtk doesn't modify input arrays, when it needs to add new
>> elements, it has to create a deep-copy and then add new elements to
>> it.
>>
>> Why can’t we have the cell data to point data filter be smart enough to
>>> add
>>> ghost cells if necessary for calculations, then throw them away?
>>>
>> Unless I am missing something, no reason. Seems to me that's just the
>> way it was written. I can't see why the filter can't request 1 extra
>> ghost-level from its input. (cc, Berk).
>>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be
>>> about
>>> 18 GB of data?
>>>
>> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>>
>> Utkarsh
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=
>> Paraview-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> https://public.kitware.com/mailman/listinfo/paraview-developers
>>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensou
> rce/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=
> Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180320/ca689bcc/attachment.html>