[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

[Boonthanome Nouanesengsy]

> Unless I am missing something, no reason. Seems to me that's just the
> way it was written. I can't see why the filter can't request 1 extra
> ghost-level from its input. (cc, Berk).
Because we start with cell data, we actually require two layers of ghost 
cells. The ghost cells filter was developed to allocate as many layers 
of ghost cells as needed for a distributed dataset.

> Any other ideas how to decrease the bloat of about 4 GB of data to be about
> 18 GB of data?
> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
It's for an interactive session. We have a user running out of memory 
trying to use a certain pipeline. More information is in the issue at 
https://gitlab.kitware.com/paraview/paraview/issues/18035

Boonth


On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>> Why does merge blocks grow so much?  Why can’t this filter just do shallow
>> copies?  Is this impossible, or just not implemented?
> No, merge blocks is not a shallow copy, it needs to combine mutliple
> vtkDataSet's into a single vtkDataSet merging all points into a single
> array, same for cells, other attribute arrays etc. Merge Blocks should
> indeed be considered as a workaround for issue when handling
> mutliblock datasets in the pipeline and not the go-to solution. There
> is work currently underway that will revamp multiblock dataset support
> in VTK/ParaView which should ultimately make Merge Blocks obsolete.
>
> Looking at your pipeline, looks like you're doing MergeBlocks so that
> you can run the Ghost Cells generator filter. What is the nature of
> your data? Is it truly a multiblock or just has 1 block in it on each
> rank? If its just 1 block, what is the type of the block? If it's a
> unstructured grid, it will trivial to create a filter that just passes
> the input block out as unstructured grid without duplicating.
>
>> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
>> implemented?
> It's impossible. See previous comment for justification based on how
> things are currently.
>
>> (I think I know this one, but here it is).  Why does the ghost cell
>> generator bloat memory so much?  Can’t you just use shallow copies of the
>> primary data, and then add the ghost cells?
> adding ghost cell etc, means adding new elements to existing arrays.
> since vtk doesn't modify input arrays, when it needs to add new
> elements, it has to create a deep-copy and then add new elements to
> it.
>
>> Why can’t we have the cell data to point data filter be smart enough to add
>> ghost cells if necessary for calculations, then throw them away?
> Unless I am missing something, no reason. Seems to me that's just the
> way it was written. I can't see why the filter can't request 1 extra
> ghost-level from its input. (cc, Berk).
>
>> Any other ideas how to decrease the bloat of about 4 GB of data to be about
>> 18 GB of data?
> Is this for a Catalyst or pvbatch pipeline or for an interactive session?
>
> Utkarsh
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers