[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)

[Utkarsh Ayachit]

> Why does merge blocks grow so much?  Why can’t this filter just do shallow
> copies?  Is this impossible, or just not implemented?

No, merge blocks is not a shallow copy, it needs to combine mutliple
vtkDataSet's into a single vtkDataSet merging all points into a single
array, same for cells, other attribute arrays etc. Merge Blocks should
indeed be considered as a workaround for issue when handling
mutliblock datasets in the pipeline and not the go-to solution. There
is work currently underway that will revamp multiblock dataset support
in VTK/ParaView which should ultimately make Merge Blocks obsolete.

Looking at your pipeline, looks like you're doing MergeBlocks so that
you can run the Ghost Cells generator filter. What is the nature of
your data? Is it truly a multiblock or just has 1 block in it on each
rank? If its just 1 block, what is the type of the block? If it's a
unstructured grid, it will trivial to create a filter that just passes
the input block out as unstructured grid without duplicating.

> Why can’t Merge Blocks do shallow copies?  Is this impossible, or just not
> implemented?

It's impossible. See previous comment for justification based on how
things are currently.

> (I think I know this one, but here it is).  Why does the ghost cell
> generator bloat memory so much?  Can’t you just use shallow copies of the
> primary data, and then add the ghost cells?

adding ghost cell etc, means adding new elements to existing arrays.
since vtk doesn't modify input arrays, when it needs to add new
elements, it has to create a deep-copy and then add new elements to
it.

> Why can’t we have the cell data to point data filter be smart enough to add
> ghost cells if necessary for calculations, then throw them away?

Unless I am missing something, no reason. Seems to me that's just the
way it was written. I can't see why the filter can't request 1 extra
ghost-level from its input. (cc, Berk).

> Any other ideas how to decrease the bloat of about 4 GB of data to be about
> 18 GB of data?

Is this for a Catalyst or pvbatch pipeline or for an interactive session?

Utkarsh