[Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)
Boonthanome Nouanesengsy
boonth at lanl.gov
Wed Mar 21 15:06:37 EDT 2018
> Boonth or John, would you mind shooting me an email that you got this
> email. I'm worried that a spambot may not let this get through based
> on the attachments.
Yes, got the email.
> Going back to the Merge Blocks filter, I'd say that if the input grid
> is a vtkUnstructuredGrid and there's only a single block on that
> process in this case the Merge Blocks filter should do a shallow copy
> and exit quickly, which it doesn't do. This isn't a silver bullet fix
> and probably won't matter after the composite dataset rework but for
> what LANL wants in this case, this would be a very simple and
> efficient solution.
The problem is that in many use cases the number of processes used for
writing the data will be different from the number of processes used for
visualization. In this particular use case it won't help, but thanks for
the programmable filter. It should be helpful in the future.
Boonth
On 03/20/2018 08:18 PM, Scott, W Alan wrote:
>
> Totally, totally agree. Lets go with the programmable filter solution.
>
> *From:* Cory Quammen [mailto:cory.quammen at kitware.com]
> *Sent:* Tuesday, March 20, 2018 7:27 PM
> *To:* Bauer, Andy (External Contacts) <andy.bauer at kitware.com>
> *Cc:* Scott, W Alan <wascott at sandia.gov>;
> paraview-developers at paraview.org; Patchett, John M. (LANL)
> <patchett at lanl.gov>
> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView |
> Memory consumption on Paraview (#18035)
>
> On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <andy.bauer at kitware.com
> <mailto:andy.bauer at kitware.com>> wrote:
>
> Well, I could probably have this done by tomorrow but in the end I
> leave the decision on whether we wait on the release for this
> change to Utkarsh and Cory. I'm not sure how much of a wrench it
> would put into their plans.
>
> In my judgement, it's too late to put this in 5.5. It may seem like a
> simple change, but experience shows time and again that simple changes
> do not exist :-) We do not have sufficient time to test it and get
> feedback prior to the release.
>
> On the other hand, if this is really the specific use case then a
> Python Programmable Filter could just as easily do the trick and
> would be nearly negligible cost. In fact, I'll put together the a
> python script that does that and send it out shortly.
>
> Sounds like a good plan.
>
> Thanks,
>
> Cory
>
> On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <wascott at sandia.gov
> <mailto:wascott at sandia.gov>> wrote:
>
> Andy,
>
> I assume we are way too late in the release process to
> implement this for 5.5.0.
>
> John P., should this be implemented early next release, so you
> can build an alpha/ master build? Or will you just wait for
> 5.6.0, out early fall, which should make this question irrelevant?
>
> Alan
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com
> <mailto:andy.bauer at kitware.com>]
> *Sent:* Tuesday, March 20, 2018 6:35 PM
> *To:* Nouanesengsy, Boonthanome (LANL) <boonth at lanl.gov
> <mailto:boonth at lanl.gov>>
> *Cc:* Ayachit, Utkarsh (External Contacts)
> <utkarsh.ayachit at kitware.com
> <mailto:utkarsh.ayachit at kitware.com>>; Scott, W Alan
> <wascott at sandia.gov <mailto:wascott at sandia.gov>>;
> paraview-developers at paraview.org
> <mailto:paraview-developers at paraview.org>; Patchett, John M.
> (LANL) <patchett at lanl.gov <mailto:patchett at lanl.gov>>
> *Subject:* Re: [Paraview-developers] FW: [EXTERNAL] Re:
> ParaView | Memory consumption on Paraview (#18035)
>
> Going back to the Merge Blocks filter, I'd say that if the
> input grid is a vtkUnstructuredGrid and there's only a single
> block on that process in this case the Merge Blocks filter
> should do a shallow copy and exit quickly, which it doesn't
> do. This isn't a silver bullet fix and probably won't matter
> after the composite dataset rework but for what LANL wants in
> this case, this would be a very simple and efficient solution.
>
> On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy
> <boonth at lanl.gov <mailto:boonth at lanl.gov>> wrote:
>
> Unless I am missing something, no reason. Seems to me
> that's just the
> way it was written. I can't see why the filter can't
> request 1 extra
> ghost-level from its input. (cc, Berk).
>
> Because we start with cell data, we actually require two
> layers of ghost cells. The ghost cells filter was
> developed to allocate as many layers of ghost cells as
> needed for a distributed dataset.
>
> Any other ideas how to decrease the bloat of about 4
> GB of data to be about
> 18 GB of data?
> Is this for a Catalyst or pvbatch pipeline or for an
> interactive session?
>
> It's for an interactive session. We have a user running
> out of memory trying to use a certain pipeline. More
> information is in the issue at
> https://gitlab.kitware.com/paraview/paraview/issues/18035
>
> Boonth
>
>
>
>
> On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:
>
> Why does merge blocks grow so much? Why can’t
> this filter just do shallow
> copies? Is this impossible, or just not implemented?
>
> No, merge blocks is not a shallow copy, it needs to
> combine mutliple
> vtkDataSet's into a single vtkDataSet merging all
> points into a single
> array, same for cells, other attribute arrays etc.
> Merge Blocks should
> indeed be considered as a workaround for issue when
> handling
> mutliblock datasets in the pipeline and not the go-to
> solution. There
> is work currently underway that will revamp multiblock
> dataset support
> in VTK/ParaView which should ultimately make Merge
> Blocks obsolete.
>
> Looking at your pipeline, looks like you're doing
> MergeBlocks so that
> you can run the Ghost Cells generator filter. What is
> the nature of
> your data? Is it truly a multiblock or just has 1
> block in it on each
> rank? If its just 1 block, what is the type of the
> block? If it's a
> unstructured grid, it will trivial to create a filter
> that just passes
> the input block out as unstructured grid without
> duplicating.
>
> Why can’t Merge Blocks do shallow copies? Is this
> impossible, or just not
> implemented?
>
> It's impossible. See previous comment for
> justification based on how
> things are currently.
>
> (I think I know this one, but here it is). Why
> does the ghost cell
> generator bloat memory so much? Can’t you just use
> shallow copies of the
> primary data, and then add the ghost cells?
>
> adding ghost cell etc, means adding new elements to
> existing arrays.
> since vtk doesn't modify input arrays, when it needs
> to add new
> elements, it has to create a deep-copy and then add
> new elements to
> it.
>
> Why can’t we have the cell data to point data
> filter be smart enough to add
> ghost cells if necessary for calculations, then
> throw them away?
>
> Unless I am missing something, no reason. Seems to me
> that's just the
> way it was written. I can't see why the filter can't
> request 1 extra
> ghost-level from its input. (cc, Berk).
>
> Any other ideas how to decrease the bloat of about
> 4 GB of data to be about
> 18 GB of data?
>
> Is this for a Catalyst or pvbatch pipeline or for an
> interactive session?
>
> Utkarsh
> _______________________________________________
> Powered by www.kitware.com <http://www.kitware.com>
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at:
> http://markmail.org/search/?q=Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>
> _______________________________________________
> Powered by www.kitware.com <http://www.kitware.com>
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at:
> http://markmail.org/search/?q=Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>
> _______________________________________________
> Powered by www.kitware.com <http://www.kitware.com>
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> <http://www.kitware.com/opensource/opensource.html>
>
> Search the list archives at:
> http://markmail.org/search/?q=Paraview-developers
> <http://markmail.org/search/?q=Paraview-developers>
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
>
>
>
> --
>
> Cory Quammen
> Staff R&D Engineer
> Kitware, Inc.
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=Paraview-developers
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview-developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/paraview-developers/attachments/20180321/2e8894e4/attachment-0001.html>
More information about the Paraview-developers
mailing list