<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Totally, totally agree. Lets go with the programmable filter solution.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Cory Quammen [mailto:cory.quammen@kitware.com] <br>
<b>Sent:</b> Tuesday, March 20, 2018 7:27 PM<br>
<b>To:</b> Bauer, Andy (External Contacts) <andy.bauer@kitware.com><br>
<b>Cc:</b> Scott, W Alan <wascott@sandia.gov>; paraview-developers@paraview.org; Patchett, John M. (LANL) <patchett@lanl.gov><br>
<b>Subject:</b> Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 20, 2018 at 8:43 PM, Andy Bauer <<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Well, I could probably have this done by tomorrow but in the end I leave the decision on whether we wait on the release for this change to Utkarsh and Cory. I'm not sure how much of a wrench it would put into their plans.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In my judgement, it's too late to put this in 5.5. It may seem like a simple change, but experience shows time and again that simple changes do not exist :-) We do not have sufficient time to test it and get feedback prior to the release.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">On the other hand, if this is really the specific use case then a Python Programmable Filter could just as easily do the trick and would be nearly negligible cost. In fact, I'll put together the a python script that does that and send it
out shortly.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sounds like a good plan.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Cory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Mar 20, 2018 at 8:37 PM, Scott, W Alan <<a href="mailto:wascott@sandia.gov" target="_blank">wascott@sandia.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Andy,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I assume we are way too late in the release process to implement this for 5.5.0.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">John P., should this be implemented early next release, so you can build an alpha/ master build? Or will you just wait for 5.6.0, out early fall, which should make this question
irrelevant?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Alan<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Andy Bauer [mailto:<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>]
<br>
<b>Sent:</b> Tuesday, March 20, 2018 6:35 PM<br>
<b>To:</b> Nouanesengsy, Boonthanome (LANL) <<a href="mailto:boonth@lanl.gov" target="_blank">boonth@lanl.gov</a>><br>
<b>Cc:</b> Ayachit, Utkarsh (External Contacts) <<a href="mailto:utkarsh.ayachit@kitware.com" target="_blank">utkarsh.ayachit@kitware.com</a>>; Scott, W Alan <<a href="mailto:wascott@sandia.gov" target="_blank">wascott@sandia.gov</a>>;
<a href="mailto:paraview-developers@paraview.org" target="_blank">paraview-developers@paraview.org</a>; Patchett, John M. (LANL) <<a href="mailto:patchett@lanl.gov" target="_blank">patchett@lanl.gov</a>><br>
<b>Subject:</b> Re: [Paraview-developers] FW: [EXTERNAL] Re: ParaView | Memory consumption on Paraview (#18035)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Going back to the Merge Blocks filter, I'd say that if the input grid is a vtkUnstructuredGrid and there's only a single block on that process in this case the Merge Blocks filter
should do a shallow copy and exit quickly, which it doesn't do. This isn't a silver bullet fix and probably won't matter after the composite dataset rework but for what LANL wants in this case, this would be a very simple and efficient solution.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Mar 20, 2018 at 6:00 PM, Boonthanome Nouanesengsy <<a href="mailto:boonth@lanl.gov" target="_blank">boonth@lanl.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Unless I am missing something, no reason. Seems to me that's just the<br>
way it was written. I can't see why the filter can't request 1 extra<br>
ghost-level from its input. (cc, Berk).<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Because we start with cell data, we actually require two layers of ghost cells. The ghost cells filter was developed to allocate as many layers of ghost cells as needed for a distributed
dataset.<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any other ideas how to decrease the bloat of about 4 GB of data to be about<br>
18 GB of data?<br>
Is this for a Catalyst or pvbatch pipeline or for an interactive session?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It's for an interactive session. We have a user running out of memory trying to use a certain pipeline. More information is in the issue at
<a href="https://gitlab.kitware.com/paraview/paraview/issues/18035" target="_blank">
https://gitlab.kitware.com/paraview/paraview/issues/18035</a><br>
<br>
Boonth<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
<br>
On 03/20/2018 03:08 PM, Utkarsh Ayachit wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why does merge blocks grow so much? Why can’t this filter just do shallow<br>
copies? Is this impossible, or just not implemented?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">No, merge blocks is not a shallow copy, it needs to combine mutliple<br>
vtkDataSet's into a single vtkDataSet merging all points into a single<br>
array, same for cells, other attribute arrays etc. Merge Blocks should<br>
indeed be considered as a workaround for issue when handling<br>
mutliblock datasets in the pipeline and not the go-to solution. There<br>
is work currently underway that will revamp multiblock dataset support<br>
in VTK/ParaView which should ultimately make Merge Blocks obsolete.<br>
<br>
Looking at your pipeline, looks like you're doing MergeBlocks so that<br>
you can run the Ghost Cells generator filter. What is the nature of<br>
your data? Is it truly a multiblock or just has 1 block in it on each<br>
rank? If its just 1 block, what is the type of the block? If it's a<br>
unstructured grid, it will trivial to create a filter that just passes<br>
the input block out as unstructured grid without duplicating.<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why can’t Merge Blocks do shallow copies? Is this impossible, or just not<br>
implemented?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">It's impossible. See previous comment for justification based on how<br>
things are currently.<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">(I think I know this one, but here it is). Why does the ghost cell<br>
generator bloat memory so much? Can’t you just use shallow copies of the<br>
primary data, and then add the ghost cells?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">adding ghost cell etc, means adding new elements to existing arrays.<br>
since vtk doesn't modify input arrays, when it needs to add new<br>
elements, it has to create a deep-copy and then add new elements to<br>
it.<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why can’t we have the cell data to point data filter be smart enough to add<br>
ghost cells if necessary for calculations, then throw them away?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Unless I am missing something, no reason. Seems to me that's just the<br>
way it was written. I can't see why the filter can't request 1 extra<br>
ghost-level from its input. (cc, Berk).<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any other ideas how to decrease the bloat of about 4 GB of data to be about<br>
18 GB of data?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Is this for a Catalyst or pvbatch pipeline or for an interactive session?<br>
<br>
Utkarsh<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=Paraview-developers" target="_blank">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=Paraview-developers" target="_blank">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=Paraview-developers" target="_blank">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Cory Quammen<br>
Staff R&D Engineer<br>
Kitware, Inc.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>