<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">Boonth or John, would you mind shooting me
an email that you got this email. I'm worried that a spambot may
not let this get through based on the attachments.</blockquote>
Yes, got the email.<br>
<br>
<blockquote type="cite">Going back to the Merge Blocks filter, I'd
say that if the input grid is a vtkUnstructuredGrid and there's
only a single block on that process in this case the Merge Blocks
filter should do a shallow copy and exit quickly, which it doesn't
do. This isn't a silver bullet fix and probably won't matter after
the composite dataset rework but for what LANL wants in this case,
this would be a very simple and efficient solution.</blockquote>
The problem is that in many use cases the number of processes used
for writing the data will be different from the number of processes
used for visualization. In this particular use case it won't help,
but thanks for the programmable filter. It should be helpful in the
future.<br>
<br>
Boonth<br>
<br>
<br>
<div class="moz-cite-prefix">On 03/20/2018 08:18 PM, Scott, W Alan
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:dcba73732f424b5fb2d00a488b56f050@ES01AMSNLNT.srn.sandia.gov">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Totally, totally agree. Lets go with the
programmable filter solution.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in
0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Cory Quammen
[<a class="moz-txt-link-freetext" href="mailto:cory.quammen@kitware.com">mailto:cory.quammen@kitware.com</a>] <br>
<b>Sent:</b> Tuesday, March 20, 2018 7:27 PM<br>
<b>To:</b> Bauer, Andy (External Contacts)
<a class="moz-txt-link-rfc2396E" href="mailto:andy.bauer@kitware.com"><andy.bauer@kitware.com></a><br>
<b>Cc:</b> Scott, W Alan <a class="moz-txt-link-rfc2396E" href="mailto:wascott@sandia.gov"><wascott@sandia.gov></a>;
<a class="moz-txt-link-abbreviated" href="mailto:paraview-developers@paraview.org">paraview-developers@paraview.org</a>; Patchett, John M.
(LANL) <a class="moz-txt-link-rfc2396E" href="mailto:patchett@lanl.gov"><patchett@lanl.gov></a><br>
<b>Subject:</b> Re: [Paraview-developers] FW: [EXTERNAL]
Re: ParaView | Memory consumption on Paraview (#18035)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 20, 2018 at 8:43 PM,
Andy Bauer <<a href="mailto:andy.bauer@kitware.com"
target="_blank" moz-do-not-send="true">andy.bauer@kitware.com</a>>
wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Well, I could probably have
this done by tomorrow but in the end I leave the
decision on whether we wait on the release for
this change to Utkarsh and Cory. I'm not sure
how much of a wrench it would put into their
plans.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In my judgement, it's too late to
put this in 5.5. It may seem like a simple change,
but experience shows time and again that simple
changes do not exist :-) We do not have sufficient
time to test it and get feedback prior to the
release.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">On the other hand, if this is
really the specific use case then a Python
Programmable Filter could just as easily do the
trick and would be nearly negligible cost. In
fact, I'll put together the a python script that
does that and send it out shortly.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sounds like a good plan.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Cory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Mar 20, 2018 at
8:37 PM, Scott, W Alan <<a
href="mailto:wascott@sandia.gov"
target="_blank" moz-do-not-send="true">wascott@sandia.gov</a>>
wrote:<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Andy,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
assume we are way too late in the
release process to implement this for
5.5.0.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">John
P., should this be implemented early
next release, so you can build an
alpha/ master build? Or will you just
wait for 5.6.0, out early fall, which
should make this question irrelevant?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Alan<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div
style="border:none;border-left:solid
blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div
style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0in
0in 0in">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Andy Bauer [mailto:<a
href="mailto:andy.bauer@kitware.com"
target="_blank"
moz-do-not-send="true">andy.bauer@kitware.com</a>]
<br>
<b>Sent:</b> Tuesday, March 20,
2018 6:35 PM<br>
<b>To:</b> Nouanesengsy,
Boonthanome (LANL) <<a
href="mailto:boonth@lanl.gov"
target="_blank"
moz-do-not-send="true">boonth@lanl.gov</a>><br>
<b>Cc:</b> Ayachit, Utkarsh
(External Contacts) <<a
href="mailto:utkarsh.ayachit@kitware.com"
target="_blank"
moz-do-not-send="true">utkarsh.ayachit@kitware.com</a>>;
Scott, W Alan <<a
href="mailto:wascott@sandia.gov"
target="_blank"
moz-do-not-send="true">wascott@sandia.gov</a>>;
<a
href="mailto:paraview-developers@paraview.org"
target="_blank"
moz-do-not-send="true">paraview-developers@paraview.org</a>;
Patchett, John M. (LANL) <<a
href="mailto:patchett@lanl.gov" target="_blank" moz-do-not-send="true">patchett@lanl.gov</a>><br>
<b>Subject:</b> Re:
[Paraview-developers] FW:
[EXTERNAL] Re: ParaView | Memory
consumption on Paraview (#18035)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Going
back to the Merge Blocks
filter, I'd say that if the
input grid is a
vtkUnstructuredGrid and
there's only a single block on
that process in this case the
Merge Blocks filter should do
a shallow copy and exit
quickly, which it doesn't do.
This isn't a silver bullet fix
and probably won't matter
after the composite dataset
rework but for what LANL wants
in this case, this would be a
very simple and efficient
solution.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Tue, Mar 20, 2018 at 6:00
PM, Boonthanome Nouanesengsy
<<a
href="mailto:boonth@lanl.gov"
target="_blank"
moz-do-not-send="true">boonth@lanl.gov</a>>
wrote:<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Unless
I am missing something,
no reason. Seems to me
that's just the<br>
way it was written. I
can't see why the filter
can't request 1 extra<br>
ghost-level from its
input. (cc, Berk).<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Because
we start with cell data,
we actually require two
layers of ghost cells. The
ghost cells filter was
developed to allocate as
many layers of ghost cells
as needed for a
distributed dataset.<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any
other ideas how to
decrease the bloat of
about 4 GB of data to be
about<br>
18 GB of data?<br>
Is this for a Catalyst
or pvbatch pipeline or
for an interactive
session?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It's
for an interactive
session. We have a user
running out of memory
trying to use a certain
pipeline. More information
is in the issue at
<a
href="https://gitlab.kitware.com/paraview/paraview/issues/18035"
target="_blank"
moz-do-not-send="true">
https://gitlab.kitware.com/paraview/paraview/issues/18035</a><br>
<br>
Boonth<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
<br>
On 03/20/2018 03:08
PM, Utkarsh Ayachit
wrote:<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in 0in
0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why does
merge blocks grow
so much? Why
can’t this filter
just do shallow<br>
copies? Is this
impossible, or
just not
implemented?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">No,
merge blocks is not
a shallow copy, it
needs to combine
mutliple<br>
vtkDataSet's into a
single vtkDataSet
merging all points
into a single<br>
array, same for
cells, other
attribute arrays
etc. Merge Blocks
should<br>
indeed be considered
as a workaround for
issue when handling<br>
mutliblock datasets
in the pipeline and
not the go-to
solution. There<br>
is work currently
underway that will
revamp multiblock
dataset support<br>
in VTK/ParaView
which should
ultimately make
Merge Blocks
obsolete.<br>
<br>
Looking at your
pipeline, looks like
you're doing
MergeBlocks so that<br>
you can run the
Ghost Cells
generator filter.
What is the nature
of<br>
your data? Is it
truly a multiblock
or just has 1 block
in it on each<br>
rank? If its just 1
block, what is the
type of the block?
If it's a<br>
unstructured grid,
it will trivial to
create a filter that
just passes<br>
the input block out
as unstructured grid
without duplicating.<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why can’t
Merge Blocks do
shallow copies?
Is this
impossible, or
just not<br>
implemented?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">It's
impossible. See
previous comment for
justification based
on how<br>
things are
currently.<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">(I think I
know this one, but
here it is). Why
does the ghost
cell<br>
generator bloat
memory so much?
Can’t you just use
shallow copies of
the<br>
primary data, and
then add the ghost
cells?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">adding
ghost cell etc,
means adding new
elements to existing
arrays.<br>
since vtk doesn't
modify input arrays,
when it needs to add
new<br>
elements, it has to
create a deep-copy
and then add new
elements to<br>
it.<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Why can’t we
have the cell data
to point data
filter be smart
enough to add<br>
ghost cells if
necessary for
calculations, then
throw them away?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Unless
I am missing
something, no
reason. Seems to me
that's just the<br>
way it was written.
I can't see why the
filter can't request
1 extra<br>
ghost-level from its
input. (cc, Berk).<o:p></o:p></p>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any other
ideas how to
decrease the bloat
of about 4 GB of
data to be about<br>
18 GB of data?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Is
this for a Catalyst
or pvbatch pipeline
or for an
interactive session?<br>
<br>
Utkarsh<br>
_______________________________________________<br>
Powered by <a
href="http://www.kitware.com"
target="_blank"
moz-do-not-send="true">www.kitware.com</a><br>
<br>
Visit other Kitware
open-source projects
at <a
href="http://www.kitware.com/opensource/opensource.html"
target="_blank"
moz-do-not-send="true">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list
archives at: <a
href="http://markmail.org/search/?q=Paraview-developers"
target="_blank"
moz-do-not-send="true">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to
subscribe/unsubscribe:<br>
<a
href="https://public.kitware.com/mailman/listinfo/paraview-developers"
target="_blank"
moz-do-not-send="true">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
_______________________________________________<br>
Powered by <a
href="http://www.kitware.com"
target="_blank"
moz-do-not-send="true">www.kitware.com</a><br>
<br>
Visit other Kitware
open-source projects
at <a
href="http://www.kitware.com/opensource/opensource.html"
target="_blank"
moz-do-not-send="true">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list
archives at: <a
href="http://markmail.org/search/?q=Paraview-developers"
target="_blank"
moz-do-not-send="true">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to
subscribe/unsubscribe:<br>
<a
href="https://public.kitware.com/mailman/listinfo/paraview-developers"
target="_blank"
moz-do-not-send="true">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com"
target="_blank" moz-do-not-send="true">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a
href="http://www.kitware.com/opensource/opensource.html"
target="_blank" moz-do-not-send="true">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Search the list archives at: <a
href="http://markmail.org/search/?q=Paraview-developers"
target="_blank" moz-do-not-send="true">
http://markmail.org/search/?q=Paraview-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a
href="https://public.kitware.com/mailman/listinfo/paraview-developers"
target="_blank" moz-do-not-send="true">https://public.kitware.com/mailman/listinfo/paraview-developers</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Cory Quammen<br>
Staff R&D Engineer<br>
Kitware, Inc.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Powered by <a class="moz-txt-link-abbreviated" href="http://www.kitware.com">www.kitware.com</a>
Visit other Kitware open-source projects at <a class="moz-txt-link-freetext" href="http://www.kitware.com/opensource/opensource.html">http://www.kitware.com/opensource/opensource.html</a>
Search the list archives at: <a class="moz-txt-link-freetext" href="http://markmail.org/search/?q=Paraview-developers">http://markmail.org/search/?q=Paraview-developers</a>
Follow this link to subscribe/unsubscribe:
<a class="moz-txt-link-freetext" href="https://public.kitware.com/mailman/listinfo/paraview-developers">https://public.kitware.com/mailman/listinfo/paraview-developers</a>
</pre>
</blockquote>
<br>
</body>
</html>