<p dir="ltr">Sorry I've got no insight on this. </p>
<div class="gmail_quote">On Apr 21, 2016 6:29 AM, "Biddiscombe, John A." <<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dave<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">So it turns out that aprun/alps won’t let me run a script with dstat to monitor the memory use, and using craypat makes the execution take too long (weeks it
seems). So I tried importing part of the dstat python into my paraview python<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Using this I am able to query memory use on the node and dump it out.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Question : If I start a background python thread that polls memory like this<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">#####################<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"># start a thread that prints mem use<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">#####################<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">def dump_memory():<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> # do something here ...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> global stop_thread<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> line = ''<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> for o in stats:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> o.extract()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> line = line + ' ' + o.show()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> print line + dstat.ansi['reset']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> # call f() again in n seconds<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> if stop_thread==0:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> threading.Timer(0.5, dump_memory).start()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">and set it going at the start of my paraview python pipeline : Will it actually run and collect info whilst the paraview script is executing the rest of the
pipeline.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">intuition and brief googling tells me that this won’t work.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Can it be done?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">JB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ParaView [mailto:<a href="mailto:paraview-bounces@paraview.org" target="_blank">paraview-bounces@paraview.org</a>]
<b>On Behalf Of </b>Biddiscombe, John A.<br>
<b>Sent:</b> 18 April 2016 14:53<br>
<b>To:</b> David E DeMarle<div class="elided-text"><br>
<b>Cc:</b> <a href="mailto:paraview@paraview.org" target="_blank">paraview@paraview.org</a><br>
<b>Subject:</b> Re: [Paraview] peak memory use on remote nodes<u></u><u></u></div></span></p>
</div>
</div><div class="elided-text">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks Dave<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">A colleague here suggested I put my
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">aprun pvbatch <blah> into a script and add a ‘dtrace” command at the start of the script. This will then dump out a txt file on each node and after the job
completes I can grep the info I need.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’m going to give that a try first as it seems straightforward, but I will revert to your suggestion if I fail and nothing else comes up.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">cheers<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">JB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> David E DeMarle [<a href="mailto:dave.demarle@kitware.com" target="_blank">mailto:dave.demarle@kitware.com</a>]
<br>
<b>Sent:</b> 18 April 2016 14:48<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> <a href="mailto:paraview@paraview.org" target="_blank">paraview@paraview.org</a><br>
<b>Subject:</b> Re: [Paraview] peak memory use on remote nodes<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Perhaps extend the benchmark to glean /proc/<PID>/status's VmHWM entry on Linux at least. Get the number for t1 and at t2 and subtract to see what the maximum was in between those two times.<br>
<br>
Seems the right place to do that is kwsys/SystemInformation.cxx, but you could prototype in the script.<u></u><u></u></p>
</div>
<p class="MsoNormal">Hope that helps and please post some code. Seems like a very useful thing to have.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">David E DeMarle<br>
Kitware, Inc.<br>
R&D Engineer<br>
21 Corporate Drive<br>
Clifton Park, NY 12065-8662<br>
Phone: <a href="tel:518-881-4909" value="+15188814909" target="_blank">518-881-4909</a><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Apr 18, 2016 at 2:42 AM, Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">The python scripting module “paraview.benchmark” allows one to get the memory use using “paraview.benchmark.get_memuse” - but I presume this is the memory reported as used by the
system at the moment when the function is called.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Does anyone know a way of recording the peak memory use on remote nodes between t1 and t2 - where t1 and t2 are the start and stop of either a function, job or even paraview python
script on the remote node?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">thanks<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">JB<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">--
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">John Biddiscombe, email:biddisco @.at.@
<a href="http://cscs.ch" target="_blank">cscs.ch</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><a href="http://www.cscs.ch/" target="_blank"><span style="color:windowtext">http://www.cscs.ch/</span></a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">CSCS, Swiss National Supercomputing Centre | Tel:
<a href="tel:%2B41%20%2891%29%20610.82.07" target="_blank">+41 (91) 610.82.07</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="IT-CH" style="font-size:10.0pt;font-family:"Courier New"">Via Trevano 131, 6900 Lugano, Switzerland | Fax:
<a href="tel:%2B41%20%2891%29%20610.82.82" target="_blank">+41 (91) 610.82.82</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="IT-CH"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">
http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">
http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div>
</div>
</blockquote></div>