<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Ken<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">If you give the arguments —tdx=1 —tdy=1, ParaView should happily create a "tiled" display of a single image that you can then break up and transfer with another software service. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><<o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">That’s interesting. So if I do something along the lines of<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.0pt"><span style="font-size:10.5pt;color:black">mpiexec –n 1024 pvserver —tdx=1 —tdy=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">I’ll get just one window. But sadly I’ll have 1024 framebuffers of 0.5GB or thereabouts each. That’s a shame. It certainly be a great starting point for me though.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Question.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Would this scenario be a potential use case for running (not sure if this is the command you’d use, but it’s just for illustration)<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.0pt"><span style="font-size:10.5pt;color:black">mpiexec –n 1024 pvserver –use some kind of dataserver : -n 12 pvserver – use some kind of renderserver –tdx=1 –tdy=1 (or 4x3 or whatever)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">So that the majority of the heavy lifting is done by the data servers, and the renderservers can do the display with a reduced footprint in terms of framebuffer allocation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">I’ve never tried using the data/render server mode of operations, but it seems like this might be a good strategy – n’est ce que pas?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">JB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB">
Moreland, Kenneth [mailto:kmorel@sandia.gov] <br>
<b>Sent:</b> 04 June 2013 20:50<br>
<b>To:</b> Biddiscombe, John A.; paraview-developers@paraview.org<br>
<b>Subject:</b> Re: [Paraview-developers] DisplayCluster + ParaView<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">John,<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">I have no experience with DisplayCluster and haven't messed with tiled displays in a long time, so I don't have a complete answer for you. But perhaps I can fill in
some holes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">If you give the arguments —tdx=1 —tdy=1, ParaView should happily create a "tiled" display of a single image that you can then break up and transfer with another software
service. The only real problem I can see is that there is no way to directly specify the number of pixels in the display. The original implementation assumed that node GPUs are connected directly to displays, so just gets the size of the displays from the
Xhost screen. So specifying the size of the framebuffers will require some code modification, but they should be pretty minor ones.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">I don't know if anyone has directly interfaced ParaView with the DisplayCluster software. I have not been following it that closely. I do remember a presentation on
SAGE in the 2008 Ultrascale Visualization Workshop that described using SAGE to stream pixels from ParaView on one cluster to another cluster driving a tiled display very similarly to what you are doing (<a href="http://vis.cs.ucdavis.edu/Ultravis08/slides/Leigh-UltraViz.pdf">http://vis.cs.ucdavis.edu/Ultravis08/slides/Leigh-UltraViz.pdf</a>).
If you have no other luck, you might consider contacting those authors.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">As far as data replication is concerned, ParaView should handle that for you. The standard mode of operation is to leave the data distributed and use image composition
for parallel rendering. I believe that if the size of the data being rendering is below the remote rendering threshold, then the data will be distributed among all tiled display renders just as it would be sent to the client, but unless you screwed up your
settings that only happens when the data is small enough to be collected.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">As far as ParaView is concerned, handling 12 vs 6 vs 1 tiles is about the same because IceT takes care of all that internally. As far as performance is concerned, I
would expect the single huge display to be at least marginally better than the small displays because of better load balancing and fewer possible rendering passes. However, I do worry about the memory overhead of compositing a single 7920x3060 image. I expect
IceT to create at least 3 framebuffers to simultaneously send, receive, and compress/composite images. By my count that is over 1/2 GB per MPI process.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black">-Ken<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:
</span></b><span style="color:black"><Biddiscombe>, "John A." <<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>><br>
<b>Date: </b>Monday, June 3, 2013 2:19 PM<br>
<b>To: </b>"<a href="mailto:paraview-developers@paraview.org">paraview-developers@paraview.org</a>" <<a href="mailto:paraview-developers@paraview.org">paraview-developers@paraview.org</a>><br>
<b>Subject: </b>[EXTERNAL] [Paraview-developers] DisplayCluster + ParaView<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-right:0cm" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">Colleagues have installed a large tiled display and are running displaycluster from TACC on it.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">I’ve never used a tiled display, with paraview or any other software so I’m not sure what to do.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">The out of the box solution is to use –tdx=4 –tdy=3 and set the display env var to point to the correct gpu, but I don’t have access to the nodes hosting the wall itself (can’t run 6
pvservers on the wall nodes or send X to them) and would like to use the viz cluster to run N pvservers (where N>number of tiles) and send the pixels using the displaycluster API ( see
<a href="https://github.com/TACC/DisplayCluster">https://github.com/TACC/DisplayCluster</a> ). It essentially allows you to send rectangular blocks of pixels into the final image, so I’d simply want 4x3 regions to send.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">I’ll therefore need to setup a custom view in paraview and capture pixels to send them. Since the display could conceivably be 1980x1020 X4 X3 pixels, this will mean some very large framebuffers
on all nodes – <o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:72.0pt;text-indent:-18.0pt"><span style="color:black">1)</span><span style="font-size:7.0pt;font-family:"Times New Roman","serif";color:black">
</span><span style="color:black">Has anyone already integrated DisplayCluster into paraview and is there a paraview branch anywhere with developments in?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:72.0pt;text-indent:-18.0pt"><span style="color:black">2)</span><span style="font-size:7.0pt;font-family:"Times New Roman","serif";color:black">
</span><span style="color:black">I do not want to replicate data on each render node, which seems to be considered usual for tiled displays, but is out of the question as far as I’m concerned. Is there a better strategy for handling the display. Is 12 small
windows better than 1 very large one for example – I can imagine setting 12 views with a different frustum and pasting them into the final large image, but then perhaps the overhead of managing 12 windows is worse than 1 huge one?) (NB. It’s actually 6 windows
instead of 12 because 2 displays are driven off each GPU on the wall driver nodes and the x server considers them as one)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:54.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">Any advice gratefully received. Thanks<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black">JB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#1F497D">--
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#1F497D">John Biddiscombe, email:biddisco @.at.@ cscs.ch</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#1F497D"><a href="http://www.cscs.ch/"><span style="color:blue">http://www.cscs.ch/</span></a></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#1F497D">CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="IT-CH" style="font-size:10.5pt;font-family:"Courier New";color:#1F497D">Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="IT-CH" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</body>
</html>