<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Berk,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Finally managed to grab an allocation of some Tesla K40 nodes on our cluster, to check GPU rendering of the full 17GB file with 2 x 12GB GPUs. I see the same
thing as I did with OSMesa rendering.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The 9GB downsampled version works great, across 2 nodes both with a single K40. Go up to the 17GB original file and nothing is rendered, no errors. Same behavior
with OPENGL or OPENGL2 backends.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This is all on paraview 4.3.1 still – I need to find time to build OSMesa / MPI versions of 4.4 here. But, does 4.4. have any fixes that would be expected to
affect this?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">--</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
David Trudgian Ph.D.<br>
Computational Scientist, BioHPC<br>
UT Southwestern Medical Center<br>
Dallas, TX 75390-9039<br>
Tel: (214) 648-4833</span><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Berk Geveci [mailto:berk.geveci@kitware.com]
<br>
<b>Sent:</b> Tuesday, September 15, 2015 2:43 PM<br>
<b>To:</b> David Trudgian <David.Trudgian@UTSouthwestern.edu><br>
<b>Cc:</b> ParaView Mailing List <paraview@paraview.org><br>
<b>Subject:</b> Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hey David,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am hoping to have some time to play around with volume rendering and hopefully tracking this issue, one thing that I wanted to clarify: it sounds from you description that you have a short (2 byte) value. Is that correct?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-berk<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <<a href="mailto:david.trudgian@utsouthwestern.edu" target="_blank">david.trudgian@utsouthwestern.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Hi,<br>
<br>
We have been experimenting with using Paraview to display very volumes from very<br>
large TIFF stacks generated by whole-brain microscopy equipment. The test stack<br>
has dimensions of 5,368x10,695x150. Stack is assembled in ImageJ from individual<br>
TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti for<br>
convenience. Can view slices fine in standalone paraview client on a 256GB machine.<br>
<br>
When we attempt volume rendering on this data across multiple nodes with MPI<br>
nothing appears in the client. Surface view works as expected. On switching to<br>
volume rendering the client's display will show nothing. There are no messages<br>
from the client or servers - no output.<br>
<br>
This is happening when running pvserver across GPU nodes with NVIDIA Tesla<br>
cards, or using CPU only with OSMESA. pvserver memory usage is well below what<br>
we have on the nodes - no memory warnings/errors.<br>
<br>
Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB then we can<br>
get working volume rendering. The 17GB never works regardless of scaling<br>
nodes/mpi processes. The 4/9GB will work on 1 or 2 nodes.<br>
<br>
Am confused by the lack of rendering, as we don't have memory issues, or an<br>
other messages at all. Am wondering if there are any inherent limitation, or I'm<br>
missing something stupid.<br>
<br>
Thanks,<br>
<br>
Dave Trudgian<br>
<br>
<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">
http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">
http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br>
<hr>
<table border="0" cellpadding="0">
<tbody>
<tr>
<td><!-- Copyright 2015. UT Southwestern Medical Center 12082014v1 -->
<p align="right" style="text-align:right"><span style="font-size:17pt;letter-spacing:-3px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">UT</span><span style="font-size:17pt;letter-spacing:-0px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">
Southwestern</span> </p>
</td>
</tr>
<tr>
<td>
<p align="right" style="text-align:right"><span style="font-size:13pt;font-family:"Helvetica","sans-serif";color:#666D70;">Medical Center</span></p>
</td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td>
<p style="text-align:right"><span style="font-size:10pt;font-family:"Helvetica","sans-serif"; color:#666D70;">The future of medicine, today.</span></p>
</td>
</tr>
</tbody>
</table>
</body>
</html>