<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.im
{mso-style-name:im;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-US" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I just realized you are on PV 4.3.1, that version is probably old enough that we did not have OpenGL2 working with Mesa back then. Updating to a more recent PV/VTK will be needed to use OpenGL2 with Mesa (I think we got it working three or four months ago). As far as I know the 10.6 versions of Mesa should also work fine, anything after 10.5.4 should be good.</span></a></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sorry for the confusion</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ken</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ken Martin PhD</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Chairman & CFO</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kitware Inc.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">28 Corporate Drive</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Clifton Park NY 12065</span></p><p class="MsoNormal"><a href="mailto:ken.martin@kitware.com"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">ken.martin@kitware.com</span></a><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">919 869-8871 (w)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">This communication, including all attachments, contains confidential and legally privileged information, and it is intended only for the use of the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. If you received this communication in error please notify us immediately and destroy the original message. Thank you.</span></p></div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> David Trudgian [mailto:<a href="mailto:David.Trudgian@UTSouthwestern.edu">David.Trudgian@UTSouthwestern.edu</a>] <br><b>Sent:</b> Tuesday, September 15, 2015 12:43 PM<br><b>To:</b> Ken Martin; <a href="mailto:paraview@paraview.org">paraview@paraview.org</a><br><b>Subject:</b> RE: [Paraview] Volume Rendering 17GB 8.5 billion cell volume</span></p></div></div><p class="MsoNormal"> </p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Am currently using Mesa 10.5.4 as I’d seen that as a working config mentioned somewhere on the web, so will try newer. Any reason to avoid the 10.6 series?</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I guess I should also try moving to Paraview 4.4 now :-)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">--</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><br>David Trudgian Ph.D.<br>Computational Scientist, BioHPC<br>UT Southwestern Medical Center<br>Dallas, TX 75390-9039<br>Tel: (214) 648-4833</span><span style="font-size:11.0pt;font-family:Consolas;color:#1f497d"></span></p></div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Ken Martin [<a href="mailto:ken.martin@kitware.com">mailto:ken.martin@kitware.com</a>] <br><b>Sent:</b> Monday, September 14, 2015 2:29 PM<br><b>To:</b> <a href="mailto:paraview@paraview.org">paraview@paraview.org</a><br><b>Subject:</b> Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume</span></p></div></div><p class="MsoNormal"> </p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am not sure about the rest of the email, but the error you included is typically caused by using too old of a version of mesa. I’m not sure about specific driver/etc but I usually suggest trying Mesa version 10.5.5 or later to see if that solves the issue.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ken</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ken Martin PhD</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Chairman & CFO</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kitware Inc.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">28 Corporate Drive</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Clifton Park NY 12065</span></p><p class="MsoNormal"><a href="mailto:ken.martin@kitware.com"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">ken.martin@kitware.com</span></a></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">919 869-8871 (w)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">This communication, including all attachments, contains confidential and legally privileged information, and it is intended only for the use of the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. If you received this communication in error please notify us immediately and destroy the original message. Thank you.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt">From: <b>David Trudgian</b> <<a href="mailto:David.Trudgian@utsouthwestern.edu">David.Trudgian@utsouthwestern.edu</a>><br>Date: Mon, Sep 14, 2015 at 1:35 PM<br>Subject: RE: [Paraview] Volume Rendering 17GB 8.5 billion cell volume<br>To: Aashish Chaudhary <<a href="mailto:aashish.chaudhary@kitware.com">aashish.chaudhary@kitware.com</a>><br>Cc: Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>>, ParaView list <<a href="mailto:paraview@paraview.org">paraview@paraview.org</a>><br><br><br>I've now had chance to build paraview 4.3.1 with the OpenGL2 backend.<br><br>I've not been able to test on GPU accelerated nodes (cluster is too busy to get enough right now), but I have also built as an OSMESA with OpenGL2 version. Unfortunately the symptoms are the same. With the largest 16GB VTI I see wireframe etc, but switching to volume rendering results in nothing visible in the client. No messages from the server or client. Memory usage well below RAM in the systems. Runnning across 8 256GB 24-core nodes with 8-mpi tasks per node.<br><br>As with the OpenGL backend I go to our 9GB or 4GB downsampled VTIs and things work as expected. Great rendering performance running across the 8-node allocation with OpenMPI and OSMESA.<br><br>I did also come across another issue. Switching back to surface view on the 16GB data I had a crash shortly after I started manipulating the view with the mouse. I couldn't replicate this crash with the smaller datasets though.<br><br>ERROR: In /home2/dtrudgian/paraview/ParaView-v4.3.1-source/VTK/Rendering/OpenGL2/vtkShaderProgram.cxx, line 292<br>vtkShaderProgram (0x6488a60): 0:22(12): warning: extension `GL_EXT_gpu_shader4' unsupported in fragment shader<br>0:130(12): error: `gl_PrimitiveID' undeclared<br>0:130(12): error: operands to arithmetic operators must be numeric<br>0:130(12): error: operands to arithmetic operators must be numeric<br>0:131(28): error: operator '%' is reserved in GLSL 1.10 (GLSL 1.30 or GLSL ES 3.00 required)<br>0:131(22): error: cannot construct `float' from a non-numeric data type<br>0:131(22): error: operands to arithmetic operators must be numeric<br>0:131(17): error: cannot construct `vec4' from a non-numeric data type<br><br>Cheers,<br><br><span class="im">--</span><br><span class="im">David Trudgian Ph.D.</span><br><span class="im">Computational Scientist, BioHPC</span><br><span class="im">UT Southwestern Medical Center</span><br><span class="im">Dallas, TX 75390-9039</span><br><span class="im">Tel: <a href="tel:%28214%29%20648-4833">(214) 648-4833</a></span></p><div><div><p class="MsoNormal" style="margin-bottom:12.0pt">-----Original Message-----<br>From: David Trudgian<br>Sent: Thursday, September 10, 2015 10:13 AM<br>To: Aashish Chaudhary <<a href="mailto:aashish.chaudhary@kitware.com">aashish.chaudhary@kitware.com</a>><br>Cc: Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>>; ParaView list <<a href="mailto:paraview@paraview.org">paraview@paraview.org</a>><br>Subject: RE: [Paraview] Volume Rendering 17GB 8.5 billion cell volume<br><br>Aashish,<br><br>(sorry - didn't hit reply-all first time)<br><br>> Would it be possible for you to try OpenGL2 backend?<br><br>Yes - I can try this, but probably next week. I just change VTK_RENDERING_BACKENDS? Do you know if OSMESA has to be built with any particularly flags itself?<br><br>Thanks,<br><br>DT<br><br><br>________________________________________<br>From: Aashish Chaudhary [<a href="mailto:aashish.chaudhary@kitware.com">aashish.chaudhary@kitware.com</a>]<br>Sent: Thursday, September 10, 2015 9:59 AM<br>To: David Trudgian<br>Cc: Berk Geveci; ParaView list<br>Subject: Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume<br><br>Thanks Dave. Haven' t looked at your email in detail (will do in a moment) but another thought would be some sort of limit we are hitting on the indices (MAX_INT or MAX_<TYPE>) being used when dealing with very large dataset such as yours.<br><br>Would it be possible for you to try OpenGL2 backend?<br><br>- Aashish<br><br>On Thu, Sep 10, 2015 at 10:55 AM, David Trudgian <<a href="mailto:david.trudgian@utsouthwestern.edu">david.trudgian@utsouthwestern.edu</a><mailto:<a href="mailto:david.trudgian@utsouthwestern.edu">david.trudgian@utsouthwestern.edu</a>>> wrote:<br>Berk (and others), thanks for your replies!<br><br>> This is pretty awesome. I am assuming that this has something to do<br>> with things not fitting on the GPU memory or exceeding some texture<br>> memory limitation. Can you provide some more details?<br><br>Sure - thanks for your help.<br><br>> * Which version of ParaView are you using?<br><br>This is with Paraview 4.3.1<br><br>> * It sounds like you have multiple GPUs and multiple nodes. What is<br>> the setup? Are you running in parallel with MPI?<br><br>Have tried in two ways, both are using MPI (OpenMPI/1.8.3 on an InfiniBand FDR<br>network):<br><br>Setup 1) Paraview 4.3.1 pvserver is running with MPI across multiple cluster nodes, each with a Tesla K20 GPU. Only up to 4 nodes total, each one has a single Tesla K20. Have used various numbers of MPI tasks. The machines are 16 physical cores, with hyper-threading on for 32 logical cores. 256GB RAM and the Tesla K20 has 5GB.<br><br>... when this didn't work we did suspect out of GPU memory. Since we have a limited number of GPU nodes then decided to try the CPU approach...<br><br>Setup 2) Paraview 4.3.1 rebuilt with OSMESA support, to run pvserver on a larger number of cluster nodes without any GPUs. These are 16 or 24 core machines with 128/256/384GB RAM. Tried various numbers of nodes (up to 16) and MPI tasks per node, allowing for OSMESA threading per the docs/graphs on the Paraview wiki page.<br><br>Watching the pvserver processes when running across 16 nodes I wasn't seeing more than ~2GB RAM usage per process. Across 16 nodes I ran with 8 tasks per node, so at 2GB each this is well under the minimum of 128GB RAM per node.<br><br>> * If you are running parallel with MPI and you have multiple GPUs per<br>> node, did you setup the DISPLAYs to leverage the GPUs?<br><br>As above, only 1 GPU per node, or 0 when switched to the OSMESA approach to try with across more nodes.<br><br>As mentioned before, we can view a smaller version of the data without issue on both GPU and OSMESA setups. I just opened a 4GB version (approx 25% of full size) using the OSMESA setup on a single node (8 MPI tasks) without issue. The responsiveness is really great - but the 16GB file is a no-go even scaling up across 16 nodes. The VTI itself seems fine, as slices and surface look as expected.<br><br>Thanks again for any and all suggestions!<br><br>DT<br><br>> On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <<br>> <a href="mailto:david.trudgian@utsouthwestern.edu">david.trudgian@utsouthwestern.edu</a><mailto:<a href="mailto:david.trudgian@utsouthwestern.edu">david.trudgian@utsouthwestern.edu</a>>> wrote:<br>><br>> > Hi,<br>> ><br>> > We have been experimenting with using Paraview to display very<br>> > volumes from very large TIFF stacks generated by whole-brain<br>> > microscopy equipment. The test stack has dimensions of<br>> > 5,368x10,695x150. Stack is assembled in ImageJ from individual<br>> > TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti<br>> > for convenience. Can view slices fine in standalone paraview client<br>> > on a 256GB machine.<br>> ><br>> > When we attempt volume rendering on this data across multiple nodes<br>> > with MPI nothing appears in the client. Surface view works as<br>> > expected. On switching to volume rendering the client's display will<br>> > show nothing. There are no messages from the client or servers - no<br>> > output.<br>> ><br>> > This is happening when running pvserver across GPU nodes with NVIDIA<br>> > Tesla cards, or using CPU only with OSMESA. pvserver memory usage is<br>> > well below what we have on the nodes - no memory warnings/errors.<br>> ><br>> > Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB<br>> > then we can get working volume rendering. The 17GB never works<br>> > regardless of scaling nodes/mpi processes. The 4/9GB will work on 1<br>> > or 2 nodes.<br>> ><br>> > Am confused by the lack of rendering, as we don't have memory<br>> > issues, or an other messages at all. Am wondering if there are any<br>> > inherent limitation, or I'm missing something stupid.<br>> ><br>> > Thanks,<br>> ><br>> > Dave Trudgian<br>> ><br>> ><br>> > _______________________________________________<br>> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><<a href="http://www.kitware.com" target="_blank">http://www.kitware.com</a>><br>> ><br>> > Visit other Kitware open-source projects at<br>> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>> ><br>> > Please keep messages on-topic and check the ParaView Wiki at:<br>> > <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a><br>> ><br>> > Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a><br>> ><br>> > Follow this link to subscribe/unsubscribe:<br>> > <a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>> ><br><br>--<br>David Trudgian Ph.D.<br>Computational Scientist, BioHPC<br>UT Southwestern Medical Center<br>Dallas, TX 75390-9039<br>Tel: <a href="tel:%28214%29%20648-4833">(214) 648-4833</a><<a href="tel:%28214%29%20648-4833">tel:%28214%29%20648-4833</a>><br><br><br>_______________________________________________<br>Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><<a href="http://www.kitware.com" target="_blank">http://www.kitware.com</a>><br><br>Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br><br>Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a><br><br>Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a><br><br>Follow this link to subscribe/unsubscribe:<br><a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br><br><br><br>--<br>| Aashish Chaudhary<br>| Technical Leader<br>| Kitware Inc.<br>| <a href="http://www.kitware.com/company/team/chaudhary.html" target="_blank">http://www.kitware.com/company/team/chaudhary.html</a><br><br>________________________________<br><br>UT Southwestern<br><br><br>Medical Center<br><br><br><br>The future of medicine, today.</p></div></div></div><p class="MsoNormal"> </p></div></div></div></body></html>