<div dir="ltr">> No GL error this time I am assuming? Just segfault?<div><br></div><div>Yes. With the stack totally hosed. I also see Valgrind errors deep in Mesa stack.<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 21, 2015 at 9:15 AM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Wed, Oct 21, 2015 at 7:52 AM, Berk Geveci <span dir="ltr"><<a href="mailto:berk.geveci@kitware.com" target="_blank">berk.geveci@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Yeah, it is not that easy. If you compile Mesa with llvmpipe, the limit is in something like gallium/drivers/llvmpipe/lp_limits.h. Otherwise it doesn't work. However, I had crashes when I increased that to something like 8 GBs and then volume rendered something large. </div></blockquote><div><br></div></span><div>No GL error this time I am assuming? Just segfault?</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">It also looks like other drivers have their own limits that are pretty small. I will talk to some folks doing work with Mesa about this. Hopefully, we can address it in the upcoming OpenSWR driver. We'll have to support streaming for other drivers though...</div></blockquote><div><br></div></span><div>Absolutely. </div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><span><font color="#888888"><div><br></div><div>-berk<br><div><br></div></div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 20, 2015 at 5:55 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Here it is. It would be great if someone else try it as well: <div><br><div><div style="font-size:16px">----Steps----</div><div style="font-size:16px">1. In src/mesa/main/config.h, there should MAX_TEXTURE_MBYTES defined. I believe by default it was 1024 Mbytes. Please change it to 4096 or something higher. </div><div style="font-size:16px"><br></div><div style="font-size:16px">2. Then Compile and Install MESA again (do not forget to set the <span style="color:rgb(0,0,0);font-family:'Lucida Grande',Geneva,Arial,Verdana,sans-serif;font-size:14px">MESA_GL_VERSION_OVERRIDE</span> to 3.2). </div><div style="font-size:16px"><br></div><div style="font-size:16px">3. Compile paraview again (server)</div><div style="font-size:16px"><br></div></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 20, 2015 at 5:48 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Berk, <br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Oct 20, 2015 at 4:00 PM, Berk Geveci <span dir="ltr"><<a href="mailto:berk.geveci@kitware.com" target="_blank">berk.geveci@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi folks,<div><br></div><div>I wanted to close the loop on this. Here are my findings:</div><div><br></div><div>* ParaView master (4.4 should also do) + OpenGL2 + NVIDIA Tesla w 12 GB memory: I verified that I can volume render data up to the capacity of the card. I could volume render a 1400x1400x1400 volume of floats.</div><div><br></div><div>* ParaView master (4.4 should also do) + OpenGL2 + Mesa (OSMesa 11, llvmpipe, swrast): Mesa has some fairly small limits on 3D texture size, which is what we use for volume rendering. So, ~ 1000x1000x1000 will be the upper end of what can be done for now. In time, we will implement multiple textures / streaming to enable rendering of larger volumes.</div></div></blockquote><div><br></div></span><div>Did you see my other email? You can change the default for OSMesa. I sent it last week. </div><span><font color="#888888"><div><br></div><div>- Aashish </div></font></span><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Best,</div><div>-berk</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 28, 2015 at 11:00 AM, David Trudgian <span dir="ltr"><<a href="mailto:David.Trudgian@utsouthwestern.edu" target="_blank">David.Trudgian@utsouthwestern.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Berk,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks very much for looking into this. Look forward to trying things out whenever they’re ready.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">DT<u></u><u></u></span></p><span>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:black">--</span><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br>
David Trudgian Ph.D.<br>
Computational Scientist, BioHPC<br>
UT Southwestern Medical Center<br>
Dallas, TX 75390-9039<br>
Tel: <a href="tel:%28214%29%20648-4833" value="+12146484833" target="_blank">(214) 648-4833</a></span><span style="font-size:11pt;font-family:Consolas;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
</span><p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Berk Geveci [mailto:<a href="mailto:berk.geveci@kitware.com" target="_blank">berk.geveci@kitware.com</a>]
<br>
<b>Sent:</b> Monday, September 28, 2015 9:58 AM</span></p><div><div><br>
<b>To:</b> David Trudgian <David.Trudgian@UTSouthwestern.edu><br>
<b>Cc:</b> ParaView Mailing List <<a href="mailto:paraview@paraview.org" target="_blank">paraview@paraview.org</a>><br>
<b>Subject:</b> Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume<u></u><u></u></div></div><p></p><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi David,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have been trying to find some cycles to check this out myself with ParaView 4.4. Thanks to hardware issues (i.e. my big workstation's disk dying), I haven't been able to. Good news is that I found issues with OSMesa + OpenGL2 that we
are working through. Give me another 1-1.5 weeks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-berk<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Sep 28, 2015 at 10:46 AM, David Trudgian <<a href="mailto:David.Trudgian@utsouthwestern.edu" target="_blank">David.Trudgian@utsouthwestern.edu</a>> wrote:<u></u><u></u></p>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Berk,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Finally managed to grab an allocation of some Tesla K40 nodes on our cluster, to check GPU rendering
of the full 17GB file with 2 x 12GB GPUs. I see the same thing as I did with OSMesa rendering.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The 9GB downsampled version works great, across 2 nodes both with a single K40. Go up to the 17GB original
file and nothing is rendered, no errors. Same behavior with OPENGL or OPENGL2 backends.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">This is all on paraview 4.3.1 still – I need to find time to build OSMesa / MPI versions of 4.4 here.
But, does 4.4. have any fixes that would be expected to affect this?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:black">--</span><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br>
David Trudgian Ph.D.<br>
Computational Scientist, BioHPC<br>
UT Southwestern Medical Center<br>
Dallas, TX 75390-9039<br>
Tel: <a href="tel:%28214%29%20648-4833" target="_blank">(214) 648-4833</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Berk Geveci [mailto:<a href="mailto:berk.geveci@kitware.com" target="_blank">berk.geveci@kitware.com</a>]
<br>
<b>Sent:</b> Tuesday, September 15, 2015 2:43 PM<br>
<b>To:</b> David Trudgian <<a href="mailto:David.Trudgian@UTSouthwestern.edu" target="_blank">David.Trudgian@UTSouthwestern.edu</a>><br>
<b>Cc:</b> ParaView Mailing List <<a href="mailto:paraview@paraview.org" target="_blank">paraview@paraview.org</a>><br>
<b>Subject:</b> Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hey David,<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I am hoping to have some time to play around with volume rendering and hopefully tracking this issue, one thing that I wanted to clarify: it sounds from you description that you
have a short (2 byte) value. Is that correct?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-berk<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <<a href="mailto:david.trudgian@utsouthwestern.edu" target="_blank">david.trudgian@utsouthwestern.edu</a>> wrote:<u></u><u></u></p>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<p class="MsoNormal">Hi,<br>
<br>
We have been experimenting with using Paraview to display very volumes from very<br>
large TIFF stacks generated by whole-brain microscopy equipment. The test stack<br>
has dimensions of 5,368x10,695x150. Stack is assembled in ImageJ from individual<br>
TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti for<br>
convenience. Can view slices fine in standalone paraview client on a 256GB machine.<br>
<br>
When we attempt volume rendering on this data across multiple nodes with MPI<br>
nothing appears in the client. Surface view works as expected. On switching to<br>
volume rendering the client's display will show nothing. There are no messages<br>
from the client or servers - no output.<br>
<br>
This is happening when running pvserver across GPU nodes with NVIDIA Tesla<br>
cards, or using CPU only with OSMESA. pvserver memory usage is well below what<br>
we have on the nodes - no memory warnings/errors.<br>
<br>
Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB then we can<br>
get working volume rendering. The 17GB never works regardless of scaling<br>
nodes/mpi processes. The 4/9GB will work on 1 or 2 nodes.<br>
<br>
Am confused by the lack of rendering, as we don't have memory issues, or an<br>
other messages at all. Am wondering if there are any inherent limitation, or I'm<br>
missing something stupid.<br>
<br>
Thanks,<br>
<br>
Dave Trudgian<br>
<br>
<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">
http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">
http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<table border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0in">
<p align="right" style="text-align:right"><b><span style="font-size:17pt;font-family:Helvetica,sans-serif;color:rgb(0,84,160);letter-spacing:-2.25pt">UT</span></b><b><span style="font-size:17pt;font-family:Helvetica,sans-serif;color:rgb(0,84,160)"> Southwestern</span></b>
<u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0in">
<p align="right" style="text-align:right"><span style="font-size:13pt;font-family:Helvetica,sans-serif;color:rgb(102,109,112)">Medical Center</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0in"></td>
</tr>
<tr>
<td style="padding:0in">
<p align="right" style="text-align:right"><span style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(102,109,112)">The future of medicine, today.</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank">http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" rel="noreferrer" target="_blank">http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>
<br></blockquote></div></div></div><br><br clear="all"><span><div><br></div>-- <br><div><div dir="ltr"><font face="trebuchet ms, sans-serif"><i>| Aashish Chaudhary <br>| Technical Leader <br>| Kitware Inc. <br></i></font><div><i><font face="trebuchet ms, sans-serif">| </font><a href="http://www.kitware.com/company/team/chaudhary.html" target="_blank">http://www.kitware.com/company/team/chaudhary.html</a></i></div></div></div>
</span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><font face="trebuchet ms, sans-serif"><i>| Aashish Chaudhary <br>| Technical Leader <br>| Kitware Inc. <br></i></font><div><i><font face="trebuchet ms, sans-serif">| </font><a href="http://www.kitware.com/company/team/chaudhary.html" target="_blank">http://www.kitware.com/company/team/chaudhary.html</a></i></div></div></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><font face="trebuchet ms, sans-serif"><i>| Aashish Chaudhary <br>| Technical Leader <br>| Kitware Inc. <br></i></font><div><i><font face="trebuchet ms, sans-serif">| </font><a href="http://www.kitware.com/company/team/chaudhary.html" target="_blank">http://www.kitware.com/company/team/chaudhary.html</a></i></div></div></div>
</div></div></div></div>
</blockquote></div><br></div></div></div>