[Paraview] Volume Rendering 17GB 8.5 billion cell volume

[David Trudgian]

Berk (and others), thanks for your replies!

> This is pretty awesome. I am assuming that this has something to do with
> things not fitting on the GPU memory or exceeding some texture memory
> limitation. Can you provide some more details?

Sure - thanks for your help.

> * Which version of ParaView are you using?

This is with Paraview 4.3.1

> * It sounds like you have multiple GPUs and multiple nodes. What is the
> setup? Are you running in parallel with MPI?

Have tried in two ways, both are using MPI (OpenMPI/1.8.3 on an InfiniBand FDR
network):

Setup 1) Paraview 4.3.1 pvserver is running with MPI across multiple cluster
nodes, each with a Tesla K20 GPU. Only up to 4 nodes total, each one has a
single Tesla K20. Have used various numbers of MPI tasks. The machines are 16
physical cores, with hyper-threading on for 32 logical cores. 256GB RAM and the
Tesla K20 has 5GB.

... when this didn't work we did suspect out of GPU memory. Since we have a
limited number of GPU nodes then decided to try the CPU approach...

Setup 2) Paraview 4.3.1 rebuilt with OSMESA support, to run pvserver on a larger
number of cluster nodes without any GPUs. These are 16 or 24 core machines with
128/256/384GB RAM. Tried various numbers of nodes (up to 16) and
MPI tasks per node, allowing for OSMESA threading per the docs/graphs on the
Paraview wiki page.

Watching the pvserver processes when running across 16 nodes I wasn't seeing
more than ~2GB RAM usage per process. Across 16 nodes I ran with 8 tasks per
node, so at 2GB each this is well under the minimum of 128GB RAM per node.

> * If you are running parallel with MPI and you have multiple GPUs per node,
> did you setup the DISPLAYs to leverage the GPUs?

As above, only 1 GPU per node, or 0 when switched to the OSMESA approach to try
with across more nodes.

As mentioned before, we can view a smaller version of the data without issue on
both GPU and OSMESA setups. I just opened a 4GB version (approx 25% of full size)
using the OSMESA setup on a single node (8 MPI tasks) without issue. The
responsiveness is really great - but the 16GB file is a no-go even scaling up 
across 16 nodes. The VTI itself seems fine, as slices and surface look as 
expected.

Thanks again for any and all suggestions!

DT

> On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <
> david.trudgian at utsouthwestern.edu> wrote:
> 
> > Hi,
> >
> > We have been experimenting with using Paraview to display very volumes
> > from very
> > large TIFF stacks generated by whole-brain microscopy equipment. The test
> > stack
> > has dimensions of 5,368x10,695x150. Stack is assembled in ImageJ from
> > individual
> > TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti for
> > convenience. Can view slices fine in standalone paraview client on a 256GB
> > machine.
> >
> > When we attempt volume rendering on this data across multiple nodes with
> > MPI
> > nothing appears in the client. Surface view works as expected. On
> > switching to
> > volume rendering the client's display will show nothing. There are no
> > messages
> > from the client or servers - no output.
> >
> > This is happening when running pvserver across GPU nodes with NVIDIA Tesla
> > cards, or using CPU only with OSMESA. pvserver memory usage is well below
> > what
> > we have on the nodes - no memory warnings/errors.
> >
> > Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB then
> > we can
> > get working volume rendering. The 17GB never works regardless of scaling
> > nodes/mpi processes. The 4/9GB will work on 1 or 2 nodes.
> >
> > Am confused by the lack of rendering, as we don't have memory issues, or an
> > other messages at all. Am wondering if there are any inherent limitation,
> > or I'm
> > missing something stupid.
> >
> > Thanks,
> >
> > Dave Trudgian
> >
> >
> > _______________________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Please keep messages on-topic and check the ParaView Wiki at:
> > http://paraview.org/Wiki/ParaView
> >
> > Search the list archives at: http://markmail.org/search/?q=ParaView
> >
> > Follow this link to subscribe/unsubscribe:
> > http://public.kitware.com/mailman/listinfo/paraview
> >

-- 
David Trudgian Ph.D.
Computational Scientist, BioHPC
UT Southwestern Medical Center
Dallas, TX 75390-9039
Tel: (214) 648-4833