[Paraview] Volume Rendering 17GB 8.5 billion cell volume

Thu Sep 10 11:12:54 EDT 2015

Aashish,

(sorry - didn't hit reply-all first time)

> Would it be possible for you to try OpenGL2 backend?

Yes - I can try this, but probably next week. I just change VTK_RENDERING_BACKENDS? Do you know if OSMESA has to be built with any particularly flags itself?

Thanks,

DT

________________________________________
From: Aashish Chaudhary [aashish.chaudhary at kitware.com]
Sent: Thursday, September 10, 2015 9:59 AM
To: David Trudgian
Cc: Berk Geveci; ParaView list
Subject: Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume

Thanks Dave. Haven' t looked at your email in detail (will do in a moment) but another thought would be some sort of limit we are hitting on the indices (MAX_INT or MAX_<TYPE>) being used when dealing with very large dataset such as yours.

Would it be possible for you to try OpenGL2 backend?

- Aashish

On Thu, Sep 10, 2015 at 10:55 AM, David Trudgian <david.trudgian at utsouthwestern.edu<mailto:david.trudgian at utsouthwestern.edu>> wrote:
Berk (and others), thanks for your replies!

> This is pretty awesome. I am assuming that this has something to do with
> things not fitting on the GPU memory or exceeding some texture memory
> limitation. Can you provide some more details?

Sure - thanks for your help.

> * Which version of ParaView are you using?

This is with Paraview 4.3.1

> * It sounds like you have multiple GPUs and multiple nodes. What is the
> setup? Are you running in parallel with MPI?

Have tried in two ways, both are using MPI (OpenMPI/1.8.3 on an InfiniBand FDR
network):

Setup 1) Paraview 4.3.1 pvserver is running with MPI across multiple cluster
nodes, each with a Tesla K20 GPU. Only up to 4 nodes total, each one has a
single Tesla K20. Have used various numbers of MPI tasks. The machines are 16
physical cores, with hyper-threading on for 32 logical cores. 256GB RAM and the
Tesla K20 has 5GB.

... when this didn't work we did suspect out of GPU memory. Since we have a
limited number of GPU nodes then decided to try the CPU approach...

Setup 2) Paraview 4.3.1 rebuilt with OSMESA support, to run pvserver on a larger
number of cluster nodes without any GPUs. These are 16 or 24 core machines with
128/256/384GB RAM. Tried various numbers of nodes (up to 16) and
MPI tasks per node, allowing for OSMESA threading per the docs/graphs on the
Paraview wiki page.

Watching the pvserver processes when running across 16 nodes I wasn't seeing
more than ~2GB RAM usage per process. Across 16 nodes I ran with 8 tasks per
node, so at 2GB each this is well under the minimum of 128GB RAM per node.

> * If you are running parallel with MPI and you have multiple GPUs per node,
> did you setup the DISPLAYs to leverage the GPUs?

As above, only 1 GPU per node, or 0 when switched to the OSMESA approach to try
with across more nodes.

As mentioned before, we can view a smaller version of the data without issue on
both GPU and OSMESA setups. I just opened a 4GB version (approx 25% of full size)
using the OSMESA setup on a single node (8 MPI tasks) without issue. The
responsiveness is really great - but the 16GB file is a no-go even scaling up
across 16 nodes. The VTI itself seems fine, as slices and surface look as
expected.

Thanks again for any and all suggestions!

DT

> On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <
> david.trudgian at utsouthwestern.edu<mailto:david.trudgian at utsouthwestern.edu>> wrote:
>
> > Hi,
> >
> > We have been experimenting with using Paraview to display very volumes
> > from very
> > large TIFF stacks generated by whole-brain microscopy equipment. The test
> > stack
> > has dimensions of 5,368x10,695x150. Stack is assembled in ImageJ from
> > individual
> > TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti for
> > convenience. Can view slices fine in standalone paraview client on a 256GB
> > machine.
> >
> > When we attempt volume rendering on this data across multiple nodes with
> > MPI
> > nothing appears in the client. Surface view works as expected. On
> > switching to
> > volume rendering the client's display will show nothing. There are no
> > messages
> > from the client or servers - no output.
> >
> > This is happening when running pvserver across GPU nodes with NVIDIA Tesla
> > cards, or using CPU only with OSMESA. pvserver memory usage is well below
> > what
> > we have on the nodes - no memory warnings/errors.
> >
> > Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB then
> > we can
> > get working volume rendering. The 17GB never works regardless of scaling
> > nodes/mpi processes. The 4/9GB will work on 1 or 2 nodes.
> >
> > Am confused by the lack of rendering, as we don't have memory issues, or an
> > other messages at all. Am wondering if there are any inherent limitation,
> > or I'm
> > missing something stupid.
> >
> > Thanks,
> >
> > Dave Trudgian
> >
> >
> > _______________________________________________
> > Powered by www.kitware.com<http://www.kitware.com>
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Please keep messages on-topic and check the ParaView Wiki at:
> > http://paraview.org/Wiki/ParaView
> >
> > Search the list archives at: http://markmail.org/search/?q=ParaView
> >
> > Follow this link to subscribe/unsubscribe:
> > http://public.kitware.com/mailman/listinfo/paraview
> >

--
David Trudgian Ph.D.
Computational Scientist, BioHPC
UT Southwestern Medical Center
Dallas, TX 75390-9039
Tel: (214) 648-4833<tel:%28214%29%20648-4833>

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView

Search the list archives at: http://markmail.org/search/?q=ParaView

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/paraview

--
| Aashish Chaudhary
| Technical Leader
| Kitware Inc.
| http://www.kitware.com/company/team/chaudhary.html

________________________________

UT Southwestern

Medical Center

The future of medicine, today.