[Paraview] Volume Rendering 17GB 8.5 billion cell volume

Mon Sep 14 11:28:22 EDT 2015

Hey,

did you find a solution for this? I have a little smaller data set of
just around 3.4 billion cells in a 3D unstructured grid volume, climate
simulation data. It reads and displays the 2D slices (22 million cells
each) fine and quiet fast, but if I want to load all 150 slices, I get
this error message:

ERROR: In
/tmp/xas/build/paraview_paraview_4.3.1_default_gcc48/src/ParaView/VTK/Common/Core/vtkDataArrayTemplate.txx,
line 308
vtkIdTypeArray (0x3cecaf0): Unable to allocate 33587986431 elements of
size 8 bytes.

ParaView consumes around 2/3 of the systems memory by this time.

Cheers,
Niklas

> Aashish,
>
> (sorry - didn't hit reply-all first time)
>
>> Would it be possible for you to try OpenGL2 backend?
> Yes - I can try this, but probably next week. I just change VTK_RENDERING_BACKENDS? Do you know if OSMESA has to be built with any particularly flags itself?
>
> Thanks,
>
> DT
>
>
> ________________________________________
> From: Aashish Chaudhary [aashish.chaudhary at kitware.com]
> Sent: Thursday, September 10, 2015 9:59 AM
> To: David Trudgian
> Cc: Berk Geveci; ParaView list
> Subject: Re: [Paraview] Volume Rendering 17GB 8.5 billion cell volume
>
> Thanks Dave. Haven' t looked at your email in detail (will do in a moment) but another thought would be some sort of limit we are hitting on the indices (MAX_INT or MAX_<TYPE>) being used when dealing with very large dataset such as yours.
>
> Would it be possible for you to try OpenGL2 backend?
>
> - Aashish
>
> On Thu, Sep 10, 2015 at 10:55 AM, David Trudgian <david.trudgian at utsouthwestern.edu<mailto:david.trudgian at utsouthwestern.edu>> wrote:
> Berk (and others), thanks for your replies!
>
>> This is pretty awesome. I am assuming that this has something to do with
>> things not fitting on the GPU memory or exceeding some texture memory
>> limitation. Can you provide some more details?
> Sure - thanks for your help.
>
>> * Which version of ParaView are you using?
> This is with Paraview 4.3.1
>
>> * It sounds like you have multiple GPUs and multiple nodes. What is the
>> setup? Are you running in parallel with MPI?
> Have tried in two ways, both are using MPI (OpenMPI/1.8.3 on an InfiniBand FDR
> network):
>
> Setup 1) Paraview 4.3.1 pvserver is running with MPI across multiple cluster
> nodes, each with a Tesla K20 GPU. Only up to 4 nodes total, each one has a
> single Tesla K20. Have used various numbers of MPI tasks. The machines are 16
> physical cores, with hyper-threading on for 32 logical cores. 256GB RAM and the
> Tesla K20 has 5GB.
>
> ... when this didn't work we did suspect out of GPU memory. Since we have a
> limited number of GPU nodes then decided to try the CPU approach...
>
> Setup 2) Paraview 4.3.1 rebuilt with OSMESA support, to run pvserver on a larger
> number of cluster nodes without any GPUs. These are 16 or 24 core machines with
> 128/256/384GB RAM. Tried various numbers of nodes (up to 16) and
> MPI tasks per node, allowing for OSMESA threading per the docs/graphs on the
> Paraview wiki page.
>
> Watching the pvserver processes when running across 16 nodes I wasn't seeing
> more than ~2GB RAM usage per process. Across 16 nodes I ran with 8 tasks per
> node, so at 2GB each this is well under the minimum of 128GB RAM per node.
>
>> * If you are running parallel with MPI and you have multiple GPUs per node,
>> did you setup the DISPLAYs to leverage the GPUs?
> As above, only 1 GPU per node, or 0 when switched to the OSMESA approach to try
> with across more nodes.
>
> As mentioned before, we can view a smaller version of the data without issue on
> both GPU and OSMESA setups. I just opened a 4GB version (approx 25% of full size)
> using the OSMESA setup on a single node (8 MPI tasks) without issue. The
> responsiveness is really great - but the 16GB file is a no-go even scaling up
> across 16 nodes. The VTI itself seems fine, as slices and surface look as
> expected.
>
> Thanks again for any and all suggestions!
>
> DT
>
>> On Wed, Sep 9, 2015 at 5:00 PM, David Trudgian <
>> david.trudgian at utsouthwestern.edu<mailto:david.trudgian at utsouthwestern.edu>> wrote:
>>
>>> Hi,
>>>
>>> We have been experimenting with using Paraview to display very volumes
>>> from very
>>> large TIFF stacks generated by whole-brain microscopy equipment. The test
>>> stack
>>> has dimensions of 5,368x10,695x150. Stack is assembled in ImageJ from
>>> individual
>>> TIFFs, exported as a RAW and loaded into paraview. Saved as a .vti for
>>> convenience. Can view slices fine in standalone paraview client on a 256GB
>>> machine.
>>>
>>> When we attempt volume rendering on this data across multiple nodes with
>>> MPI
>>> nothing appears in the client. Surface view works as expected. On
>>> switching to
>>> volume rendering the client's display will show nothing. There are no
>>> messages
>>> from the client or servers - no output.
>>>
>>> This is happening when running pvserver across GPU nodes with NVIDIA Tesla
>>> cards, or using CPU only with OSMESA. pvserver memory usage is well below
>>> what
>>> we have on the nodes - no memory warnings/errors.
>>>
>>> Data is about 17GB, 8 billion cells. If we downsize to ~4GB or ~9GB then
>>> we can
>>> get working volume rendering. The 17GB never works regardless of scaling
>>> nodes/mpi processes. The 4/9GB will work on 1 or 2 nodes.
>>>
>>> Am confused by the lack of rendering, as we don't have memory issues, or an
>>> other messages at all. Am wondering if there are any inherent limitation,
>>> or I'm
>>> missing something stupid.
>>>
>>> Thanks,
>>>
>>> Dave Trudgian
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com<http://www.kitware.com>
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>
> --
> David Trudgian Ph.D.
> Computational Scientist, BioHPC
> UT Southwestern Medical Center
> Dallas, TX 75390-9039
> Tel: (214) 648-4833<tel:%28214%29%20648-4833>
>
>
> _______________________________________________
> Powered by www.kitware.com<http://www.kitware.com>
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
>
>
> --
> | Aashish Chaudhary
> | Technical Leader
> | Kitware Inc.
> | http://www.kitware.com/company/team/chaudhary.html
>
> ________________________________
>
> UT Southwestern
>
>
> Medical Center
>
>
>
> The future of medicine, today.
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4916 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://public.kitware.com/pipermail/paraview/attachments/20150914/d86eb832/attachment.bin>