[Paraview] PV crashed when processed big(7GB) VTK file with multi-core enabled

[Burlen Loring]

KuN,

Normally as datasets get larger you'd  want to use multiple nodes each 
with multiple cores. 7GB unstructured data is maybe big for one node. I 
think a 64GB node should be able to handle it, although you're in 
territory where you may want more nodes depending on what you're doing 
in PV.

One issue may arise from the reader and how the data is stored on disk. 
The legacy VTK format is the worst, the XML format can be very good 
although with the XML formats I think it works best when the data is 
partitioned into chunks on disk. Which format are you using and if it's 
the XML format how many partitions on disk are there?

A related issue is that MPI used int's in its API. This sometimes is a 
problem when dealing with really large data. This won't be an issue if 
your data is in the XML format and partitioned into 2G or smaller chunks.

In PV 4.1 the command line option --enable-bt will produce a stack trace 
on stderr if PV crashes or aborts, you'd need to compile with debug 
symbols, CMAKE_BUILD_TYPE=Debug or RelWithDebInfo. This may shed light 
on what's crashing and why. In earlier versions backtrace could be 
enabled from the memory inspector panel but didn't catch abort and will 
not help for debugging MPI errors.

Burlen

On 12/13/2013 03:50 AM, KuN wrote:
> Hi all,
>
> I've been using PV for a while. Just recently I start to feed 
> hi-resoultion LiDAR to my model. The output file is huge, 7gb with 23 
> million node and even more triangles. Of course my workstation can't 
> render this kind of data (I process them on a Amazon EC2 with 64GB 
> ram), so I opened another EC2 and installed a new PV with MPI&OSMESA 
> enabled. The remote server works well with small files. But it crashed 
> with bigger one while multi-cores were used.
>
>     "BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES"
>     "EXIT CODE :134"
>
>
> It did work with one core only, but the speed is unbearable. I also 
> noticed that when it worked (with one core), it only used about 11.1 
> GB of ram while the release ver. used 13.6GB in my Ubuntu 12.04 before 
> crashing.
>
> Sorry I can't provide anymore useful information. But I'd like to know 
> how you guys work with this kind of big VTK file.
>
> Thanks and have a good weekend,
>
> KuN
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20131213/e1104eaa/attachment.htm>