[Paraview] 400GB dataset out of memory on 1k cores?

[Berk Geveci]

Hi Anton,

Let's debug this a big. I suspect that the thresholds are the issue here.
Does this script work?

ext1    = 4640       # data extent along 1
ext2    = 4650       # data extent along 2
ext3    = 4650       # data extent along 3
ffile   = "./large/1000/zf5.raw" # fracture file
imsize1 = 1200         # size, in pixels, of the resulting image along 1
imsize2 = 1200         # size, in pixels, of the resulting image along 2

cor1 = 0.5 * ext1
cor2 = 0.5 * ext2
cor3 = 0.5 * ext3

from paraview.simple import *

# the extents start from zero, so need to lower
# the upper extents by 1
cracks = ImageReader( FilePrefix= ffile )
cracks.DataExtent=[ 0, ext1-1, 0, ext2-1, 0, ext3-1 ]
cracks.DataByteOrder = 'LittleEndian'
cracks.DataScalarType = 'int'

RenderView1 = GetRenderView()
DataRepresentation1 = Show()

If this works, let's discuss how to avoid using Threshold but get the same
result. The issue with Threshold is that it creates a volume as an
unstructured grid, which is heavy-weight. For example, a ~40KB volume (one
float variable) turns into a ~800KB if I extract all of it as an
unstructured grid. It seems like all you need is to extract the interfaces
between the grains, correct?

Best,
-berk

On Mon, Sep 21, 2015 at 10:50 AM, Anton Shterenlikht <mexas at bris.ac.uk>
wrote:

> >From mexas Mon Sep 21 14:29:16 2015
> >To: paraview at paraview.org
> >Subject: 400GB dataset out of memory on 1k cores?
> >Reply-To: mexas at bris.ac.uk
> >
> >I get OOM on a ~400GB raw dataset
> >that I process with a script below.
> >I use 24 core nodes with 64GB per node.
> >I tried using up to 1008 nodes.
>
> error... 1008 cores (42 nodes).
>
> Anton
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20150921/5b2e5efc/attachment.html>