[Paraview] [EXTERNAL] Memory explosion and strange behavior -- Linux -- 'clip' -- 360 MB file needs 60 GB ??

[Robert.Atwood at diamond.ac.uk]

Im sure that should be possible somehow, maybe not the specific data I mentioned which is an external user's unpublished work so far, but we can just get some more if there's some useful collaboration to be had. 

I've been 'playing' with Paraview since the CCPi and Harwell Imaging Partnership meeting showed the latest release and also because the latest de-bugging of our central file system is arguing against allowing the 'visualization workstation' being directly connected to the central storage for technical reasons, thus the client/server model becomes very attractive, allowing the machine that actually reads data to be inside the computer cluster rack, while another machine displays the result , I think this require our cluster admins to do something-or-other , so I need to have some good argument that it will indeed do what we want (which also means figuring out what we actually want) to get that task added to their already-lengthy list. 

In the first instance, we want external facility users to at least be able to do SOMETHING with their data while still at their experiment. So it takes about 1/2 hour to collect the raw data, another 40 min to reconstruct the 160 GB 3-d image , then what??? 






> -----Original Message-----
> From: Marcus D. Hanwell [mailto:marcus.hanwell at kitware.com]
> Sent: 03 June 2014 17:19
> To: Atwood, Robert (DLSLtd,RAL,SCI)
> Cc: Alan W Scott; ParaView
> Subject: Re: [Paraview] [EXTERNAL] Memory explosion and strange
> behavior -- Linux -- 'clip' -- 360 MB file needs 60 GB ??
> 
> On Mon, Jun 2, 2014 at 1:31 PM,  <Robert.Atwood at diamond.ac.uk> wrote:
> >
> > Aren't you glad I already reduced the data to 8bit grayscale and
> clipped out only 850x850x500 out of 4008x4008x2672 32-bit floating
> point values??? They keep building bigger cameras ...
> >
> > Did I mention it is part of a time-series?
> 
> This sounds like really interesting data! We are working on a Phase I
> SBIR project looking at materials TEM tomography data in the range of
> 1024^3, and looking at strategies to accelerate marching cubes,
> clipping, etc. This is in a custom ParaView application, and there may
> be some outcomes that help you too. Is it possible to share any samples
> of this data, and operations you are most interested in performing on
> it?
> >
> >>Some of the greatest minds in the world are still trying to figure
> out how to visualize data of this size.
> >
> > Isn't that you guys? That's why I've tried Paraview! :-)  or is it
> Ajay?
> >
> > There is indeed a cluster lurking, once we figure out how to
> configure
> > it. It has several (50?)  GPU nodes too. That's how we get such
> images
> > reconstructed :-)
> >
> That should help!
> 
> Thanks,
> 
> Marcus