[Paraview] Streaming (was: Problem with big files)

[Kevin H. Hobbs]

On Thu, 2006-03-30 at 01:52 +0200, Christoph Moder wrote:
> > Oh, hehe. That wasn't meant to look like a type.  What I meant was that
> > LONG before a program starts to behave badly because it's using up the
> > address space or all the RAM + swap it starts to pay off to use
> > streaming.
> 
> I didn't understand the sentence because I had no idea what you mean by 
> "streaming abilities" and that it is a VTK feature. Now I have seen in 
> the Paraview FAQ that this must be a kind of filter that can be added to 
> a VTK pipeline, but unfortunately I have no idea how to do that. I would 
> like to deal with Tcl, but I don't know how to start...
> 
> Christoph

Oh OK.  Streaming large data sets keeps the overall memory usage down. I
don't know how to use it in Paraview but I do whatever I can beforehand
in VTK. If every filter supports streaming than telling the last one to
divide the pipeline into a number pieces divides the total amount of
memory used at once. The file reader and any filter only load enough
data to process the piece.

Thinking about streaming matters for Paraview even when run in parallel
because not all readers can load only a part of a data set. If every
process on the cluster tries to load the whole thing and throw out what
it does not need then there is a lot of waste. I saw a big speedup by
switching to the XML file types broken into pieces. Your mileage will of
course vary.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : http://public.kitware.com/pipermail/paraview/attachments/20060329/62222d4c/attachment.pgp