[Paraview] Time Parallelism + vtkFileCachePipeline + vtkNetCDFReader

[Berk Geveci]

A few things:

* ParaView caching geometry when generating animations. Some of your
works sounds similar and it might be a good idea to merge the two
efforts. Take a look at vtkPVUpdateSuppressor.

* This sort of time support is being discussed, specially by Sandia
developers. I believe there will be some discussion at Vis '05 VTK
BOF.

* The new pipeline is great to do things like this. The composite
pipeline is a good example of how the execution model can be modified
by sub-classing the existing pipeline objects.

* The multi-block datasets are a good way of storing multiple
timesteps. Specially for filters that might require multiple steps
(streaklines, particle paths, any integration filter, histograms)

On 9/24/05, John Biddiscombe <biddisco at cscs.ch> wrote:
> I've been working on parallelising vtkNetCDFReader for use in ParaView
> and have some questions regarding pieces/pipelines and time variables.
>
> I'm noticing that one individual dataset is generally not huge, but when
> 1600 time steps are present, the size of data is on the increase. The
> reader currently manages extents using vtkExtentTranslator which does a
> very nice job of partitioning the dataset into blocks and the data is
> being split across processors nicely. However the reader will only reads
> the current timeFrame and this seems like a terrible waste of resources,
> when in fact 8 nodes could in fact read 100 time steps each and still
> have room to spare.
>
> Some time ago I put together a convenient filter called
> vtkFileCachePipelineFilter which is a wrapper around a sub pipeline : to
> save myself writing too much in this email, I've put a cuople of slides
> (actually several almost duplicated ones) into here
> ftp://ftp.cscs.ch/out/biddisco/vtk/vtkFileCachePipeline.ppt
>
> The idea is very simple. Every time you run your program (eg demo of
> something nice), you spend ages waiting for the reader/contour/glyph etc
> etc to execute, and then you render it. In my code, I tend to save the
> results to disk, put some #ifdef statements in to allow myself to
> reexecute the pipleline if I want to regenerate the data (eg with new
> parameters). I got fed up with this eventually (maintenance), and so
> created a pipeline wrapper which caches the data (Terminology I use is,
> shallow cache = on disk, deep cache=in memory). when you rerun the
> program, the cache filter intercepts the pipeline update, checks the
> disk and if the cache file is there, loads it in (and optionally keeps
> it in memory). The file name can be set in the cache filter in a
> sprintf(%) type way so that cache with param X is stored in one file, if
> param Y is requested, it gets cached in another file etc etc. I've added
> an option to use time variables as the param, so when running an
> animation, the whole series of time data gets cached (memory permitting)
> and it looks lovely from the user's viewpoint.The first time they run
> it, the data gets loaded, but once the animation goes back to step zero,
> it's all in memory and all is well.
>
> Now in actual fact, the filter is extremely simple and does very little,
> but now that I'm putting together the parallelism on the netCDF reader,
> I see that there's an opportunity here to do something a bit nice and
> combine the time variable caching and the parallel streaming into a
> combined concept.
>
> My initial thought is that the Heirarchical Dataset (the netDCF reader
> is now a subclass of vtkHeirarchicalDatasetAlgorithm) could be populated
> with N datasets (each representing a time step) and the one that is
> passed downstream to the rest of the pipeline is selected from this by
> the time step variable/parameter. I'd want to then use the animation
> within paraview to stop through the data more rapidly than previously.
> (Combining this with the caching ability so that downstream filters
> could do contours/glyphs etc and have those results cached - would
> actually be more useful in many cases thasn the raw data).
>
> I'm told that others are working on similar ideas. If anyone has any
> documents on this subject that I might be able to read, I'm going on
> holiday in a couple of days and would welcome some extra material. In
> particular, is there already a mechanism for splitting data across time
> as well as spatial extents. etc etc.
>
> Many thanks
>
> JB
>
> --
> John Biddiscombe,                            email:biddisco @ cscs.ch
> http://www.cscs.ch/about/BJohn.php
> CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
> Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82
>
>
>
> _______________________________________________
> ParaView mailing list
> ParaView at paraview.org
> http://www.paraview.org/mailman/listinfo/paraview
>