[vtkusers] merging multiple vtkStructuredGrid's?

[Peter Schmitt]

I am still stuck with merging two structured grids into one.  I need
something that is almost the opposite of vtkExtractGrid
[http://www.vtk.org/doc/release/4.2/html/classvtkExtractGrid.html].  I
want to set two pieces of grid data into one grid.

vtkMergeFilter doesn't seem like it will work for me as it appears to
merge scalars or vectors with the points of a dataset.

vtkStructuredGridToStructuredGrid is used for vtkExtractGrid, so that's out.

I looked at the source code for vtkXMLPStructuredGridReader,
vtkXMLPStructuredGridWriter, vtkXMLStructuredGridReader,
vtkXMLStructuredGridWriter hoping to see how to merge multiple grids
into one large grid (the xml classes do exactly this as the XML format
allows an individual to set pieces of data to be read in parallel).  I
found this code confusing as far as a starting point to implement my
own class/function to merge multiple grids into one.

I already have a small dumb program that converts from my fileformat
to the vtkXMLPStructuredGrid format, but this is unacceptable for my
final product as I will be generating very large quantities of data in
parallel and visualizing it in near real-time as data is sent to
multiple visualization nodes.

Any suggestions are appreciated!
Pete

PS:  Tom: Thanks for your help.  Since this functionality seems to be
already available once data is stored in the vtkXMLP* format, one
would think that it would be easy to implement directly into VTK...

Making one very large structuredgrid and setting sub-sections of the
data might be my only route as you had suggested.  I could implement
something like:
1. Generate data on compute-nodes
2. Send data to one or more data-processing node that stores data in
vtkXMLP* format
3.  Send vtkXMLP* data to visualization nodes

I am currently working with smaller datasets as a proof of concept. 
However, a colleague of mine was working with a 2gig dataset in
paraview on 4 nodes.  Loading data was slow, but once it was in memory
of all 4 nodes, the window was refreshing at interactive rates (I
would guess around 15 fps).   I will try to write back to the list
with my own results when I get there.

Allright.. now I'm rambling, happy vizing


On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
>  <a530bfb5050616155867164a47 at mail.gmail.com>Peter Schmitt writes:
> <snip>
> >As far as performance is concerned, I will eventually be running all
> >of this on a large Linux cluster using parallel vtk + MPI  Once I get
> >rolling with this project, I'll try & post my results to vtkusers
> >sometime in late July/early August.
> 
> sounds like fun =). i'll look forward to hearing about it.
> 
> >I have already written a small program that converts my data into the
> >*.pvts (vtkXMLpStructuredGrid) format using a series of printf()
> >statements.  I have verified that my data is okay by loading it into
> >Paraview.   Translating from the *.pvts printf code to code that
> >imports directly into vtk objects is a little confusing since I do not
> >understand how to implement "pieces" of structured grids directly in
> >vtk (ala this thread).
> 
> neither do I. This is something I've been wrestling with recently too.
> 
> Recently I wrote a data convertor that takes an old data format we have
> and creates an XMLRectilinearGrid file. Getting all the XML correct and
> such looked difficult to me, so I took the easy route -- I create 1
> ridiculous sized RectilinearGrid, pass it to an
> XMLRectilinearGridWriter, and then tell the Writer that I want it in
> <X> pieces. The writer handles the segmenting of the data transparently
> (in addition to making sure the XML is correct such that VTK can read
> it).
> Maybe a similar approach would work for you.
> 
> My current problem is that the streaming ("pieces") doesn't seem to
> solve the large data problems. With 250 pieces in a 1.6 gig RGrid,
> performance is abysmal on a workstation with 2 gigs of memory. I would
> very much like to hear if you're getting similar peformance issues.
> 
> Currently I'm working on an HDF5 reader that might be 'smarter'; the
> XML reader almost seems like its reading in the whole dataset right
> from the bat, and then giving back the corresponding piece of that
> in-memory dataset when requested.
> 
> Anyway I'm starting to ramble, but the point is I'd very much like to
> hear how you (or anyone!) solve(s) the large datafile issue.
> 
> -tom
> 
> <snip>
> >On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
> >>  <a530bfb505061614571fbe2fa6 at mail.gmail.com>Peter Schmitt writes:
> >> <snip>
> >> >Does anyone have suggstions on how to merge the two StructuredGrids
> >> >into one larger (40x20x20) structured grid?
> >>
> >> vtkMergeFilter sounds like it will do what you want. If you want more
> >> control, it looks like vtkMergeDataObjectFilter might give you that.
> >>
> >> In the future, you could try a quick grep through:
> >>
> >> http://www.vtk.org/doc/nightly/html/classes.html
> >>
> >> sometimes just the name of a class is enough to clue you on what you'll
> >> need to do.
> >>
> >> >(eventually I will need to merge 100's or 1000's of structuredgrids together).
> >>
> >> I don't know how well vtkMergeFilter scales, sorry...
> >> Intuitively though, merging a thousand 40x20x20 grids sounds like it
> >> will take a \emph{lot} of resources.
> >>
> >> >In the end, I hope to use vtkXMLPStructuredGridWriter to store one
> >> >pvts file with one structuredgrid consiting of multiple pieces.
> >>
> >> I would be \emph{very} interested to hear how well that scales for you.
> >>
> >> -tom
> >>
> >
>