[vtkusers] merging multiple vtkStructuredGrid's?

[tom fogal]

 <a530bfb5050620134858cbce85 at mail.gmail.com>Peter Schmitt writes:
>I am still stuck with merging two structured grids into one.  I need
<snip>

>vtkStructuredGridToStructuredGrid is used for vtkExtractGrid, so that's out.

vtkSGToSG is an abstract class IIRC. It seems to me like it might be a
good choice as a base class if you were to write your own filter for
this application.

>I looked at the source code for vtkXMLPStructuredGridReader,
>vtkXMLPStructuredGridWriter, vtkXMLStructuredGridReader,
>vtkXMLStructuredGridWriter hoping to see how to merge multiple grids
>into one large grid (the xml classes do exactly this as the XML format
>allows an individual to set pieces of data to be read in parallel).  I
>found this code confusing as far as a starting point to implement my
>own class/function to merge multiple grids into one.

<snip>

>PS:  Tom: Thanks for your help.  Since this functionality seems to be
>already available once data is stored in the vtkXMLP* format, one
>would think that it would be easy to implement directly into VTK...
>
>Making one very large structuredgrid and setting sub-sections of the
>data might be my only route as you had suggested.  I could implement
>something like:
>1. Generate data on compute-nodes
>2. Send data to one or more data-processing node that stores data in
>vtkXMLP* format
>3.  Send vtkXMLP* data to visualization nodes

Hrm, this got me thinking. What if you could make the data
\emph{appear} to be XML data, but in reality come from a socket
recv()ing from the computer nodes?

I was thinking this might be possible if you could take a
vtkXMLStructuredGridReader, and instead of invoking read/update, there
was some type of 'SetPiece' method that took a vtkStructuredGrid and
some sort of piece index... probably need some information about that
StructuredGrid's extent within the global extent, etc. too.

It doesn't look like that type of model is supported, but looking at:

http://www.vtk.org/doc/nightly/html/classvtkXMLStructuredDataReader.html

You \emph{might} be able to get away with deriving from
vtkXMLStructuredDataReader and overloading the ReadPiece() function
and its friends (perhaps SetupPieces, DestroyPieces, ReadSubExtent?).
After you've done that, it will \emph{hopefully} be as easy as
copy-and-paste to make a new class that has
vtkXMLStructuredGridReader's code, but derives from the class that you
have just created. Maybe a diagram is good here:

vtkXMLStructuredDataReader
            |
            v
vtkPeterSchmittsNetworkStructuredDataReader
            |
            v
vtkPeterSchmittsNetworkStructuredGridReader

If your data are already naturally setup in pieces like that, it would
be a shame to destroy the pieces in some type of merging algorithm.

In any case it sounds like a lot of work, particularly if you've never
written a VTK filter before =(. Sorry.

-tom

<snip>

>On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
>>  <a530bfb5050616155867164a47 at mail.gmail.com>Peter Schmitt writes:
>> <snip>
>> >As far as performance is concerned, I will eventually be running all
>> >of this on a large Linux cluster using parallel vtk + MPI  Once I get
>> >rolling with this project, I'll try & post my results to vtkusers
>> >sometime in late July/early August.
>> 
>> sounds like fun =). i'll look forward to hearing about it.
>> 
>> >I have already written a small program that converts my data into the
>> >*.pvts (vtkXMLpStructuredGrid) format using a series of printf()
>> >statements.  I have verified that my data is okay by loading it into
>> >Paraview.   Translating from the *.pvts printf code to code that
>> >imports directly into vtk objects is a little confusing since I do not
>> >understand how to implement "pieces" of structured grids directly in
>> >vtk (ala this thread).
>> 
>> neither do I. This is something I've been wrestling with recently too.
>> 
>> Recently I wrote a data convertor that takes an old data format we have
>> and creates an XMLRectilinearGrid file. Getting all the XML correct and
>> such looked difficult to me, so I took the easy route -- I create 1
>> ridiculous sized RectilinearGrid, pass it to an
>> XMLRectilinearGridWriter, and then tell the Writer that I want it in
>> <X> pieces. The writer handles the segmenting of the data transparently
>> (in addition to making sure the XML is correct such that VTK can read
>> it).
>> Maybe a similar approach would work for you.
>> 
>> My current problem is that the streaming ("pieces") doesn't seem to
>> solve the large data problems. With 250 pieces in a 1.6 gig RGrid,
>> performance is abysmal on a workstation with 2 gigs of memory. I would
>> very much like to hear if you're getting similar peformance issues.
>> 
>> Currently I'm working on an HDF5 reader that might be 'smarter'; the
>> XML reader almost seems like its reading in the whole dataset right
>> from the bat, and then giving back the corresponding piece of that
>> in-memory dataset when requested.
>> 
>> Anyway I'm starting to ramble, but the point is I'd very much like to
>> hear how you (or anyone!) solve(s) the large datafile issue.
>> 
>> -tom
>> 
>> <snip>
>> >On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
>> >>  <a530bfb505061614571fbe2fa6 at mail.gmail.com>Peter Schmitt writes:
>> >> <snip>
>> >> >Does anyone have suggstions on how to merge the two StructuredGrids
>> >> >into one larger (40x20x20) structured grid?
>> >>
>> >> vtkMergeFilter sounds like it will do what you want. If you want more
>> >> control, it looks like vtkMergeDataObjectFilter might give you that.
>> >>
>> >> In the future, you could try a quick grep through:
>> >>
>> >> http://www.vtk.org/doc/nightly/html/classes.html
>> >>
>> >> sometimes just the name of a class is enough to clue you on what you'll
>> >> need to do.
>> >>
>> >> >(eventually I will need to merge 100's or 1000's of structuredgrids together).
>> >>
>> >> I don't know how well vtkMergeFilter scales, sorry...
>> >> Intuitively though, merging a thousand 40x20x20 grids sounds like it
>> >> will take a \emph{lot} of resources.
>> >>
>> >> >In the end, I hope to use vtkXMLPStructuredGridWriter to store one
>> >> >pvts file with one structuredgrid consiting of multiple pieces.
>> >>
>> >> I would be \emph{very} interested to hear how well that scales for you.
>> >>
>> >> -tom
>> >>
>> >
>>
>