[vtkusers] merging multiple vtkStructuredGrid's?

[Peter Schmitt]

Wow, that's a really good idea!  Unforunately, I have never written a
VTK filter, but your logic seems to make sense.

In the short term, I am going to make a large vtkStructuredGrid and
manually merge the data myself.  Should I get some more free time, I
will definitely look into making my own network reader.

Thanks again for the advice!

Pete

On 6/21/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
>  <a530bfb5050620134858cbce85 at mail.gmail.com>Peter Schmitt writes:
> >I am still stuck with merging two structured grids into one.  I need
> <snip>
> 
> >vtkStructuredGridToStructuredGrid is used for vtkExtractGrid, so that's out.
> 
> vtkSGToSG is an abstract class IIRC. It seems to me like it might be a
> good choice as a base class if you were to write your own filter for
> this application.
> 
> >I looked at the source code for vtkXMLPStructuredGridReader,
> >vtkXMLPStructuredGridWriter, vtkXMLStructuredGridReader,
> >vtkXMLStructuredGridWriter hoping to see how to merge multiple grids
> >into one large grid (the xml classes do exactly this as the XML format
> >allows an individual to set pieces of data to be read in parallel).  I
> >found this code confusing as far as a starting point to implement my
> >own class/function to merge multiple grids into one.
> 
> <snip>
> 
> >PS:  Tom: Thanks for your help.  Since this functionality seems to be
> >already available once data is stored in the vtkXMLP* format, one
> >would think that it would be easy to implement directly into VTK...
> >
> >Making one very large structuredgrid and setting sub-sections of the
> >data might be my only route as you had suggested.  I could implement
> >something like:
> >1. Generate data on compute-nodes
> >2. Send data to one or more data-processing node that stores data in
> >vtkXMLP* format
> >3.  Send vtkXMLP* data to visualization nodes
> 
> Hrm, this got me thinking. What if you could make the data
> \emph{appear} to be XML data, but in reality come from a socket
> recv()ing from the computer nodes?
> 
> I was thinking this might be possible if you could take a
> vtkXMLStructuredGridReader, and instead of invoking read/update, there
> was some type of 'SetPiece' method that took a vtkStructuredGrid and
> some sort of piece index... probably need some information about that
> StructuredGrid's extent within the global extent, etc. too.
> 
> It doesn't look like that type of model is supported, but looking at:
> 
> http://www.vtk.org/doc/nightly/html/classvtkXMLStructuredDataReader.html
> 
> You \emph{might} be able to get away with deriving from
> vtkXMLStructuredDataReader and overloading the ReadPiece() function
> and its friends (perhaps SetupPieces, DestroyPieces, ReadSubExtent?).
> After you've done that, it will \emph{hopefully} be as easy as
> copy-and-paste to make a new class that has
> vtkXMLStructuredGridReader's code, but derives from the class that you
> have just created. Maybe a diagram is good here:
> 
> vtkXMLStructuredDataReader
>             |
>             v
> vtkPeterSchmittsNetworkStructuredDataReader
>             |
>             v
> vtkPeterSchmittsNetworkStructuredGridReader
> 
> If your data are already naturally setup in pieces like that, it would
> be a shame to destroy the pieces in some type of merging algorithm.
> 
> In any case it sounds like a lot of work, particularly if you've never
> written a VTK filter before =(. Sorry.
> 
> -tom
> 
> <snip>
> 
> >On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
> >>  <a530bfb5050616155867164a47 at mail.gmail.com>Peter Schmitt writes:
> >> <snip>
> >> >As far as performance is concerned, I will eventually be running all
> >> >of this on a large Linux cluster using parallel vtk + MPI  Once I get
> >> >rolling with this project, I'll try & post my results to vtkusers
> >> >sometime in late July/early August.
> >>
> >> sounds like fun =). i'll look forward to hearing about it.
> >>
> >> >I have already written a small program that converts my data into the
> >> >*.pvts (vtkXMLpStructuredGrid) format using a series of printf()
> >> >statements.  I have verified that my data is okay by loading it into
> >> >Paraview.   Translating from the *.pvts printf code to code that
> >> >imports directly into vtk objects is a little confusing since I do not
> >> >understand how to implement "pieces" of structured grids directly in
> >> >vtk (ala this thread).
> >>
> >> neither do I. This is something I've been wrestling with recently too.
> >>
> >> Recently I wrote a data convertor that takes an old data format we have
> >> and creates an XMLRectilinearGrid file. Getting all the XML correct and
> >> such looked difficult to me, so I took the easy route -- I create 1
> >> ridiculous sized RectilinearGrid, pass it to an
> >> XMLRectilinearGridWriter, and then tell the Writer that I want it in
> >> <X> pieces. The writer handles the segmenting of the data transparently
> >> (in addition to making sure the XML is correct such that VTK can read
> >> it).
> >> Maybe a similar approach would work for you.
> >>
> >> My current problem is that the streaming ("pieces") doesn't seem to
> >> solve the large data problems. With 250 pieces in a 1.6 gig RGrid,
> >> performance is abysmal on a workstation with 2 gigs of memory. I would
> >> very much like to hear if you're getting similar peformance issues.
> >>
> >> Currently I'm working on an HDF5 reader that might be 'smarter'; the
> >> XML reader almost seems like its reading in the whole dataset right
> >> from the bat, and then giving back the corresponding piece of that
> >> in-memory dataset when requested.
> >>
> >> Anyway I'm starting to ramble, but the point is I'd very much like to
> >> hear how you (or anyone!) solve(s) the large datafile issue.
> >>
> >> -tom
> >>
> >> <snip>
> >> >On 6/16/05, tom fogal <tfogal at apollo.sr.unh.edu> wrote:
> >> >>  <a530bfb505061614571fbe2fa6 at mail.gmail.com>Peter Schmitt writes:
> >> >> <snip>
> >> >> >Does anyone have suggstions on how to merge the two StructuredGrids
> >> >> >into one larger (40x20x20) structured grid?
> >> >>
> >> >> vtkMergeFilter sounds like it will do what you want. If you want more
> >> >> control, it looks like vtkMergeDataObjectFilter might give you that.
> >> >>
> >> >> In the future, you could try a quick grep through:
> >> >>
> >> >> http://www.vtk.org/doc/nightly/html/classes.html
> >> >>
> >> >> sometimes just the name of a class is enough to clue you on what you'll
> >> >> need to do.
> >> >>
> >> >> >(eventually I will need to merge 100's or 1000's of structuredgrids together).
> >> >>
> >> >> I don't know how well vtkMergeFilter scales, sorry...
> >> >> Intuitively though, merging a thousand 40x20x20 grids sounds like it
> >> >> will take a \emph{lot} of resources.
> >> >>
> >> >> >In the end, I hope to use vtkXMLPStructuredGridWriter to store one
> >> >> >pvts file with one structuredgrid consiting of multiple pieces.
> >> >>
> >> >> I would be \emph{very} interested to hear how well that scales for you.
> >> >>
> >> >> -tom
> >> >>
> >> >
> >>
> >
>