[vtkusers] multicore support

[Paul Harris]

2008/12/2 David E DeMarle <dave.demarle at kitware.com>

> > One major goal of mine is to allow data to be processed in the
> background,
> > and for it to "appear" on the screen when it is available.
> >
> > To cut a long story short (that long story is below if you keep reading),
> I
> > want to know if VTK's pipeline can be part-push and part-pull ?
>
> Theoretically one could modify vtkDemandDrivenPipeline (DDP) to
> support push. In practice writing a new Executive class would be far
> easier because DDP and its decendendents still need to do all the
> things there code is fairly complex,
>

I've noticed a lot of the filters seem to be built to directly support DDP,
especially to support streamed or partial processing... so the work may be a
lot broader than just a new Executive class.


>
>
> >
> > longer story...
> >
> > I already have this working with a different 3D system... I have a
> data-push
> > system that farms out jobs to a thread pool, and I had thought about
> > reimplementing vtkDataDrivenPipeline so that certain jobs could be done
> in
> > the background and when the job is complete, the data becomes available
> to
> > the pipeline.
> >
> > But, I see a few problems with that...
> >
> > 1) I get the impression that when an actor or filter demands data, it
> > expects to have the data after the call has completed.
>
> This is true.
>

another reason why Push would take a lot more work (i think) because the
callers who are demanding data would have to be changed to either

a) filters expect async data in the middle of operations ... means they
would have to be able to postpone their operations until later -
difficult/impossible/slow if its in the middle of a lot of GL calls

b) know what it needs first, send the requests upstream, and only when the
data is available, then process.  forecasting requirements may be impossible
in some filters, so it may just degenerate into a non-streaming/partial
solution.

If you remove the need to support streaming, then all the filters could just
execute and calculate their whole extents and then push that data up the
chain...  but you would need to ensure that all a filter's sources have
finished updating before executing otherwise it would be either wasted
computation or worse, executing on data sources that do not match up with
each other.



> > This can't be done in my thread-job world... the data is ready when its
> ready, and its pushed
> > through the outputs, not pulled.
> >
> > 2) I am afraid that I'll lose the ability to use MPI to distribute load
> via
> > the streaming system.  In my system, the job is processed fully before
> the
> > data is passed on (as a read-only reference or as a copy).  Although that
> > might be possible if I choose a filter at the end to put in a threaded
> job,
> > and assign all its inputs into MPI executives.  Then it'll execute in a
> > thread but stream the data from its (MPI) inputs.   That will benefit
> from
> > task-level parallelism and better memory usage.
> > Note that memory usage is not currently a problem, but things change...
> >
> > 3) This is doing things in threads.  As we have said, its bad.  I could
> > potentially farm out the jobs to an MPI processor pool, which seems to be
> > what VTK would do, but I am still stuck with a pull-based pipeline which
> > freezes the GUI while it waits for processing to complete.
> >
> >
> > What could I do?
>
> I have done this before in ParaView by building a reader which has a
> method which can safely determine if the asynchronous process has new
> data ready (via asynchronous messages or file status) when it is
> ready, the reader calls Modified() on itself. Idle events at the
> source end of the pipeline (RenderWindowInteractor for example)
> periodically call this method and when necessary update the pipeline
> (pull) by calling render. In ParaView it is a bit more complicated
> than that because of the dataserver/renderserver/client breakup.
>

when the reader calls Modified(), at which point is the data loaded into the
pipeline?  at the moment i must construct and copy the data into the
pipeline while in the main thread, as I cannot build vtkDataObjects in a
separate thread.

thanks,
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20081204/237685ee/attachment.htm>