[vtkusers] multicore support

[David E DeMarle]

> One major goal of mine is to allow data to be processed in the background,
> and for it to "appear" on the screen when it is available.
>
> To cut a long story short (that long story is below if you keep reading), I
> want to know if VTK's pipeline can be part-push and part-pull ?

Theoretically one could modify vtkDemandDrivenPipeline (DDP) to
support push. In practice writing a new Executive class would be far
easier because DDP and its decendendents still need to do all the
things there code is fairly complex,


>
> longer story...
>
> I already have this working with a different 3D system... I have a data-push
> system that farms out jobs to a thread pool, and I had thought about
> reimplementing vtkDataDrivenPipeline so that certain jobs could be done in
> the background and when the job is complete, the data becomes available to
> the pipeline.
>
> But, I see a few problems with that...
>
> 1) I get the impression that when an actor or filter demands data, it
> expects to have the data after the call has completed.

This is true.

> This can't be done in my thread-job world... the data is ready when its ready, and its pushed
> through the outputs, not pulled.
>
> 2) I am afraid that I'll lose the ability to use MPI to distribute load via
> the streaming system.  In my system, the job is processed fully before the
> data is passed on (as a read-only reference or as a copy).  Although that
> might be possible if I choose a filter at the end to put in a threaded job,
> and assign all its inputs into MPI executives.  Then it'll execute in a
> thread but stream the data from its (MPI) inputs.   That will benefit from
> task-level parallelism and better memory usage.
> Note that memory usage is not currently a problem, but things change...
>
> 3) This is doing things in threads.  As we have said, its bad.  I could
> potentially farm out the jobs to an MPI processor pool, which seems to be
> what VTK would do, but I am still stuck with a pull-based pipeline which
> freezes the GUI while it waits for processing to complete.
>
>
> What could I do?

I have done this before in ParaView by building a reader which has a
method which can safely determine if the asynchronous process has new
data ready (via asynchronous messages or file status) when it is
ready, the reader calls Modified() on itself. Idle events at the
source end of the pipeline (RenderWindowInteractor for example)
periodically call this method and when necessary update the pipeline
(pull) by calling render. In ParaView it is a bit more complicated
than that because of the dataserver/renderserver/client breakup.

>
> thanks,
> Paul
>
>

-- 
David E DeMarle
Kitware, Inc.
R&D Engineer
28 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-371-3971 x109