[vtkusers] multicore support

[Paul Harris]

2008/11/21 David E DeMarle <dave.demarle at kitware.com>

> On Thu, Nov 20, 2008 at 9:07 PM, Paul Harris <harris.pc at gmail.com> wrote:
> > Hi,
> >
> > I am just starting out with VTK, and I want to know how to properly
> support
> > multicore desktop/laptop PCs...
> >
> > I have done a lot of reading, I've read the VTK OO book, and now I'm
> > checking out ParaView...
> >
> > Am I correct in thinking that MPI is the main (only?) way that people are
> > processing their data in parallel?
> >
> > I have tried without luck to search for info and messages about executing
> on
> > a shared memory system (ie a multicore desktop), or even threaded (seems
> to
> > complicate things a lot).
> > Is the solution to use MPI, but on the one node?
> > Or is there some sort of decent shared-memory support?
> > From what I read on vtkDataObject, it looks like threading is a path to a
> > mine field, as a lot of classes may not be thread-safe, right?
> >
> >
> > Am I also correct to say that ParaView adds a lot of parallel processing
> > support to VTK ?  Support that is not available to VTK ?
> >
> >
> > I have lots more questions, but I'll stop there for now...
> > I eagerly await your responses!
> >
> > thanks for your time,
> > Paul
> >
> >
> > _______________________________________________
> > This is the private VTK discussion list.
> > Please keep messages on-topic. Check the FAQ at:
> > http://www.vtk.org/Wiki/VTK_FAQ
> > Follow this link to subscribe/unsubscribe:
> > http://www.vtk.org/mailman/listinfo/vtkusers
> >
> >
>
> Many of VTK's  imaging filters support thread level parallelism. When
> VTK is compiled with threading support, those filters in the Imaging
> kit will automatically spawn threads to take advantage of the
> available parallelism near to the innermost loop. When processing this
> type of data, threading is fairly easy to do and scales, in terms of
> execution time, very well.
>
> ParaView heavily uses VTK's process level parallelism to make all data
> types (images and unstructured types) scale in terms of memory size.
> That is, given enough aggregate RAM, any size problem can be
> processed. This is available in VTK through the Parallel toolkit. For
> the most part it is only accessible through MPI and parallelizes near
> the outermost loop.
>
> To make a long story short, if you are working with image data, VTK
> has built in multicore support. If you are working with unstructured
> data, your best bet is to use mpi on your multicore machine. If you
> want to use VTK with threading otherwise, it can be done but as you
> say you are heading into a minefield.
>


Thanks for the reply David :)

One major goal of mine is to allow data to be processed in the background,
and for it to "appear" on the screen when it is available.

To cut a long story short (that long story is below if you keep reading), I
want to know if VTK's pipeline can be part-push and part-pull ?


longer story...

I already have this working with a different 3D system... I have a data-push
system that farms out jobs to a thread pool, and I had thought about
reimplementing vtkDataDrivenPipeline so that certain jobs could be done in
the background and when the job is complete, the data becomes available to
the pipeline.

But, I see a few problems with that...

1) I get the impression that when an actor or filter demands data, it
expects to have the data after the call has completed.  This can't be done
in my thread-job world... the data is ready when its ready, and its pushed
through the outputs, not pulled.

2) I am afraid that I'll lose the ability to use MPI to distribute load via
the streaming system.  In my system, the job is processed fully before the
data is passed on (as a read-only reference or as a copy).  Although that
might be possible if I choose a filter at the end to put in a threaded job,
and assign all its inputs into MPI executives.  Then it'll execute in a
thread but stream the data from its (MPI) inputs.   That will benefit from
task-level parallelism and better memory usage.
Note that memory usage is not currently a problem, but things change...

3) This is doing things in threads.  As we have said, its bad.  I could
potentially farm out the jobs to an MPI processor pool, which seems to be
what VTK would do, but I am still stuck with a pull-based pipeline which
freezes the GUI while it waits for processing to complete.


What could I do?

thanks,
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20081121/0ad0009a/attachment.htm>