<html>
<br>
We have received several questions about how to use new multiprocessing
features in VTK (i.e. MPI with VTK). Here is a summary of what has
been done so far, and what we plan to do in the future:<br>
<br>
Kitware has been contracted to extend VTK to handle extremely large data
sets on multiprocessing/distributed systems. The parallel-VTK
tri-lab project (LosAlamos, Sandia, Lawence Livermoore) is being funded
by the ASCI/VIEWS project and is being led by Jim Ahrens from LANL.<br>
<br>
<b>Controllers and MPI :<br>
</b>VTK2.4 had new objects to manage VTK in multiple processes.
Unfortunately, these objects were not operational in the official VTK3.1
release. They are now working and available in the nightly
releases.<br>
<br>
vtkMultiProcessController provides an api for communication. It
initially sets up the processes at the start of the program, and manages
communication between these processes. It can sends arrays, and has
methods for sending vtk data objects between processes. It also
implements remote method invocations (RMIs). Currently, there are
two concrete implementations for the controller object. The default
vtkThreadedController just runs multiple threads and takes advantage of
shared memory to implement the communication methods.
vtkMPIController uses MPI for its communications. The MPI
controller allows VTK to be run on distributed clusters.<br>
<br>
In order to use MPI you have to do several things. The first thing
you need to have are the MPI libraries an header files. Argonne
Nat. Lab. has a free version of MPI at
<a href="http://www-unix.mcs.anl.gov/mpi/mpich/" eudora="autourl">http://www-unix.mcs.anl.gov/mpi/mpich/</a>
. You need to edit you user.make to point to the MPI libraries and include directories. Here is an example from my user.make:<br>
<br>
MPI_INCLUDE=-I/home/lawcc/mpich/include<br>
MPI_LIBS=/home/lawcc/mpich/build/LINUX/ch_p4/lib/libmpich.a /home/lawcc/mpich/build/LINUX/ch_p4/lib/libpmpich.a <br>
<br>
The last step is to use the option "--with-mpi" when you configure VTK. This will make the vtkMPIController the default implementation of the vtkMultiProcessController object. If you want to switch back to the threaded controller, you can set an environment variable (setenv <br>
VTK_CONTROLLER Threaded).<br>
<br>
.../vtk/contrib/examplesCxx/TestController.cxx has a simple example of how controllers are used.<br>
<br>
<b>Ports<br>
</b>vtkInputPort and vtkOutputPort are classes that were initially developed and contributed by Jim Ahrens. They allow you to connect pipelines in separate processes. They use the default controller for communication, and manage pipeline protocol necessary for updating the pipeline. Examples are:<br>
.../vtk/contrib/examplesCxx/TestImagePort.cxx<br>
.../vtk/contrib/examplesCxx/TestStructuredGridPort.cxx<br>
.../vtk/contrib/examplesCxx/TestStructuredGridPort.cxx<br>
.../vtk/contrib/examplesCxx/TestPolyDataPort.cxx<br>
<br>
<br>
<b>Data Parallelism<br>
</b>To take advantage of parallelism, you can break up processing between multiple processes. The easiest way to do this is to divide up the data and have each process handle a piece of it. The image processing pipeline has always had the ability process structured volumes in pieces. This featured has been used to breakup the datasets for streaming (pieces processed one after another). VTK3.1 extended the graphics (unstructured) pipeline to also process pieces. Unstructured pieces are specified by "Borg" notation: piece n of m. It is the responsibility of the source to determine which cells belong in what piece.<br>
<br>
For an example of how unstructured pieces can be used to implement data parallelism take a look at:<br>
.../vtk/contrib/examplesCxx/ParallelIso.cxx<br>
In this example, the append filter asks for different pieces from its inputs.<br>
<br>
Although the pipeline can handle unstructured pieces, only a hand full of sources have been extended to supply pieces. Most use the default behavior of putting the entire dataset in the first piece, and leaving the other pieces empty. We are currently working on expanding the number of sources that can divide their output intelligently.<br>
<br>
<b>Boundaries<br>
</b>The above approach for defining pieces has a problem when filters use information from neighboring cells (i.e. smoothing). The filters treat piece boundaries as dataset boundaries. This will produce seams between pieces. We are working on ways to solve this problem. Filters that require neighboring information would at least give the user an option to disallow pieces. Another proposed solution is to implement ghost cells. Boundary cells would be duplicated in pieces with a special "ghost" status. The last solution will use custom communication during the execute methods of equivalent filters in different processes.<br>
<br>
<br>
<br>
Charles Law.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</html>