[Paraview] How to create a parallele reader for paraview ?

[Moreland, Kenneth]

> Context:
> - We suppose that Paraview is deployed over a cluster of "n" nodes
> 
> 1) The user want to read a file. (click and select files in the menu)
> 2) Paraview create "n" instance of the reader. One instance on each
> cluster node.
> 3) Paraview will apply the same data processing behind each reader by
> setting in vtkInformation of each specific request
>     - vtkStreamingDemandDrivenPipeline::UPDATE_PIECE_NUMBER()  equal
to
> the MPI_ProcessId
>     - and vtkStreamingDemandDrivenPipeline::UPDATE_NUMBER_OF_PIECES()
> equal to the total number of nodes. ("n" for us)
>     - and something about GHOST_LEVEL depending on the need of the
> filter involved in the pipeline.

That's pretty much it.  The only minor thing missing is that the reader
should set vtkStreamingDemandDrivenPipeline::MAXIMUM_NUMBER_OF_PIECES()
in the output information during the RequestInformation pipeline phase.
You usually set this to -1 to signal that you support any number of
pieces.

> 
> If so, this mean that each reader is reading indepently the same file
> and each one is reading and extracting its requesting part ?
> The reader don't need to know anything about MPI. Paraview manage
> everything.

That's the original idea.  In theory, it should not matter whether you
are part of an MPI job or some serial process is streaming data.  In
practice, however, it can make a big difference.  Thus, it is becoming
quite common for reader to be "MPI-aware" so that they may coordinate
the reads.

> This also mean that the file must be available on the same path on
each
> node of the cluster ?

Most people design their clusters to have a file system that mounted on
all the nodes and thus the files will be accessible everywhere.  It
sounds like you have a file accessible to node 0 only and want to stream
data from there to the other nodes.  That should work OK if that's how
you design your reader.  Just be aware that the built in readers will
not work that way.

> I suppose that the reader will have to provide the requested level of
> ghost cell ?

Ideally, the reader will produce ghost cells, and the readers of
structured data typically do because they are easy to find.  However, I
think all the unstructured data readers (those that ready vtkPolyData or
vtkUnstructuredGrid) do not produce ghost cells unless they are
conveniently accessible.  It is usually difficult to code and can take
longer than a user is willing to wait.  In that case, ignore the ghost
cell requests.  Use the D3 filter if you need to generate them.

> 
> Which is unfortunately not the behaviour that I'm looking for... ;-)
> I will look at the vtkSpyPlotReader and will come back to you if I
have
> more question.

Although it does not fit exactly within what was originally intended
with the parallel VTK parallel readers, I think you should just try to
do what you originally wanted.  Nothing you have said so far leads me to
believe that it will not work.

-Ken