[Paraview] How to create a parallele reader for paraview ?

[Sebastien Jourdain]

Thank's a lot Ken,

you pretty well answear at all my questions...
And thanks for the MAXIMUM_NUMBER_OF_PIECES() that I forgot... ;-)

Regards,

Seb

Moreland, Kenneth a écrit :
>> Context:
>> - We suppose that Paraview is deployed over a cluster of "n" nodes
>>
>> 1) The user want to read a file. (click and select files in the menu)
>> 2) Paraview create "n" instance of the reader. One instance on each
>> cluster node.
>> 3) Paraview will apply the same data processing behind each reader by
>> setting in vtkInformation of each specific request
>>     - vtkStreamingDemandDrivenPipeline::UPDATE_PIECE_NUMBER()  equal
>>     
> to
>   
>> the MPI_ProcessId
>>     - and vtkStreamingDemandDrivenPipeline::UPDATE_NUMBER_OF_PIECES()
>> equal to the total number of nodes. ("n" for us)
>>     - and something about GHOST_LEVEL depending on the need of the
>> filter involved in the pipeline.
>>     
>
> That's pretty much it.  The only minor thing missing is that the reader
> should set vtkStreamingDemandDrivenPipeline::MAXIMUM_NUMBER_OF_PIECES()
> in the output information during the RequestInformation pipeline phase.
> You usually set this to -1 to signal that you support any number of
> pieces.
>
>   
>> If so, this mean that each reader is reading indepently the same file
>> and each one is reading and extracting its requesting part ?
>> The reader don't need to know anything about MPI. Paraview manage
>> everything.
>>     
>
> That's the original idea.  In theory, it should not matter whether you
> are part of an MPI job or some serial process is streaming data.  In
> practice, however, it can make a big difference.  Thus, it is becoming
> quite common for reader to be "MPI-aware" so that they may coordinate
> the reads.
>
>   
>> This also mean that the file must be available on the same path on
>>     
> each
>   
>> node of the cluster ?
>>     
>
> Most people design their clusters to have a file system that mounted on
> all the nodes and thus the files will be accessible everywhere.  It
> sounds like you have a file accessible to node 0 only and want to stream
> data from there to the other nodes.  That should work OK if that's how
> you design your reader.  Just be aware that the built in readers will
> not work that way.
>
>   
>> I suppose that the reader will have to provide the requested level of
>> ghost cell ?
>>     
>
> Ideally, the reader will produce ghost cells, and the readers of
> structured data typically do because they are easy to find.  However, I
> think all the unstructured data readers (those that ready vtkPolyData or
> vtkUnstructuredGrid) do not produce ghost cells unless they are
> conveniently accessible.  It is usually difficult to code and can take
> longer than a user is willing to wait.  In that case, ignore the ghost
> cell requests.  Use the D3 filter if you need to generate them.
>
>   
>> Which is unfortunately not the behaviour that I'm looking for... ;-)
>> I will look at the vtkSpyPlotReader and will come back to you if I
>>     
> have
>   
>> more question.
>>     
>
> Although it does not fit exactly within what was originally intended
> with the parallel VTK parallel readers, I think you should just try to
> do what you originally wanted.  Nothing you have said so far leads me to
> believe that it will not work.
>
> -Ken
>
>
> .
>
>