[Paraview-developers] [EXTERNAL] Re: Reading multiple files in parallel

Tue Aug 16 17:28:44 EDT 2016

ParaView will set the UPDATE_NUMBER_OF_PIECES to the number of processes and UPDATE_PIECE_NUMBER to the rank.

-Ken

From: sumeet kumar [mailto:sumeet.kumar507 at gmail.com]
Sent: Tuesday, August 16, 2016 3:08 PM
To: Moreland, Kenneth <kmorel at sandia.gov>; Sumeet Kumar Sinha <skssinha at ucdavis.edu>
Cc: ParaView Developers <paraview-developers at paraview.org>
Subject: [EXTERNAL] Re: [Paraview-developers] Reading multiple files in parallel

I compiled ParaView with MPI enabled and as you said and now I can get UPDATE_NUMBER_OF_PIECES() and UPDATE_NUMBER_OF_PIECES() keys.
But, as you said in your previous mail that the number of piece requests will not match with the number of files. So, how can I now what is the range of piece requests that will be generated.
For example, if I have 8 files and the user runs para-view on 4 processes.  The optimal way is to read two files per processes.
But my question will how can i know that the user is running on 4 nodes only. Is there any key ??/ for that.

Thanks
Sumeet

On Tue, Aug 16, 2016 at 8:03 AM, Moreland, Kenneth <kmorel at sandia.gov<mailto:kmorel at sandia.gov>> wrote:
Are you running ParaView in MPI parallel mode? If not, it might not set the number of pieces.

Also, I don't think you need to set WHOLE_EXTENT. That is for structured type of data.

-Ken

Sent from my iPad so blame autocorrect.

On Aug 15, 2016, at 5:38 PM, sumeet kumar <sumeet.kumar507 at gmail.com<mailto:sumeet.kumar507 at gmail.com>> wrote:
Hey Ken,
I am sorry that I am asking you a lot of stupid questions. But I am stuck here. I followed what you said, But I don't see the piece number getting updated. And also when would that stop being updated...etc. Here is my code

int pvESSI::RequestInformation( vtkInformation *request, vtkInformationVector **vtkNotUsed(inVec), vtkInformationVector* outVec){

    this->Initialize();

    vtkInformation* Node_Mesh = outVec->GetInformationObject(0);

    double Time_range[2]={Time[0],Time[Number_Of_Time_Steps-1]};

    Node_Mesh->Set(vtkStreamingDemandDrivenPipeline::TIME_STEPS(),Time, this->Number_Of_Time_Steps);
    Node_Mesh->Set(vtkStreamingDemandDrivenPipeline::TIME_RANGE(),Time_range,2);

    Node_Mesh->Set(vtkStreamingDemandDrivenPipeline::WHOLE_EXTENT(),EXTENT,6);

    int num_of_piec = 10;
    Node_Mesh->Set(CAN_HANDLE_PIECE_REQUEST(), 1);

    return 1;
}

int pvESSI::RequestData(vtkInformation *vtkNotUsed(request),vtkInformationVector **vtkNotUsed(inputVector),    vtkInformationVector *outputVector){

     vtkInformation *Node_Mesh = outputVector->GetInformationObject(0);
    // outInfo->Print(std::cout);

    piece_no = Node_Mesh->Get(vtkStreamingDemandDrivenPipeline::UPDATE_PIECE_NUMBER());
    num_of_pieces = Node_Mesh->Get(vtkStreamingDemandDrivenPipeline::UPDATE_NUMBER_OF_PIECES());
    cout << "Piece_No " << piece_no << endl;
    cout << "Number_of_Pieces " << piece_no << endl;

     Step_Initializer(Piece_No);     // makes the file and variables ready for that piece

      this->Node_Mesh_Current_Time = Time_Map.find( Node_Mesh->Get(vtkStreamingDemandDrivenPipeline::UPDATE_TIME_STEP()))->second;

    if (!Whether_Node_Mesh_Build){
        this->Get_Node_Mesh(UGrid_Node_Mesh);
        UGrid_Current_Node_Mesh->ShallowCopy(UGrid_Node_Mesh);
    }

    Build_Node_Attributes(UGrid_Current_Node_Mesh, this->Node_Mesh_Current_Time );
    Build_Stress_Field_At_Nodes(UGrid_Current_Node_Mesh, this->Node_Mesh_Current_Time);

    // get the ouptut pointer to paraview
    vtkUnstructuredGrid *Output_Node_Mesh = vtkUnstructuredGrid::SafeDownCast(Node_Mesh->Get(vtkDataObject::DATA_OBJECT()));

    Output_Node_Mesh->ShallowCopy(UGrid_Current_Node_Mesh);
    return 1;
}
When I run this code, The Piece_no and Number_of_Pieces both remain 0.

Sumeet

On Mon, Aug 15, 2016 at 3:58 PM, Moreland, Kenneth <kmorel at sandia.gov<mailto:kmorel at sandia.gov>> wrote:
During the RequestData pass of the pipeline, you should be able to get the keys vtkStreamingDemandDrivenPipeline::UPDATE_PIECE_NUMBER() and vtkStreamingDemandDrivenPipeline::UPDATE_NUMBER_OF_PIECES() from the output information object that comes from downstream. This will tell you how many pieces the pipeline expects and which piece is expected to be read.

Generally the number of pieces requested will not match exactly the number of files that you have. This means you will have to adjust by either reading more than 1 file at a time or sometimes reading nothing and returning an empty object.

-Ken

From: sumeet kumar [mailto:sumeet.kumar507 at gmail.com<mailto:sumeet.kumar507 at gmail.com>]
Sent: Monday, August 15, 2016 3:52 PM
To: Moreland, Kenneth <kmorel at sandia.gov<mailto:kmorel at sandia.gov>>
Cc: ParaView Developers <paraview-developers at paraview.org<mailto:paraview-developers at paraview.org>>
Subject: [EXTERNAL] Re: [Paraview-developers] Reading multiple files in parallel

Thanks Kenneth, for you help.
I get the vtkAlgorithm::CAN_HANDLE_PIECE_REQUEST(), set 1 part.
But, what to do next.

How can i get the piece no, so that i can update the data for it.

vtkmXMLPUnstructuredGridDataReader is not doing me a great help.
Sumeet

On Mon, Aug 15, 2016 at 12:57 PM, Moreland, Kenneth <kmorel at sandia.gov<mailto:kmorel at sandia.gov>> wrote:
I think the key you are looking for is vtkAlgorithm::CAN_HANDLE_PIECE_REQUEST(), which you want to set to 1 to signify that you can provide multiple pieces. You might want to look at vtkmXMLPUnstructuredGridDataReader for an example.

If I remember correctly, this has recently been changed in VTK. It used to be that you would give the number of pieces you supported, but that functionality was not useful in practice and consequently readers reported that they could produce any number of pieces in order to be useful. Thus, the behavior was change to allow readers simply to report whether or not they support multiple pieces. Berk can probably provide more information.

-Ken

From: Paraview-developers [mailto:paraview-developers-bounces at paraview.org<mailto:paraview-developers-bounces at paraview.org>] On Behalf Of sumeet kumar
Sent: Monday, August 15, 2016 12:58 PM
To: ParaView Developers <paraview-developers at paraview.org<mailto:paraview-developers at paraview.org>>
Subject: [EXTERNAL] [Paraview-developers] Reading multiple files in parallel

Hello all,
I have a number of files which contains information about pieces of unstructured mesh. For example :-

t_2.h5.01.feioutput
t_2.h5.03.feioutput
t_2.h5.04.feioutput
t_2.h5.05.feioutput
t_2.h5.06.feioutput
t_2.h5.07.feioutput
t_2.h5.08.feioutput
t_2.h5.09.feioutput
t_2.h5.feioutput
My reader can read each file individually but I want to read them all in parallel so that I can visualize the whole model. I looked at the http://www.paraview.org/Wiki/Writing_ParaView_Readers and followed the steps for unstructured mesh.
When I try to set the number of pieces in Request Information function as this

vtkInformation* Node_Mesh = outVec->GetInformationObject(0);
Node_Mesh->Set(vtkStreamingDemandDrivenPipeline::MAXIMUM_NUMBER_OF_PIECES(), 10);

I get the following error

error: ‘MAXIMUM_NUMBER_OF_PIECES’ is not a member of ‘vtkStreamingDemandDrivenPipeline’
  Node_Mesh->Set(vtkStreamingDemandDrivenPipeline::MAXIMUM_NUMBER_OF_PIECES(), 10);

Also, it would be highly appreciated if anyone could show/(send me link) me to a sample example.
Regards

--
Error! Filename not specified.
Sumeet Kumar Sinha
Graduate Student
Phone: (+1) <tel:%28%2B91%29%209910516219> 5306018271<tel:5306018271>
Website : http://www.sumeetksinha.com/

--
<image001.jpg>
Sumeet Kumar Sinha
Graduate Student
Phone: (+1) <tel:%28%2B91%29%209910516219> 5306018271<tel:5306018271>
Website : http://www.sumeetksinha.com/

--
[Image removed by sender.]
Sumeet Kumar Sinha
Graduate Student
Phone: (+1) <tel:%28%2B91%29%209910516219> 5306018271<tel:5306018271>
Website : http://www.sumeetksinha.com/

--
[Image removed by sender.]
Sumeet Kumar Sinha
Graduate Student
Phone: (+1) <tel:%28%2B91%29%209910516219> 5306018271
Website : http://www.sumeetksinha.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20160816/2bac3b1e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 407 bytes
Desc: image001.jpg
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20160816/2bac3b1e/attachment-0001.jpg>