<div dir="ltr"><div><div><div><div>Hi Tim,<br><br></div>Yep, you're use case looks like a really good choice to do in transit. As a start I'd suggest doing the slice in situ with Catalyst, write it out to a .pvtp file every nth time step and then having a separate process reading the files in (don't read the nth step until the nth+1 .pvtp file is available to make sure all of the data from the nth step is written to disk). In PV 5.2 (and maybe in PV 5.1.2) only the processes that have data will write out the .vtp files when using the .pvtp writer. Then in the next step you can work on using something like ADIOS to send the data from Catalyst to the separate running process. This isn't yet a trivial thing to do with Catalyst but I'm working on a small project to do exactly this so hopefully in the near future there will be more details on how to set this up. Hopefully I'll find time to put this in the Catalyst User's Guide so that others can easily customize it for their use case.<br><br></div>As an alternative, you may be able to do this through the Live functionality and a pvpython/pvbatch process. Note that with the Live connection in the Catalyst script you can specify how often to send data from the simulation to a pvserver.<br><br></div>Best,<br></div>Andy<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 5:02 PM, Gallagher, Timothy P <span dir="ltr"><<a href="mailto:tim.gallagher@gatech.edu" target="_blank">tim.gallagher@gatech.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-1129704688037279450divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi Andy,</p>
<p><br>
</p>
<p>I'll take a look at the in transit approach. I'm essentially extracting a slice every N steps and collecting it as a single slice on a single processor each time, accumulating the slices as I go. Then every M*N steps, I want that single processor to do an
expensive operation and save the output. <br>
</p>
<p><br>
</p>
<p>So if the in transit approach you mentioned would work well for that, I'll give it a shot. I'm doing this on some SGI and Cray machines, I don't know if that has special ways to do this like you mentioned exists at NERSC.
<br>
</p>
<p><br>
</p>
<p>Thanks,</p>
<p><br>
</p>
<p>Tim<br>
</p>
<br>
<br>
<div style="color:rgb(49,55,57)">
<hr style="display:inline-block;width:98%">
<div id="m_-1129704688037279450divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Andy Bauer <<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>><br>
<b>Sent:</b> Tuesday, October 25, 2016 4:43 PM<br>
<b>To:</b> Ufuk Utku Turuncoglu (BE)<br>
<b>Cc:</b> Gallagher, Timothy P; <a href="mailto:paraview@paraview.org" target="_blank">paraview@paraview.org</a><br>
<b>Subject:</b> Re: [Paraview] Non-blocking coprocessing</font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Tim,<br>
<br>
</div>
This may be better to do as an in transit set up. This way the processes would be independent. Through Catalyst I'd worry about all of the processes waiting on the global rank 0 doing work before all of the other Catalyst ranks return control to the simulation.
Depending on the system you're on you could do this communication through file IO for something like Cori@NERSC with its burst buffers.
<br>
<br>
</div>
If you want to do down the in transit path, let me know and I can see about digging up some scripts that I had for that.<br>
<br>
</div>
Best,<br>
</div>
Andy<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Oct 25, 2016 at 3:57 AM, Ufuk Utku Turuncoglu (BE)
<span dir="ltr"><<a href="mailto:u.utku.turuncoglu@be.itu.edu.tr" target="_blank">u.utku.turuncoglu@be.itu.edu.<wbr>tr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF">
<div class="m_-1129704688037279450m_-3727395816115817685moz-cite-prefix">Hi Tim,<br>
<br>
I am not sure about the non-blocking type communication is supported by ParaView, Catalyst or not but i think that assigning an extra core for global reduction is possible. You could use MPI communication for this purpose. So, look at following code of mine
for overloaded coprocessorinitializewithpytho<wbr>n function. As you can see, it also gets the MPI communicatior and it allows to use a pool of processor (or cores) for co-processing. In my case, it is running smoothly without any problem. I hope it helps.<br>
<br>
--ufuk<br>
<br>
extern "C" void my_coprocessorinitializewithpy<wbr>thon_(int *fcomm, const char* pythonScriptName, const char strarr[][255], int *size) {<br>
if (pythonScriptName != NULL) {<br>
if (!g_coprocessor) {<br>
g_coprocessor = vtkCPProcessor::New();<br>
MPI_Comm handle = MPI_Comm_f2c(*fcomm);<br>
vtkMPICommunicatorOpaqueComm *Comm = new vtkMPICommunicatorOpaqueComm(&<wbr>handle);<br>
g_coprocessor->Initialize(*Com<wbr>m);<br>
vtkSmartPointer<vtkCPPythonScr<wbr>iptPipeline> pipeline = vtkSmartPointer<vtkCPPythonScr<wbr>iptPipeline>::New();<br>
pipeline->Initialize(pythonScr<wbr>iptName);<br>
g_coprocessor->AddPipeline(pip<wbr>eline);<br>
//pipeline->FastDelete();<br>
}<br>
<br>
if (!g_coprocessorData) {<br>
g_coprocessorData = vtkCPDataDescription::New();<br>
// must be input port for all model components and for all dimensions<br>
for (int i = 0; i < *size; i++) {<br>
g_coprocessorData->AddInput(st<wbr>rarr[i]);<br>
std::cout << "adding input port [" << i << "] = " << strarr[i] << std::endl;
<div>
<div class="m_-1129704688037279450h5"><br>
}<br>
}<br>
}<br>
}<br>
<br>
On 25/10/16 01:56, Gallagher, Timothy P wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="m_-1129704688037279450h5">
<div id="m_-1129704688037279450m_-3727395816115817685divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hello again!</p>
<p><br>
</p>
<p>I'm looking at using coprocessing for something that may take awhile to actually compute, so I would like to do it in a non-blocking fashion. Essentially I am going to be extracting data from the simulation into some numpy arrays (so once copied, the original
data in the pipeline can change) and then send it to the root processor to do some global operations.
<br>
</p>
<p><br>
</p>
<p>The global operations may take some time (not minutes, but longer than I want my simulation to wait for it to complete). Is there a way to do part of the pipeline in a non-blocking fashion, where the script calls a function that will write out a data file
when processing and then returns control to the simulation prior to the function completion? Will I have to do something in native-python, like spawn a new thread to do the function call, or is there a way to do it with how Paraview operates?</p>
<p><br>
</p>
<p>On a related note, I may not want to have the root processor of the coprocessing to have any simulation code running on it. If I am running my simulation on N cores, is it possible to have N+1 cores running the coprocessor pipeline where the extra core receives
the global data reduction from the N cores and does the crunching? Or am I starting to ask for too much there?</p>
<p><br>
</p>
<p>Thanks as always,</p>
<p><br>
</p>
<p>Tim<br>
</p>
</div>
<br>
<fieldset class="m_-1129704688037279450m_-3727395816115817685mimeAttachmentHeader"></fieldset> <br>
</div>
</div>
<pre>______________________________<wbr>_________________
Powered by <a class="m_-1129704688037279450m_-3727395816115817685moz-txt-link-abbreviated" href="http://www.kitware.com" target="_blank">www.kitware.com</a>
Visit other Kitware open-source projects at <a class="m_-1129704688037279450m_-3727395816115817685moz-txt-link-freetext" href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensou<wbr>rce/opensource.html</a>
Please keep messages on-topic and check the ParaView Wiki at: <a class="m_-1129704688037279450m_-3727395816115817685moz-txt-link-freetext" href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaV<wbr>iew</a>
Search the list archives at: <a class="m_-1129704688037279450m_-3727395816115817685moz-txt-link-freetext" href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=<wbr>ParaView</a>
Follow this link to subscribe/unsubscribe:
<a class="m_-1129704688037279450m_-3727395816115817685moz-txt-link-freetext" href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mail<wbr>man/listinfo/paraview</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
<br>
______________________________<wbr>_________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">
http://www.kitware.com/opensou<wbr>rce/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank">
http://paraview.org/Wiki/ParaV<wbr>iew</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" rel="noreferrer" target="_blank">
http://markmail.org/search/?q=<wbr>ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mail<wbr>man/listinfo/paraview</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br></div>