<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 1:11 PM, Christon, Mark <span dir="ltr"><<a href="mailto:christon@lanl.gov" target="_blank">christon@lanl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:16px;font-family:Helvetica,sans-serif">
<div>
<div>Hi Robert, et al.,</div>
<div><br>
</div>
<div>Thanks for the information. I believe that ctest –j <#-of-cores> is equivalent to specifying the processor count with ctest_test that you pointed me to. </div></div></div></blockquote><div><br></div>The PROCESSORS property states how many of #-of-cores a single test will use, it is meant as a way to make sure you don't use n * n cores.<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:16px;font-family:Helvetica,sans-serif"><div>
<div><br>
</div>
<div>This does not correct the issue of distributing multiple jobs across multiple nodes on a cluster via ctest.</div>
<div><br>
</div>
<div>Again, considering the prototype below, it is possible to get a workable solution to this issue of running multiple parallel jobs with ctest across multiple nodes by using a command line to execute each test that looks like</div>
<div><br>
</div>
<div>“mpirun -–mca opal_<span style="font-style:italic">paffinity</span>_alone 0 –bynode –np …</div>
<div><br>
</div>
<div>The –dynode does a sort of round-robin assignment of tasks to cores and keeps the jobs from piling up on the cores associated with the first node.</div>
<div><br>
</div>
<div>Caveats: This works with openmpi 1.6.5 on our local clusters. Your mileage may vary with other driving conditions. The trade-off is that you pay a bit more in communication, or at least it’s a bit less optimal, but the work load is manageable.</div></div></div></blockquote><div><br></div><div><br></div><div>I believe that ctest doesn't have anything out of the box that does this, but instead people generally use a test driver program to marshall the options to mpirun. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:16px;font-family:Helvetica,sans-serif"><div><span class="">
<div><br>
</div>
<div>
<div>- Mark</div>
<div><br>
</div>
<div>-- </div>
<div>Mark A. Christon</div>
<div>Computational Physics Group (CCS-2)</div>
<div>Computer, Computational and Statistical Sciences Division</div>
<div>Los Alamos National Laboratory</div>
<div>MS D413, P.O. Box 1663</div>
<div>Los Alamos, NM 87545</div>
<div><br>
</div>
<div>E-mail: <a href="http://christon@lanl.gov" target="_blank">christon@lanl.gov</a></div>
<div>Phone : <a href="tel:%28505%29%20665-9063" value="+15056659063" target="_blank">(505) 665-9063</a> </div>
<div>Mobile: <a href="tel:%28505%29%20695-5649" value="+15056955649" target="_blank">(505) 695-5649</a> (voice mail)</div>
<div><br>
</div>
<div><a href="http://www.wileyonlinelibrary.com/journal/fld" target="_blank">International Journal For Numerical Methods in Fluids</a></div>
</div>
</span></div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;border-top-color:rgb(181,196,223)">
<span style="font-weight:bold">From: </span>Robert Maynard <<a href="mailto:robert.maynard@kitware.com" target="_blank">robert.maynard@kitware.com</a>><br>
<span style="font-weight:bold">Date: </span>Monday, April 6, 2015 at 2:25 PM<br>
<span style="font-weight:bold">To: </span>Mark Christon <<a href="mailto:christon@lanl.gov" target="_blank">christon@lanl.gov</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:cmake@cmake.org" target="_blank">cmake@cmake.org</a>" <<a href="mailto:cmake@cmake.org" target="_blank">cmake@cmake.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [CMake] CTest for multiple parallel jobs<br>
</div><div><div class="h5">
<div><br>
</div>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div>
<div>
<div dir="ltr">You can specify the PROCESSORS ( <a href="http://www.cmake.org/cmake/help/v3.0/prop_test/PROCESSORS.html" target="_blank">http://www.cmake.org/cmake/help/v3.0/prop_test/PROCESSORS.html</a> ) property on a test that will tell ctest how man processors from the
pool a test should take. Note that this still requires that you specify the affinity for the MPI processes. </div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Apr 6, 2015 at 4:04 PM, Christon, Mark <span dir="ltr">
<<a href="mailto:christon@lanl.gov" target="_blank">christon@lanl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:16px;font-family:Helvetica,sans-serif">
<div>Hi Folks,</div>
<div><br>
</div>
<div>This is actually a ctest question. It would be nice to be able to run a series of test suites, serial (1-core), parallel (4-core) and long (8-core) with a single execution of ctest using –j <# of cores> across multiple nodes of a cluster. As a prototype,
consider 2 nodes, 24-cores on each node, so a total of 48 cores.</div>
<div><br>
</div>
<div>It appears that using ctest –j 48 simply overloads the first node. Now, using –mca opal_<span style="font-style:italic">paffinity</span>_alone 0 with openmpi does not really correct the issue.</div>
<div><br>
</div>
<div>Is there a way to spawn and manage a series of regression tests that range from 1 to 8 cores and have the jobs distributed across all of the available cores with some combination of ctest and mpi options? So far, I have been unable to find a combination
that works properly for more than one node.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>- Mark</div>
<div><br>
</div>
<div>-- </div>
<div>Mark A. Christon</div>
<div>Computational Physics Group (CCS-2)</div>
<div>Computer, Computational and Statistical Sciences Division</div>
<div>Los Alamos National Laboratory</div>
<div>MS D413, P.O. Box 1663</div>
<div>Los Alamos, NM 87545</div>
<div><br>
</div>
<div>E-mail: <a href="http://christon@lanl.gov" target="_blank">christon@lanl.gov</a></div>
<div>Phone : <a href="tel:%28505%29%20665-9063" value="+15056659063" target="_blank">
(505) 665-9063</a> </div>
<div>Mobile: <a href="tel:%28505%29%20695-5649" value="+15056955649" target="_blank">
(505) 695-5649</a> (voice mail)</div>
<div><br>
</div>
<div><a href="http://www.wileyonlinelibrary.com/journal/fld" target="_blank">International Journal For Numerical Methods in Fluids</a></div>
</div>
</div>
<br>
--<br>
<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" target="_blank">
http://www.cmake.org/Wiki/CMake_FAQ</a><br>
<br>
Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br>
<br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" target="_blank">
http://cmake.org/cmake/help/support.html</a><br>
CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" target="_blank">
http://cmake.org/cmake/help/consulting.html</a><br>
CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" target="_blank">
http://cmake.org/cmake/help/training.html</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/cmake" target="_blank">http://public.kitware.com/mailman/listinfo/cmake</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div></div></span>
</div>
</blockquote></div><br></div></div>