[CMake] CTest for multiple parallel jobs

[Robert Maynard]

On Tue, Apr 7, 2015 at 1:11 PM, Christon, Mark <christon at lanl.gov> wrote:

>  Hi Robert, et al.,
>
>  Thanks for the information.   I believe that ctest –j <#-of-cores> is
> equivalent to specifying the processor count with ctest_test that you
> pointed me to.
>

The PROCESSORS property states how many of #-of-cores a single test will
use, it is meant as a way to make sure you don't use n * n cores.



>
>  This does not correct the issue of distributing multiple jobs across
> multiple nodes on a cluster via ctest.
>
>  Again, considering the prototype below, it is possible to get a workable
> solution to this issue of running multiple parallel jobs with ctest across
> multiple nodes by using a command line to execute each test that looks like
>
>  “mpirun -–mca opal_paffinity_alone 0 –bynode –np …
>
>  The –dynode does a sort of round-robin assignment of tasks to cores and
> keeps the jobs from piling up on the cores associated with the first node.
>
>  Caveats:  This works with openmpi 1.6.5 on our local clusters.   Your
> mileage may vary with other driving conditions.   The trade-off is that you
> pay a bit more in communication, or at least it’s a bit less optimal, but
> the work load is manageable.
>


I believe that ctest doesn't have anything out of the box that does this,
but instead people generally use a test driver program to marshall the
options to mpirun.

>
>  - Mark
>
>  --
> Mark A. Christon
> Computational Physics Group (CCS-2)
> Computer, Computational and Statistical Sciences Division
> Los Alamos National Laboratory
> MS D413, P.O. Box 1663
> Los Alamos, NM 87545
>
>  E-mail: christon at lanl.gov
> Phone : (505) 665-9063
> Mobile: (505) 695-5649 (voice mail)
>
>  International Journal For Numerical Methods in Fluids
> <http://www.wileyonlinelibrary.com/journal/fld>
>
>   From: Robert Maynard <robert.maynard at kitware.com>
> Date: Monday, April 6, 2015 at 2:25 PM
> To: Mark Christon <christon at lanl.gov>
> Cc: "cmake at cmake.org" <cmake at cmake.org>
> Subject: Re: [CMake] CTest for multiple parallel jobs
>
>   You can specify the PROCESSORS (
> http://www.cmake.org/cmake/help/v3.0/prop_test/PROCESSORS.html ) property
> on a test that will tell ctest how man processors from the pool a test
> should take. Note that this still requires that you specify the affinity
> for the MPI processes.
>
> On Mon, Apr 6, 2015 at 4:04 PM, Christon, Mark <christon at lanl.gov> wrote:
>
>>  Hi Folks,
>>
>>  This is actually a ctest question.   It would be nice to be able to run
>> a series of test suites, serial (1-core), parallel (4-core) and long
>> (8-core) with a single execution of ctest using –j <# of cores> across
>> multiple nodes of a cluster.   As a prototype, consider 2 nodes, 24-cores
>> on each node, so a total of 48 cores.
>>
>>  It appears that using ctest –j 48 simply overloads the first node.
>> Now, using –mca opal_paffinity_alone 0 with openmpi does not really
>> correct the issue.
>>
>>  Is there a way to spawn and manage a series of regression tests that
>> range from 1 to 8 cores and have the jobs distributed across all of the
>> available cores with some combination of ctest and mpi options?   So far, I
>> have been unable to find a combination that works properly for more than
>> one node.
>>
>>
>>  - Mark
>>
>>  --
>> Mark A. Christon
>> Computational Physics Group (CCS-2)
>> Computer, Computational and Statistical Sciences Division
>> Los Alamos National Laboratory
>> MS D413, P.O. Box 1663
>> Los Alamos, NM 87545
>>
>>  E-mail: christon at lanl.gov
>> Phone : (505) 665-9063
>> Mobile: (505) 695-5649 (voice mail)
>>
>>  International Journal For Numerical Methods in Fluids
>> <http://www.wileyonlinelibrary.com/journal/fld>
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20150407/76e1dd64/attachment-0001.html>