[CMake] CTest for multiple parallel jobs

[Christon, Mark]

Hi Robert, et al.,

Thanks for the information.   I believe that ctest –j <#-of-cores> is equivalent to specifying the processor count with ctest_test that you pointed me to.

This does not correct the issue of distributing multiple jobs across multiple nodes on a cluster via ctest.

Again, considering the prototype below, it is possible to get a workable solution to this issue of running multiple parallel jobs with ctest across multiple nodes by using a command line to execute each test that looks like

“mpirun -–mca opal_paffinity_alone 0 –bynode –np …

The –dynode does a sort of round-robin assignment of tasks to cores and keeps the jobs from piling up on the cores associated with the first node.

Caveats:  This works with openmpi 1.6.5 on our local clusters.   Your mileage may vary with other driving conditions.   The trade-off is that you pay a bit more in communication, or at least it’s a bit less optimal, but the work load is manageable.

- Mark

--
Mark A. Christon
Computational Physics Group (CCS-2)
Computer, Computational and Statistical Sciences Division
Los Alamos National Laboratory
MS D413, P.O. Box 1663
Los Alamos, NM 87545

E-mail: christon at lanl.gov
Phone : (505) 665-9063
Mobile: (505) 695-5649 (voice mail)

International Journal For Numerical Methods in Fluids<www.wileyonlinelibrary.com/journal/fld>

From: Robert Maynard <robert.maynard at kitware.com<mailto:robert.maynard at kitware.com>>
Date: Monday, April 6, 2015 at 2:25 PM
To: Mark Christon <christon at lanl.gov<mailto:christon at lanl.gov>>
Cc: "cmake at cmake.org<mailto:cmake at cmake.org>" <cmake at cmake.org<mailto:cmake at cmake.org>>
Subject: Re: [CMake] CTest for multiple parallel jobs

You can specify the PROCESSORS ( http://www.cmake.org/cmake/help/v3.0/prop_test/PROCESSORS.html ) property on a test that will tell ctest how man processors from the pool a test should take. Note that this still requires that you specify the affinity for the MPI processes.

On Mon, Apr 6, 2015 at 4:04 PM, Christon, Mark <christon at lanl.gov<mailto:christon at lanl.gov>> wrote:
Hi Folks,

This is actually a ctest question.   It would be nice to be able to run a series of test suites, serial (1-core), parallel (4-core) and long (8-core) with a single execution of ctest using –j <# of cores> across multiple nodes of a cluster.   As a prototype, consider 2 nodes, 24-cores on each node, so a total of 48 cores.

It appears that using ctest –j 48 simply overloads the first node.   Now, using –mca opal_paffinity_alone 0 with openmpi does not really correct the issue.

Is there a way to spawn and manage a series of regression tests that range from 1 to 8 cores and have the jobs distributed across all of the available cores with some combination of ctest and mpi options?   So far, I have been unable to find a combination that works properly for more than one node.


- Mark

--
Mark A. Christon
Computational Physics Group (CCS-2)
Computer, Computational and Statistical Sciences Division
Los Alamos National Laboratory
MS D413, P.O. Box 1663
Los Alamos, NM 87545

E-mail: christon at lanl.gov<http://christon@lanl.gov>
Phone : (505) 665-9063<tel:%28505%29%20665-9063>
Mobile: (505) 695-5649<tel:%28505%29%20695-5649> (voice mail)

International Journal For Numerical Methods in Fluids<http://www.wileyonlinelibrary.com/journal/fld>

--

Powered by www.kitware.com<http://www.kitware.com>

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20150407/37c4d43a/attachment.html>