[CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Eric Noulard]

2018-01-12 11:14 GMT+01:00 Eric Noulard <eric.noulard at gmail.com>:

>
>
> 2018-01-12 10:58 GMT+01:00 Alan W. Irwin <irwin at beluga.phys.uvic.ca>:
>
>> I am looking into the practicality of using clusters containing ~5
>> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
>> and test software (since even with ccache I am currently spending a
>> lot of my time waiting for tests to complete as I develop my
>> software).  Such clusters would give you ~40 cores which is a lot of
>> computer power for ~$500 or so.  But, of course, the issue with
>> clusters is how to use them efficiently?
>>
>> For one 8-core M3 you would expect
>>
>> make -j<NNN> target
>>
>> to scale linearly with NNN up to the saturation limit of roughly NNN=8
>>
>
A side note on that.
Sometimes you want to control the 'load' more than the "parallelism".
With make (or ninja) you can ask for **load** limitation using:

make -l <N>
and
ninja -l <N>

i.e. if you have 8 core and you want to keep 1 of them "calm" then
make -l 7

is interesting.


>
> AFAIU what you want is "distributed ctest" not really distributed build
> right?
> For distributed build you may have a look at distcc (http://distcc.org/)
> which seems already used
> by CMake-enabled project like ccache is
> (https://stackoverflow.com/questions/32752446/using-
> compiler-prefix-commands-with-cmake-distcc-ccache/34317588)
>
> Now I'm not quite sure to understand why you speak of MPI-awareness ?
> Are your test including MPI which would mean that some executable are
> already eating up more than one core?
> Or do you have a big collection of mono-processor tests you want to run on
> the cluster?
>

Concerning high-level distributed execution on a cluster I would have a
look at something
like OpenPBS: http://www.rguha.net/writing/pub/opbs.html (now apparently
http://pbspro.org/).

Once you have your PBS cluster setup I would say that you would need to
write a small test driver
that would launch the tests on the cluster instead of launching them
locally.


-- 
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180112/bcb62f92/attachment.html>