[CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Eric Noulard]

2018-01-12 10:58 GMT+01:00 Alan W. Irwin <irwin at beluga.phys.uvic.ca>:

> I am looking into the practicality of using clusters containing ~5
> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
> and test software (since even with ccache I am currently spending a
> lot of my time waiting for tests to complete as I develop my
> software).  Such clusters would give you ~40 cores which is a lot of
> computer power for ~$500 or so.  But, of course, the issue with
> clusters is how to use them efficiently?
>
> For one 8-core M3 you would expect
>
> make -j<NNN> target
>
> to scale linearly with NNN up to the saturation limit of roughly NNN=8
> (where target depends on ~hundreds of different and mostly independent
> tests). But my understanding from recent Google searching is that the
> -j option for GNU Make is not MPI-aware, i.e., only works for the
> cores for a single computer (node) in a cluster.  So in a cluster you
> would have to assign the above task to just one node and simply accept
> that you cannot scale NNN up to the total number of cores in that
> cluster.
>

>
AFAIU what you want is "distributed ctest" not really distributed build
right?
For distributed build you may have a look at distcc (http://distcc.org/)
which seems already used
by CMake-enabled project like ccache is
(
https://stackoverflow.com/questions/32752446/using-compiler-prefix-commands-with-cmake-distcc-ccache/34317588
)

Now I'm not quite sure to understand why you speak of MPI-awareness ?
Are your test including MPI which would mean that some executable are
already eating up more than one core?
Or do you have a big collection of mono-processor tests you want to run on
the cluster?

-- 
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180112/49a275b6/attachment.html>