[CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Alan W. Irwin]

I am looking into the practicality of using clusters containing ~5
cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
and test software (since even with ccache I am currently spending a
lot of my time waiting for tests to complete as I develop my
software).  Such clusters would give you ~40 cores which is a lot of
computer power for ~$500 or so.  But, of course, the issue with
clusters is how to use them efficiently?

For one 8-core M3 you would expect

make -j<NNN> target

to scale linearly with NNN up to the saturation limit of roughly NNN=8
(where target depends on ~hundreds of different and mostly independent
tests). But my understanding from recent Google searching is that the
-j option for GNU Make is not MPI-aware, i.e., only works for the
cores for a single computer (node) in a cluster.  So in a cluster you
would have to assign the above task to just one node and simply accept
that you cannot scale NNN up to the total number of cores in that
cluster.

I did find one reference to an effort (called the pmake project at
Lawrence Livermore, see <https://computing.llnl.gov/code/pmake.html>)
to modify GNU make to become MPI-aware so the above command would
roughly scale NNN up to the number of cores in the cluster.  But the
single developer of the project seemed unable to overcome a bug in the
result so that parallel make project was abandoned.

I hope someone here can point me to a make variant (does Ninja
qualify?) that does have the capability of scaling NNN up to the
number of cores in the cluster since that would make clusters with a
large number of cores become extremely useful for the purpose of quick
testing by software developers.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________