[CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Alan W. Irwin]

On 2018-01-12 11:14+0100 Eric Noulard wrote:

> 2018-01-12 10:58 GMT+01:00 Alan W. Irwin <irwin at beluga.phys.uvic.ca>:
>
>> I am looking into the practicality of using clusters containing ~5
>> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
>> and test software (since even with ccache I am currently spending a
>> lot of my time waiting for tests to complete as I develop my
>> software).  Such clusters would give you ~40 cores which is a lot of
>> computer power for ~$500 or so.  But, of course, the issue with
>> clusters is how to use them efficiently?
>>
>> For one 8-core M3 you would expect
>>
>> make -j<NNN> target
>>
>> to scale linearly with NNN up to the saturation limit of roughly NNN=8
>> (where target depends on ~hundreds of different and mostly independent
>> tests). But my understanding from recent Google searching is that the
>> -j option for GNU Make is not MPI-aware, i.e., only works for the
>> cores for a single computer (node) in a cluster.  So in a cluster you
>> would have to assign the above task to just one node and simply accept
>> that you cannot scale NNN up to the total number of cores in that
>> cluster.
>>
>
>>
> AFAIU what you want is "distributed ctest" not really distributed build
> right?

Not ctest, but actually distributed "build", but with "build" in
quotes because instead of exclusively building software, most of the
targets that are built are configured with
add_custom_command/add_custom_target and actually run tests with
proper dependencies on other test targets as well as targets that
actually build software.  So this testing paradigm makes initial
testing of one particular test much quicker because you don't have to
build the "all" target before the test like is required with ctest.

> For distributed build you may have a look at distcc (http://distcc.org/)
> which seems already used
> by CMake-enabled project like ccache is
> (
> https://stackoverflow.com/questions/32752446/using-compiler-prefix-commands-with-cmake-distcc-ccache/34317588
> )

It looks like distcc could speed up compilations tremendously on a
cluster, but only a minority of my targets are actually builds of
software.  So I need a "distmake".  Just after writing that sentence I
decided I liked that made-up name so on the off-chance someone else
had liked it as well, I did a google search and found the SF distmake project
that apparently does exactly what I need!  Instead of using MPI (Message
Passing Interface) standard to coordinate the targets that are built
between the various nodes, it looks like it uses something a lot
simpler.

> Now I'm not quite sure to understand why you speak of MPI-awareness ?

I thought to coordinate target builds over a cluster, that the make
programme itself would have to have MPI directives inserted into its source
code so the whole build coordination could be handled by MPI.  But it
appears that complexity is not necessary (see distmake, above).

> Are your test including MPI which would mean that some executable are
> already eating up more than one core?

No.

> Or do you have a big collection of mono-processor tests you want to run on
> the cluster?

Yes, but with complete dependencies as explained above.

In sum, it looks like this question now boils down to whether anyone
here has used distmake (or something equivalent to it) on a cluster,
and if so, are there any gotcha's?

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________