[CMake] requiring parallel building but sequential linking

[Eric Noulard]

Le lun. 18 févr. 2019 à 00:01, Craig Scott <craig.scott at crascit.com> a
écrit :

>
>
> On Mon, Feb 18, 2019 at 9:35 AM Domen Vrankar <domen.vrankar at gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm building llvm project with CMake.
>> While build process is compiling the code I prefer running "make -j3" on
>> my 4 core pc (bumping processor to 100% on 3 of 4 cores and using up ~5 GB
>> of ram - part of it is system and not build related).
>> While build process is linking I must run "make" with a single job
>> because otherwise I run out of memory (cores do little work but ram usage
>> goes past 23 GB).
>>
>> Currently I'm handling this so that I first run "make -j3" and once I get
>> to about 90% (first larger linking) I hit ctrl+c and run "make".
>>
>> Is there a feature in CMake that would support handling this transition
>> of parallel builds, sequential linking out of the box?
>> (while I'm mostly interested in Linux/make support, having the same
>> feature for Windows/nmake/ninja through the same command line cmake command
>> would be even better)
>>
>
> This is available for Ninja with the JOB_POOL_LINK
> <https://cmake.org/cmake/help/latest/prop_tgt/JOB_POOL_LINK.html> target
> property, the default for which can be set project-wide with the
> CMAKE_JOB_POOL_LINK
> <https://cmake.org/cmake/help/latest/variable/CMAKE_JOB_POOL_LINK.html> variable.
> You can control the number of parallel jobs for a given pool with the
> JOB_POOLS <https://cmake.org/cmake/help/latest/prop_gbl/JOB_POOLS.html> global
> property.
>

I use that a lot and I shall add that, on my side, link jobs memory
consumption heavily depends on the type of build (debug, release,
profiling, etc...), moreover
the available memory amount vary a lot as well whether if the build occurs
on a "local" developer desktop or on some CI runner.
So we end up doing some very basic CMake computation in order to
automatically adapt to the local resource.
Nothing fancy but it has been proven very useful for us:

if (CMAKE_BUILD_TYPE STREQUAL "Debug")
   cmake_host_system_information(RESULT MYMEM QUERY TOTAL_PHYSICAL_MEMORY)
   # Compute the number of authorized number of link jobs by:
   #   - "saving" 4 GiB of memory
   #   - assume each debug link job may consume 2GiB
   math(EXPR NLJ "(${MYMEM}-4096)/2048")
   set_property(GLOBAL PROPERTY JOB_POOLS link_jobs=${NLJ})
   set(CMAKE_JOB_POOL_LINK link_jobs)
elseif (CMAKE_BUILD_TYPE STREQUAL "Release")
...
endif()

-- 
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20190218/ee9e2ac6/attachment.html>