[CMake] dependencies of cross compiliations

[Miller Henry]

I like the idea, some tweaks that I think you need

add_host_build - executables built here need to be available not as targets, but as variables as if FIND_EXECUTABLE was called on them. This is probably what you meant, but it didn't come across that way.

In the case where you are cross compiling for more than one target it would be useful if you could build the host tools once and then use that build in multiple target builds. There are two different workflows for this: building locally where the host build directory just needs to be shared (how?); building on a CI system where you want to build/package the host tools and then send the package to downstream jobs.  In my opinion this second workflow is the more important one to have work. 

I would suggest that it work something like this: A new option -DCMAKE_HOST_TOOLS_BUILD_DIR which if set will specify the directory to find host tools in - if this directory already has a built use, the tools already built there (of course re-run that build as required on source changes); otherwise configure and build host tools in that directory.  If this options is NOT specified first run find_package to see if a cmake config file can be found for the correct version of host tools - if so use them (I think we only want to support cmake config files not the other options find_package looks for), it is the users responsibility to update the installed tools if the source changes.  If all of the above fails fall back to some directory under the build directory (as you proposed?).

The above also solves the problem of what if you want to pass a ton of options to the host build.  Adding overrides to cmake options for host vs target (ie -DHOST_CMAKE_BUILD_TYPE) gets out of hand quickly, both from the developer that has to write all the cmake options and also the user who has to maintain long command lines.  

Keep in mind there are some projects where the above cannot work.  I know of one program where the host tool creates data packed for the specific CPU (endian and bits in an int) and so you cannot share then host build. This project doesn't support cross compiling the host tool so I run it under QEMU, but you can imagine it being properly supported in the future and so add_host_build may need to have something to indicate the target to build the host tool for.

Just some thoughts. Since I'm not doing the work I don't get to tell you how to do it.

-----Original Message-----
From: CMake [mailto:cmake-bounces at cmake.org] On Behalf Of Rolf Eike Beer
Sent: Tuesday, November 27, 2018 4:28 AM
To: cmake at cmake.org
Cc: cmake-developers at cmake.org
Subject: Re: [CMake] dependencies of cross compiliations

Am 2018-11-09 10:04, schrieb Torsten Robitzki:
> Hi,
> I hope this question was not asked before. I work in the embedded 
> field and there it is usually to have at least two different build 
> platforms. The Host platform, where unit tests are build (and where 
> CMake is running) and an embedded Target platform, where targets are 
> build with a cross compiler. Sometimes such a system comes with 
> self-written tools that are build and run on the Host platform to 
> build a target for the embedded Target platform (adding meta data to a 
> binary to be used by a bootloader for example).
> 
> Usually I have two different build folders, one for the Host platform 
> and one for the Target platform, using different calls to cmake to 
> choose from a set of tools and targets. But when using this approach, 
> it is necessary that the Host platform build ran before the Target 
> platform build, so that tools that are required for the Target 
> platform are build during the Host target build.
> 
> One solution I’ve came up with, is to build the required tools during 
> the Target platform build, using an add_custom_target() to invoke the 
> Target compiler directly. This works fine, as long as the tools are 
> basically build just out of a couple of files.
> 
> What would be the „CMake-Way“ to add the tools (that have to be build 
> on the Target platform) as dependency to targets that have to be build 
> for the Target (cross compile) platform?

TL;DR: there is not "good" way yet. But there should be one.

I'm hijacking this and move it to the developers list, because that is something "big", and we need to think about how to do that. I find it important to correctly solve this as it would simplify a lot of things. 
Especially given that Qt is thinking to use CMake to build Qt itself, which I bet all of us would love to see. But they will be after us if we don't offer a solution for this. And given the increasing amount of cross-setups these days I'm sure that a lot of other people would benefit.

My first idea was to have something like add_host_executable(), which would only be called when this is not CMAKE_CROSSCOMPILING, but at the end I think this clutters things too much.

Then I came up with:

   add_host_build("relative source dir" "build dir" [VARS])

This would create an entirely new CMake scope (with it's own CMakeCache.txt and the like) in "${CMAKE_CURRENT_BUILD_DIR}/build dir", and would not take the CMAKE_TOOLCHAIN_FILE into account. People are free to pass "." as relative source dir in case they want to start at top level, but they may as well pass "tools", "generators" or whatever they call it. This is not intended to be called multiple times from the same project as it would scan for compiler and environment once for every call, but doing so does no harm beyond being slow(er) and the targets of one such sub-build not being visible to the others.

My idea would be that things added by add_executable() inside such a sub-build are visible as targets from the outer build. Other things like libraries and the like need not to be, they can't be linked in the outer world. The user is free to build with shared libs inside, and running the things from the build tree would result in the correct freshly build libs being picked up because of RPATH or whatever. There is no install of those targets possible from the outer build, this can entirely be managed from the host build. Of course one must be possible to set variables on the host build, that's where VARS comes into play. This holds a list of variable names that will be passed to the hostbuild. No values, to avoid all sorts of quoting issues. Helpful would be a special variable for CMAKE_INSTALL_PREFIX as this needs a bit of attention (and a non-sysroot thing prefix in the toolchain file). Confused? Granted, here is an example:

if (CMAKE_CROSSCOMPILING)
     set(HOST_INSTALL_DIR "/some/where")
     add_host_build(. host HOST_INSTALL_DIR) endif () add_executable(magic magic.cpp) install(TARGETS magic DESTINATION bin) # installs both the host and the target tool!
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND
magic) # will call the host build
if (NOT CMAKE_HOST_BUILD)
     add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
     install(TARGETS foo DESTINATION bin) endif ()

This should end up in a layout like this:

/tmp/install/prefix/tools/bin/magic.exe # Windows host /tmp/install/prefix/sysroot/usr/bin/magic # Unix guest /tmp/install/prefix/sysroot/usr/bin/foo

The toolchain file would look somehow like this:

set(CMAKE_HOST_PREFIX prefix
set(CMAKE_SYSROOT ${CMAKE_HOST_PREFIX}/sysroot)

and the CMake command would look like this:

cmake -D CMAKE_TOOLCHAIN_FILE=tc.cmake -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_HOST_INSTALL_PREFIX=tools ...

The wish-season is coming up, so that's sort of what I would like to have. Now it's your turn. No bikeshedding please, only deliveries ;)

Eike
--
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake