[CMake] [HOWTO] Easily Cross-Compile CMake (with xstatic)
Zach van Rijn
me at zv.io
Fri Oct 26 04:04:41 EDT 2018
Hi Friends,
I have, until recently, been under the impression that CMake is
rather difficult (or impossible?) to cross-compile correctly. I
believe I have devised a sane method of doing so. In addition to
being simple, the output binaries are fully static, so they may
be transferred to any compatible system without dependencies. It
has been tested on CMake version 3.12.3, on 48 platforms, of
which 19 do not build without patches (contribs welcome!)
This method does _not_ require having a recent version of CMake
on the build system, nor does it require any other dependencies
except GNU 'make', 'gcc', 'g++', and some tool like wget/curl to
fetch the source code. We leverage "bootstrap" process products.
The primary motivation for wanting to cross-compile CMake itself
is two-fold: (a) Target systems do not always have compilers for
C++, and not all CMake-based projects require a C++ compiler. If
the system does not have a sufficiently-recent CMake, and (b) An
informative guide such as this might be of interest to the CMake
and/or other communities.
In addition, official binary distributions include only x86_64
Linux and Darwin, and x86(_64) Win32. What about ARM, MIPS, PPC,
others? The obvious solution is to build natively in QEMU, a VM,
or on the target platform if one is fortunate to have this. Some
Linux distributions [1] [2] [3] provide a few such packages.
Note that, this is highly experimental and there are some kinks
that must be resolved before all platforms are supported.
Introduction
------------
Documentation for existing methods [4] is sparse, complicated,
and either incomplete or incorrect (usually stemming from a user
posting a question about how to accomplish this, and the thread
goes stale). In addition to the mailing list, queries to Google,
StackOverflow [5], etc. are often fruitless.
Below is a method that any user should be able to follow in
order to produce the following:
* Static CMake ('cmake', 'cpack', 'ctest') binaries that run
almost anywhere without dependencies;
* The associated 'doc/' and 'share/' directories (some of them
contain files necessary for proper CMake execution); and
* NOT Windows-friendly or MinGW-based binaries. This fails due
to a "fatal error: byteswap.h: No such file or directory"
problem in one of the self-contained (LZMA?) CMake deps, and
I suspect it is trivial to work around this but I've not yet
had the time to try.
The above is achieved using the 'musl' C standard library [6], a
minimal and lightweight implementation designed for static
builds. It may also be possible using 'glibc' if static versions
are available. This has not been tested.
**READ AND UNDERSTAND ALL STEPS BEFORE MODIFYING YOUR SYSTEM!**
Requirements
------------
This method is not dependent upon, but is significantly easier
to perform, if you have access to a Linux-like system that has
some sort of shell (e.g., 'dash' or 'bash'). We make use of
symbolic links and other common *nix commands. Please do not
download, copy/paste, or execute anything that you are uncertain
of or do not understand fully what it is or does.
This process involves *breaking* any existing native (BUILD)
toolchain and it is therefore recommended you perform these
steps in a chroot, container, or virtualized ephemeral space.
You will also want to empty '/usr/local/' prior to starting, or
wisely choose a different installation directory later in this
process to avoid contamination and/or system breakage.
(1) A cross-compilation toolchain, which runs on BUILD and
produces code for HOST. In this example, I use toolchains
from musl.cc [7]. These can produce static code.
- BUILD = x86_64-linux-musl (it's what I'm running)
- HOST = aarch64-linux-musl (anything except MinGW)
From the musl.cc website, this is a '-cross' toolchain.
You may pick any flavor you like, or search online for other
similar toolchains if your platform is not available here. A
toolchain which produces static code is ideal, but should be
compatible with your target (HOST) platform in either case.
(2) A native-compilation toolchain, which is required only for
the "bootstrap" phase of the CMake build process. This does
not need to be from the same site, but that's how I tested.
In this example, you only need to be able to bootstrap.
- BUILD = x86_64-linux-musl (yours can be glibc or other)
- HOST = x86_64-linux-musl (you must be able to execute)
From the musl.cc website, this is a '-native' toolchain.
You do not need this if you have a $CC and $CXX that works
properly (trial and error, but the musl.cc toolchains have
been tested). They need not produce static code.
(3) A modern-ish version of GNU 'make' (I don't know what the
minimum is, but I'm using GNU Make 4.2.1). Others may work;
this probably varies by CMake version. Check their docs?
(4) Patience, and understanding that this process may not work
on your platform(s), and that you assume all risks/etc.
Procedure
---------
The steps detailed below are also encoded in a shell script that
is part of the 'xstatic' project [8]. This is how these methods
were tested, and you may find these scripts convenient. Below is
an annotated procedure (including caveats) when cross-compiling
CMake. Please follow closely and work in an isolated directory.
If you have Docker, you can download pre-packaged versions of
the cross-compiler toolchains from here [9] and build inside.
(1) Set the following environment variables ($WORK is where you
will build CMake; make sure it exists) in your shell:
WORK=/tmp # scratch directory inside cont.
CONF=59e2ce0e6b46 # config.sub replacement (git)
XMCM=aarch64-linux-musl # HOST triple
HVER=x86_64-linux-musl # BUILD triple
VCMK=3.12.3 # CMake version to build
(2) Download a musl.cc native (BUILD) compiler toolchain. You do
not need to do this if you have a compatible compiler, but I
strongly recommend it. We will *BREAK* this compiler later.
$ cd ${WORK}
$ curl -so ${HVER}-native.tgz https://musl.cc/${HVER}-native.tgz
$ tar -xf ${HVER}-native.tgz
$ rm ${HVER}-native.tgz
(3) Download a musl.cc cross (HOST) compiler toolchain. This is
also not required, but is almost guaranteed necessary unless
you have a musl-based (or static glibc-based, etc.) one. The
cross toolchains on this website are intended for use on
x86_64 Linux systems (i686 ones are available, too).
$ cd ${WORK}
$ curl -so ${XMCM}-cross.tgz https://musl.cc/${XMCM}-cross.tgz
$ tar -xf ${XMCM}-cross.tgz
$ rm ${XMCM}-cross.tgz
The Docker images are presently configured like so. DO NOT
PERFORM THESE STEPS OUTSIDE OF A CONTAINER OR VM! These will
likely break your existing system compiler(s). Only do this
if you do not have a system compiler.
# cd ${WORK}
# cd ${XMCM}-cross
# rsync -rLq . / # WRITES TO ROOT FILESYSTEM!
# cd /bin
# rename "${XMCM}-" "" *
This causes 'gcc' to refer to the cross (HOST) compiler, and
'${WORK}/${HVER}-native/bin/gcc' to the native (BUILD) one.
Instead, you can modify your $PATH variable to point to the
correct compiler binaries, at '${WORK}/${HVER}-native/bin/'.
One other (possibly) important detail is that I've wrapped
the three compilers ('gcc', 'g++', and 'gfortran') like so:
for k in g++ gcc gfortran; do
p=$(which ${k}) || true
[ ! "x${p}" = "x" ] || continue;
[ ! -e ${p}.orig ] || continue;
mv ${p} ${p}.orig
echo > ${p} '#!/bin/sh'
echo >> ${p} "${k}.orig \${@} -static --static -g0 -s -Os"
chmod +x ${p}
done
The reason is simple: to force, absolutely, that all object
code produced by the compilers to be static, stripped, and
optimized for size. Not tested without it. Season to taste.
(4) Download and unpack the CMake source code from a tarball:
$ curl -so cmake-${VCMK}.tar.gz \
https://cmake.org/files/v$(echo ${VCMK} \
| awk -F'[.]' '{print $1 "." $2}')/cmake-${VCMK}.tar.gz
$ tar -xf cmake-${VCMK}.tar.gz
$ rm cmake-${VCMK}.tar.gz
$ cd cmake-${VCMK}
(5) "Bootstrap" CMake using the native (BUILD) compiler (this is
one long command):
$ ${WORK}/${HVER}-native/bin/gcc --version && \
CC="${WORK}/${HVER}-native/bin/gcc" \
CFLAGS="-static --static" \
CXX="${WORK}/${HVER}-native/bin/g++" \
CXXFLAGS="-static --static" \
./bootstrap --parallel=$(nproc)
(6) Overwrite (break) the native (BUILD) compiler by creating
symbolic links to is components. Until further modifications
are made to CMake itself, this is the easiest way to ensure
that the correct compilers are picked up.
In this example, it is expected that "$(which gcc)" refers
to the cross (HOST) compiler, and the native (BUILD) one is:
'${WORK}/${HVER}-native/bin/gcc'. Adjust paths accordingly.
$ ln -sf $(which g++ ) ${WORK}/${HVER}-native/bin/g++
$ ln -sf $(which gcc ) ${WORK}/${HVER}-native/bin/gcc
$ ln -sf $(which ld ) ${WORK}/${HVER}-native/bin/ld
$ ln -sf $(which strip) ${WORK}/${HVER}-native/bin/strip
(7) Build the "real" CMake binaries using the cross (HOST)
compilers (which is now accessible via the location of the
old (now broken) native (BUILD) compilers. The previous step
of "bootstrapping" already configured CMake.
$ make -j$(nproc)
(8) Move the "real" CMake binary elsewhere (for now) as we need
to use a CMake binary that runs on the BUILD system, but the
one we just compiled only runs on HOST. So, we'll utilize a
special version of CMake (the one built during "bootstrap")
to complete the next step.
$ mv bin/cmake cmake.real # symlink is disallowed!
$ cp Bootstrap.cmk/cmake bin/cmake
(9) The typical "make install" procedure, frustratingly requires
a functioning CMake executable, and its path is hard-coded
to be the one we built in the previous step. The "bootstrap"
CMake binary will not work anywhere outside of the project
directory, but it will suffice just for installation.
$ make install
$ mv cmake.real /usr/local/bin/cmake
Now, all files contained in '/usr/local/' (or wherever you may
choose to $DESTDIR it to) are built statically for the HOST. You
no longer have a functioning BUILD compiler unless you renamed
the files before overwriting the executables with symbolic links
to your HOST compiler.
Testing
-------
Copy these files somewhere on a compatible system. They do not
need to be moved to '/usr/local/' at all; your $HOME directory
is sufficient.
$ cd /usr/local/bin
$ file *
cmake: ELF 64-bit LSB executable, ARM aarch64, version 1
(SYSV), statically linked, stripped
cpack: ELF 64-bit LSB executable, ARM aarch64, version 1
(SYSV), statically linked, stripped
ctest: ELF 64-bit LSB executable, ARM aarch64, version 1
(SYSV), statically linked, stripped
$ ls -l
total 13692
-rwxr-xr-x 1 zv zv 4518992 Oct 25 22:15 cmake
-rwxr-xr-x 1 zv zv 4510296 Oct 25 22:15 cpack
-rwxr-xr-x 1 zv zv 4982384 Oct 25 22:15 ctest
$ ./cmake --version
cmake version 3.12.3
CMake suite maintained and supported by Kitware
(kitware.com/cmake).
It is also possible to use these binaries within QEMU if there
is a compatible user-level one available:
$ qemu-aarch64 ./cmake --version
cmake version 3.12.3
CMake suite maintained and supported by Kitware
(kitware.com/cmake).
One example of a CMake-based C-only (no C++) project is the
SUNDIALS library from LLNL [10]. The system on which I am
testing this does not have CMake installed otherwise.
$ which cmake
/home/zv/aarch64-linux-musleabi/bin/cmake
$ SVER=3.2.1
$ wget https://computation.llnl.gov/projects/sundials/download/\
sundials-${SVER}.tar.gz
$ tar -xf sundials-${SVER}.tar.gz
$ rm sundials-${SVER}.tar.gz
$ mkdir sundials-${SVER}/build
$ cd sundials-${SVER}/build
$ cmake ..
$ make -j$(nproc)
Download
--------
Pre-built (again using 'xstatic') binaries are available for
testing. Please verify the checksums and build your own from
source before using them in any non-testing environment.
https://xstatic.musl.cc/cmake/
The following platforms DID NOT build successfully. Your mileage
may vary (depending on compiler, C/C++ libraries, etc.) so
please report any successful combinations you may discover:
* i686/x86_64 PE32 using MinGW; I suspect adding this:
-DCMAKE_SYSTEM_NAME:STRING="Windows"
might resolve the issue and I'll test in due course.
* MicroBlaze (LE/BE); probably needs -fPIC or a patch:
BFD (GNU Binutils) 2.31.1 assertion fail
../../src_binutils/bfd/elf32-microblaze.c:1542
* microblaze-linux-musl
* microblazeel-linux-musl
* mips-linux-musl
* mips-linux-musln32sf
* mips-linux-muslsf
* mips64-linux-musln32
* mips64-linux-musln32sf
* mips64el-linux-musln32
* mips64el-linux-musln32sf
* mipsel-linux-musl
* mipsel-linux-musln32
* mipsel-linux-musln32sf
* mipsel-linux-muslsf
* or1k-linux-musl
* riscv32-linux-musl
* riscv64-linux-musl
* x86_64-linux-musl (odd, but OK, same kind of error)
These may actually be issues with the above method, or with the
provided toolchains, and I will investigate these further when
time permits. It seems, however, that things are mostly working.
Regards,
ZV
References
----------
[1] https://packages.debian.org/search?keywords=cmake
[2] https://pkgs.alpinelinux.org/packages?name=cmake&branch=edge
[3] https://packages.ubuntu.com/search?keywords=cmake
[4] https://www.mail-archive.com/search?q=cross-compile&l=cmake%
40cmake.org (not quite 'porting CMake to other platforms') and
https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/Cross
Compiling (mainly for CMake-project developers)
[5] https://stackoverflow.com/search?q=cross-compile+cmake
[6] https://www.musl-libc.org/
[7] https://musl.cc/
[8] https://git.zv.io/me/xstatic
[9] https://hub.docker.com/r/muslcc/x86_64/tags/
[10] https://computation.llnl.gov/projects/sundials/
More information about the CMake
mailing list