[CMake] Issues trying to use the Anaconda compiler tools with CMake
Ray Donnelly
mingw.android at gmail.com
Wed Aug 15 06:44:35 EDT 2018
Hi Sebastián,
Without having time to properly go through this, I feel I should
correct some technical inaccuracies, but *all* of your issues can be
sidestepped by using conda-build. Installation and RPATHs are always
very tricky for projects to get right so we side step any issues here
by running post-build fixups on these things. We ensure RPATHs have
the opportunity to add our own, uniquify them (I think!), and make
them fully relative on macOS and Linux.
WRT RPATHs being added or not, we set LDFLAGs for RPATHs the linker
activation script *when run under conda-build*. If you want to fake
that so you get the exact same flags that are used to compile our
packages do: CONDA_BUILD=1 conda activate envname. If you don't set
CONDA_BUILD=1 then it'll set all flags *except* -I${PREFIX}/include,
-L${PREFIX}/lib and -Wl,-rpath,${PREFIX}/lib.
If you want to use our compilers in their completely 'raw' form (of
course you could unset LDFLAGS, CFLAGS, CPPFLAGS too), then install
e.g. gcc_impl_linux-64 instead of gcc_linux-64 which really are
helpers for interfacing the raw compilers with conda-build (and
setting good default compilation flags for security and performance
reasons).
You mentioned using [DY]LD_LIBRARY_PATH, I would caution no one to do
this ever outside of development workflows. FWIW, we run into lots of
trouble with LD_LIBRARY_PATH and I am *firmly* in the camp that
LD_LIBRARY_PATH is harmful (i.e. never use it systematically) and I am
utterly convinced that DT_RUNPATH is a huge mistake. We actually
consider detection of DT_RUNPATH in any of our DSOs or exes an error
condition and only allow DT_RPATH. We just had so many bugs due to the
wrong libs being used by end users due to this, for example there is
no way a Qt application linked against our Qt libs is going to be
happy with some random system Qt library that's been interposed.
For macOS, you should also set CONDA_BUILD_SYSROOT=<somewhere that
macOS 10.9, 10.10 or 10.11 SDK> (depending on the macOS version you
want to target). These old SDKs can be gotten from old Xcode builds
and also on github. Unfortunately the compilers are not compatible
with the new .tbd file format used in newer SDKs and by the newest
Apple system linker and the source to enable that has not been
released yet (there have been no new source drops for Apple's ld64 in
a while).
But please, just use conda-build! While we try to keep them working in
as many situations as we can, these tools are primarily focussed
around conda-build and the technical decisions we make with respect to
them will be from that perspective.
Hope this helps some?
Fri, Aug 10, 2018 at 10:48 PM, Sebastián Mancilla <smancill at jlab.org> wrote:
> I am trying to use Conda as a package manager for isolated C++ development
> environments. But unfortunately, when using CMake with the Anaconda-provided
> compilers [1] (which are used to compile the binary packages in the Anaconda
> repositories), things do not work as expected.
>
> I have a small test case available here [2], with an executable calling a
> shared library and a third-party dependency installed with Conda.
>
> [1]:
> https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html
> [2]: https://gist.github.com/smancill/b28ca07ac11fdf285b4d559545a1630b
>
> --------------------------------------------------
>
> First, when using the system compiler, all works fine (but I am not sure of
> the
> binary compatibility with the Conda packages, that's why I want to use the
> Anaconda compilers):
>
> # create the environment and install XercesC
> $ conda create -n test-system xerces-c
> ...
> environment location:
> /Users/smancill/.local/share/miniconda3/envs/test-system
> ...
> The following NEW packages will be INSTALLED:
>
> icu: 58.2-h4b95b61_1
> libcxx: 4.0.1-h579ed51_0
> libcxxabi: 4.0.1-hebd6815_0
> xerces-c: 3.2.1-h44e365a_0
> ...
>
> # activate the environment
> $ conda activate test-system
>
> $ mkdir build-osx-system
> $ cd build-osx-system
> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX
> -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
> -- The CXX compiler identification is AppleClang 9.0.0.9000039
> -- Check for working CXX compiler: /usr/bin/c++
> -- Check for working CXX compiler: /usr/bin/c++ -- works
> ...
> -- Found XercesC:
> /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib
> (found version "3.2.1")
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /Users/smancill/src/conda-test/build-osx-system
>
> $ make -j1 VERBOSE=1
> ...
> [100%] Linking CXX executable bar
> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt
> --verbose=1
> /usr/bin/c++ -isysroot
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> -mmacosx-version-min=10.12 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names CMakeFiles/bar.dir/bar.cpp.o -o bar
> -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-system
> -Wl,-rpath,/Users/smancill/.local/share/miniconda3/envs/test-system/lib
> libfoo.dylib
> /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib
> ...
>
> The build directory (~/src/conda-test/build-osx-system) and the conda
> environment lib directory (~/.local/share/miniconda3/envs/test-system/lib)
> are correctly added to the RPATH in the build tree by the link command:
>
> $ ./bar
> Hello, world!
>
> $ otool -L ./bar
> ./bar:
> @rpath/libfoo.dylib (compatibility version 1.0.0, current version
> 0.0.0)
> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current
> version 0.0.0)
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 400.9.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1252.0.0)
>
> $ otool -l ./bar | grep -A2 LC_RPATH
> cmd LC_RPATH
> cmdsize 56
> path /Users/smancill/src/conda-test/build-osx-system (offset 12)
> --
> cmd LC_RPATH
> cmdsize 80
> path
> /Users/smancill/.local/share/miniconda3/envs/test-system/lib (offset 12)
>
> If I install the binary, it fails because I haven't configured CMake to set
> the install RPATH:
>
> $ make install
> ...
> Install the project...
> -- Install configuration: ""
> -- Installing:
> /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libfoo.dylib
> -- Installing:
> /Users/smancill/.local/share/miniconda3/envs/test-system/include/foo.hpp
> -- Installing:
> /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar
>
> $ bar
> dyld: Library not loaded: @rpath/libfoo.dylib
> Referenced from:
> /Users/smancill/.local/share/miniconda4/envs/test-system/bin/bar
> Reason: image not found
> [1] 84611 abort bar
>
> $ otool -L $CONDA_PREFIX/bin/bar
> /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar:
> @rpath/libfoo.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current
> version 0.0.0)
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 400.9.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1252.0.0)
>
> $ otool -l $CONDA_PREFIX/bin/bar | grep -A2 LC_RPATH
> # empty
>
> # deactivate the environment to start again
> $ conda deactivate
>
> The same can be observed on Linux. Everything works as it should.
>
> --------------------------------------------------
>
> If I try to use Anaconda compilers on macOS, the build RPATH is not set
> properly anymore:
>
> # create the environment and install the Anaconda compiler for macOS,
> and XercesC
> $ conda create -n test-conda clangxx_osx-64 xerces-c
> ...
> environment location:
> /Users/smancill/.local/share/miniconda3/envs/test-conda
> ...
> The following NEW packages will be INSTALLED:
>
> cctools: 895-h7512d6f_0
> clang: 4.0.1-h662ec87_0
> clang_osx-64: 4.0.1-h1ce6c1d_11
> clangxx: 4.0.1-hc9b4283_0
> clangxx_osx-64: 4.0.1-h22b1bf0_11
> compiler-rt: 4.0.1-h5487866_0
> icu: 58.2-h4b95b61_1
> ld64: 274.2-h7c2db76_0
> libcxx: 4.0.1-h579ed51_0
> libcxxabi: 4.0.1-hebd6815_0
> llvm: 4.0.1-hc748206_0
> llvm-lto-tapi: 4.0.1-h6701bc3_0
> xerces-c: 3.2.1-h44e365a_0
> ...
>
> # activate the environment (which sets the variables to use the Anaconda
> compiler)
> $ conda activate test-conda
>
> $ mkdir build-osx-conda
> $ cd build-osx-conda
> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX
> -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
> -- The CXX compiler identification is Clang 4.0.1
> -- Check for working CXX compiler:
> /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++
> -- Check for working CXX compiler:
> /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++
> -- works
> ...
> -- Found XercesC:
> /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib
> (found version "3.2.1")
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /Users/smancill/src/conda-test/build-osx-conda
>
> $ make -j1 VERBOSE=1
> ...
> [100%] Linking CXX executable bar
> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt
> --verbose=1
>
> /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++
> -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE
> -fstack-protector-strong -O2 -pipe -stdlib=libc++
> -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -isysroot
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> -mmacosx-version-min=10.12 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names -Wl,-pie -Wl,-headerpad_max_install_names
> -Wl,-dead_strip_dylibs CMakeFiles/bar.dir/bar.cpp.o -o bar
> -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-conda libfoo.dylib
> /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib
> ...
>
> You can see that the environment lib path is not added to the RPATH by the
> link command,
> which in turns result in the executable not running from the build tree
> anymore:
>
> $ ./bar
> dyld: Library not loaded: @rpath/libxerces-c-3.2.dylib
> Referenced from: /Users/smancill/src/conda-test/build-osx-conda/./bar
> Reason: image not found
> [1] 89350 abort ./bar
>
> $ otool -L ./bar
> ./bar:
> @rpath/libfoo.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current
> version 0.0.0)
> @rpath/libc++.1.dylib (compatibility version 1.0.0, current version
> 1.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1252.0.0)
>
> $ otool -l ./bar | grep -A2 LC_RPATH
> cmd LC_RPATH
> cmdsize 56
> path /Users/smancill/src/conda-test/build-osx-conda (offset 12)
>
> # deactivate the environment
> $ conda deactivate
>
> --------------------------------------------------
>
> If I try the Anaconda compilers on Linux, there are strange results too:
>
> # create the environment and install the Anaconda compiler for Linux,
> and XercesC
> $ conda create -n test-conda gxx_linux-64 xerces-c
> ...
> environment location: /home/vagrant/miniconda3/envs/test-conda
> ...
> The following NEW packages will be INSTALLED:
>
> binutils_impl_linux-64: 2.28.1-had2808c_3
> binutils_linux-64: 7.2.0-had2808c_27
> gcc_impl_linux-64: 7.2.0-habb00fd_3
> gcc_linux-64: 7.2.0-h550dcbe_27
> gxx_impl_linux-64: 7.2.0-hdf63c60_3
> gxx_linux-64: 7.2.0-h550dcbe_27
> icu: 58.2-h9c2bf20_1
> libgcc-ng: 7.2.0-hdf63c60_3
> libstdcxx-ng: 7.2.0-hdf63c60_3
> xerces-c: 3.2.1-hac72e42_0
>
> # activate the environment (which sets the variables to use the Anaconda
> compiler)
> $ conda activate test-conda
>
> $ mkdir build-linux-conda
> $ cd build-linux-conda
> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX
> -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
> -- The CXX compiler identification is GNU 7.2.0
> -- Check for working CXX compiler:
> /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++
> -- Check for working CXX compiler:
> /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++
> -- works
> ...
> -- Found XercesC:
> /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so (found version
> "3.2.1")
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /vagrant/conda-test/build-linux-conda
>
> $ make -j1 VERBOSE=1
> ...
> [100%] Linking CXX executable bar
> /usr/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt
> --verbose=1
>
> /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++
> -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona
> -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> -pipe -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now
> CMakeFiles/bar.dir/bar.cpp.o -o bar libfoo.so
> /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so
> -Wl,-rpath,/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:
> ...
>
> You can see that the environment lib path is added to the RPATH by the link
> command (unlike macOS):
>
> $ ./bar
> Hello World!
>
> But when inspecting the dependencies, the environment lib path appears
> twice:
>
> $ readelf -d ./bar
> ...
> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
> 0x0000000000000001 (NEEDED) Shared library:
> [libxerces-c-3.2.so]
> 0x0000000000000001 (NEEDED) Shared library:
> [libstdc++.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
> 0x000000000000000f (RPATH) Library rpath:
> [/home/vagrant/miniconda3/envs/test-conda/lib:/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:]
> ...
>
> Which is wrong. Now the build tree binary will pick first any old version of
> the foo library installed in the environment instead of the version in the
> build tree.
>
> It seems that the Anaconda toolchain is setting the environment lib path
> into
> the RPATH by its own. It is also set when installing the binaries, even when
> CMake is not configured to set the install RPATH:
>
> $ make install
> ...
> Install the project...
> -- Install configuration: ""
> -- Installing: /home/vagrant/miniconda3/envs/test-conda/lib/libfoo.so
> -- Installing: /home/vagrant/miniconda3/envs/test-conda/include/foo.hpp
> -- Installing: /home/vagrant/miniconda3/envs/test-conda/bin/bar
> -- Set runtime path of
> "/home/vagrant/miniconda3/envs/test-conda/bin/bar" to ""
>
> $ bar
> Hello World!
>
> $ readelf -d $CONDA_PREFIX/bin/bar
> ...
> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
> 0x0000000000000001 (NEEDED) Shared library:
> [libxerces-c-3.2.so]
> 0x0000000000000001 (NEEDED) Shared library:
> [libstdc++.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
> 0x000000000000000f (RPATH) Library rpath:
> [/home/vagrant/miniconda3/envs/test-conda/lib]
> ...
>
> # deactivate the environment
> $ conda deactivate
>
> --------------------------------------------------
>
> TL;DR I cannot get CMake and the Anaconda compilers and packages working
> correctly.
>
> - On macOS, the Conda environment library path is not added to the build
> RPATH.
> - On Linux, the Conda environment library path is always added to the RPATH
> (in both build and install) independently of CMake.
>
> Any advice or workarounds?
>
> --
> Sebastian Mancilla
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
More information about the CMake
mailing list