[CMake] Issues trying to use the Anaconda compiler tools with CMake
Sebastián Mancilla
smancill at jlab.org
Mon Aug 13 21:12:14 EDT 2018
Thanks for the links.
The problem of setting CMAKE_BUILD_RPATH to $CONDA_PREFIX/lib is that
for binaries and libraries in the build tree, the list of RPATH
locations will be in the wrong order: the dependencies library path
will be first, and then the build tree. This will break the unit tests
if the library was installed previously (for example, to check the
library by running other installed programs), and then modified as
part of normal development, because the installed version will be
loaded instead of the version in the build tree. CMake supports this
use case out of the box and it works great (when not using Anaconda).
Note that in Linux the $CONDA_PREFIX/lib seems to be added to the
RPATH by the Anaconda toolchain itself, and it will appear first, so
there is nothing that can be done about that with CMake.
It seems that the workarounds when using Anaconda compilers, and doing
development, are:
- Always install before unit testing changes to the library being
developed (and set CMAKE_BUILD_RPATH).
- Never install the library (so the build tree not being first in the
RPATH is not an issue, and set CMAKE_BUILD_RPATH), create a different
environment to test installation.
- Don't use the build RPATH and set the proper environment variables
([DY]LD_LIBRARY_PATH) with the right path order when running from the
build tree.
- Manipulate the RPATH manually (with patchelf or similar).
- Don't use Anaconda compilers (will there be binary issues with
Anaconda dependencies?)
But they are not ideal.
If the Conda environment is not being used for development and someone
wants to just build and use some binary/library, setting
CMAKE_INSTALL_RPATH to $CONDA_PREFIX/lib and then installing will work
fine. And it looks that is not even necessary on Linux, because the
RPATH is already set by the toolchain.
El lun., 13 de ago. de 2018 a la(s) 13:35, Isaiah Norton
(isaiah.norton at gmail.com) escribió:
>
> See also:
>
> https://gitlab.kitware.com/cmake/cmake/merge_requests/207
>
>
> https://gitlab.kitware.com/cmake/cmake/issues/17483 (looks semi-related)
>
> Ray Donnelly, who filed those issues, maintains the Anaconda compiler toolchain -- so you are in good company hitting this issue. It looks like some conda recipes explicitly define INSTALL_RPATH, likely for this reason:
>
> https://github.com/AnacondaRecipes/vtk-feedstock/blob/791a1db9026fa2c81d171c38835b512adf221794/recipe/build.sh#L30
>
>
>
> On Mon, Aug 13, 2018 at 11:54 AM Isaiah Norton <isaiah.norton at gmail.com> wrote:
>>
>> The RPATH is not included when using Anaconda because CMake considers the Anaconda lib path to be an implicit link directory (as reported by the compiler). Look at `$build_dir/CMakeFiles/CMakeOutput.log` in the section about "implicit link info".
>>
>> The simplest work-around is to set BUILD_RPATH manually, and possibly also INSTALL_RPATH, depending on what conda-build's fixup routine expects.
>>
>> For example, adding this line near the top of your example CMakeList fixed the resulting binary, at least in the build directory:
>>
>> set(CMAKE_BUILD_RPATH "$ENV{CONDA_PREFIX}/lib")
>>
>>
>> On Fri, Aug 10, 2018 at 5:55 PM Sebastián Mancilla <smancill at jlab.org> wrote:
>>>
>>> I am trying to use Conda as a package manager for isolated C++ development
>>> environments. But unfortunately, when using CMake with the Anaconda-provided
>>> compilers [1] (which are used to compile the binary packages in the Anaconda
>>> repositories), things do not work as expected.
>>>
>>> I have a small test case available here [2], with an executable calling a
>>> shared library and a third-party dependency installed with Conda.
>>>
>>> [1]: https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html
>>> [2]: https://gist.github.com/smancill/b28ca07ac11fdf285b4d559545a1630b
>>>
>>> --------------------------------------------------
>>>
>>> First, when using the system compiler, all works fine (but I am not sure of the
>>> binary compatibility with the Conda packages, that's why I want to use the
>>> Anaconda compilers):
>>>
>>> # create the environment and install XercesC
>>> $ conda create -n test-system xerces-c
>>> ...
>>> environment location: /Users/smancill/.local/share/miniconda3/envs/test-system
>>> ...
>>> The following NEW packages will be INSTALLED:
>>>
>>> icu: 58.2-h4b95b61_1
>>> libcxx: 4.0.1-h579ed51_0
>>> libcxxabi: 4.0.1-hebd6815_0
>>> xerces-c: 3.2.1-h44e365a_0
>>> ...
>>>
>>> # activate the environment
>>> $ conda activate test-system
>>>
>>> $ mkdir build-osx-system
>>> $ cd build-osx-system
>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>> -- The CXX compiler identification is AppleClang 9.0.0.9000039
>>> -- Check for working CXX compiler: /usr/bin/c++
>>> -- Check for working CXX compiler: /usr/bin/c++ -- works
>>> ...
>>> -- Found XercesC: /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib (found version "3.2.1")
>>> -- Configuring done
>>> -- Generating done
>>> -- Build files have been written to: /Users/smancill/src/conda-test/build-osx-system
>>>
>>> $ make -j1 VERBOSE=1
>>> ...
>>> [100%] Linking CXX executable bar
>>> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>> /usr/bin/c++ -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.12 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/bar.dir/bar.cpp.o -o bar -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-system -Wl,-rpath,/Users/smancill/.local/share/miniconda3/envs/test-system/lib libfoo.dylib /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib
>>> ...
>>>
>>> The build directory (~/src/conda-test/build-osx-system) and the conda
>>> environment lib directory (~/.local/share/miniconda3/envs/test-system/lib)
>>> are correctly added to the RPATH in the build tree by the link command:
>>>
>>> $ ./bar
>>> Hello, world!
>>>
>>> $ otool -L ./bar
>>> ./bar:
>>> @rpath/libfoo.dylib (compatibility version 1.0.0, current version 0.0.0)
>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>
>>> $ otool -l ./bar | grep -A2 LC_RPATH
>>> cmd LC_RPATH
>>> cmdsize 56
>>> path /Users/smancill/src/conda-test/build-osx-system (offset 12)
>>> --
>>> cmd LC_RPATH
>>> cmdsize 80
>>> path /Users/smancill/.local/share/miniconda3/envs/test-system/lib (offset 12)
>>>
>>> If I install the binary, it fails because I haven't configured CMake to set the install RPATH:
>>>
>>> $ make install
>>> ...
>>> Install the project...
>>> -- Install configuration: ""
>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libfoo.dylib
>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/include/foo.hpp
>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar
>>>
>>> $ bar
>>> dyld: Library not loaded: @rpath/libfoo.dylib
>>> Referenced from: /Users/smancill/.local/share/miniconda4/envs/test-system/bin/bar
>>> Reason: image not found
>>> [1] 84611 abort bar
>>>
>>> $ otool -L $CONDA_PREFIX/bin/bar
>>> /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar:
>>> @rpath/libfoo.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>
>>> $ otool -l $CONDA_PREFIX/bin/bar | grep -A2 LC_RPATH
>>> # empty
>>>
>>> # deactivate the environment to start again
>>> $ conda deactivate
>>>
>>> The same can be observed on Linux. Everything works as it should.
>>>
>>> --------------------------------------------------
>>>
>>> If I try to use Anaconda compilers on macOS, the build RPATH is not set
>>> properly anymore:
>>>
>>> # create the environment and install the Anaconda compiler for macOS, and XercesC
>>> $ conda create -n test-conda clangxx_osx-64 xerces-c
>>> ...
>>> environment location: /Users/smancill/.local/share/miniconda3/envs/test-conda
>>> ...
>>> The following NEW packages will be INSTALLED:
>>>
>>> cctools: 895-h7512d6f_0
>>> clang: 4.0.1-h662ec87_0
>>> clang_osx-64: 4.0.1-h1ce6c1d_11
>>> clangxx: 4.0.1-hc9b4283_0
>>> clangxx_osx-64: 4.0.1-h22b1bf0_11
>>> compiler-rt: 4.0.1-h5487866_0
>>> icu: 58.2-h4b95b61_1
>>> ld64: 274.2-h7c2db76_0
>>> libcxx: 4.0.1-h579ed51_0
>>> libcxxabi: 4.0.1-hebd6815_0
>>> llvm: 4.0.1-hc748206_0
>>> llvm-lto-tapi: 4.0.1-h6701bc3_0
>>> xerces-c: 3.2.1-h44e365a_0
>>> ...
>>>
>>> # activate the environment (which sets the variables to use the Anaconda compiler)
>>> $ conda activate test-conda
>>>
>>> $ mkdir build-osx-conda
>>> $ cd build-osx-conda
>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>> -- The CXX compiler identification is Clang 4.0.1
>>> -- Check for working CXX compiler: /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++
>>> -- Check for working CXX compiler: /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++ -- works
>>> ...
>>> -- Found XercesC: /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib (found version "3.2.1")
>>> -- Configuring done
>>> -- Generating done
>>> -- Build files have been written to: /Users/smancill/src/conda-test/build-osx-conda
>>>
>>> $ make -j1 VERBOSE=1
>>> ...
>>> [100%] Linking CXX executable bar
>>> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>> /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++ -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.12 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs CMakeFiles/bar.dir/bar.cpp.o -o bar -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-conda libfoo.dylib /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib
>>> ...
>>>
>>> You can see that the environment lib path is not added to the RPATH by the link command,
>>> which in turns result in the executable not running from the build tree anymore:
>>>
>>> $ ./bar
>>> dyld: Library not loaded: @rpath/libxerces-c-3.2.dylib
>>> Referenced from: /Users/smancill/src/conda-test/build-osx-conda/./bar
>>> Reason: image not found
>>> [1] 89350 abort ./bar
>>>
>>> $ otool -L ./bar
>>> ./bar:
>>> @rpath/libfoo.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>
>>> $ otool -l ./bar | grep -A2 LC_RPATH
>>> cmd LC_RPATH
>>> cmdsize 56
>>> path /Users/smancill/src/conda-test/build-osx-conda (offset 12)
>>>
>>> # deactivate the environment
>>> $ conda deactivate
>>>
>>> --------------------------------------------------
>>>
>>> If I try the Anaconda compilers on Linux, there are strange results too:
>>>
>>> # create the environment and install the Anaconda compiler for Linux, and XercesC
>>> $ conda create -n test-conda gxx_linux-64 xerces-c
>>> ...
>>> environment location: /home/vagrant/miniconda3/envs/test-conda
>>> ...
>>> The following NEW packages will be INSTALLED:
>>>
>>> binutils_impl_linux-64: 2.28.1-had2808c_3
>>> binutils_linux-64: 7.2.0-had2808c_27
>>> gcc_impl_linux-64: 7.2.0-habb00fd_3
>>> gcc_linux-64: 7.2.0-h550dcbe_27
>>> gxx_impl_linux-64: 7.2.0-hdf63c60_3
>>> gxx_linux-64: 7.2.0-h550dcbe_27
>>> icu: 58.2-h9c2bf20_1
>>> libgcc-ng: 7.2.0-hdf63c60_3
>>> libstdcxx-ng: 7.2.0-hdf63c60_3
>>> xerces-c: 3.2.1-hac72e42_0
>>>
>>> # activate the environment (which sets the variables to use the Anaconda compiler)
>>> $ conda activate test-conda
>>>
>>> $ mkdir build-linux-conda
>>> $ cd build-linux-conda
>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>> -- The CXX compiler identification is GNU 7.2.0
>>> -- Check for working CXX compiler: /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++
>>> -- Check for working CXX compiler: /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
>>> ...
>>> -- Found XercesC: /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so (found version "3.2.1")
>>> -- Configuring done
>>> -- Generating done
>>> -- Build files have been written to: /vagrant/conda-test/build-linux-conda
>>>
>>> $ make -j1 VERBOSE=1
>>> ...
>>> [100%] Linking CXX executable bar
>>> /usr/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>> /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++ -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now CMakeFiles/bar.dir/bar.cpp.o -o bar libfoo.so /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so -Wl,-rpath,/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:
>>> ...
>>>
>>> You can see that the environment lib path is added to the RPATH by the link
>>> command (unlike macOS):
>>>
>>> $ ./bar
>>> Hello World!
>>>
>>> But when inspecting the dependencies, the environment lib path appears twice:
>>>
>>> $ readelf -d ./bar
>>> ...
>>> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
>>> 0x0000000000000001 (NEEDED) Shared library: [libxerces-c-3.2.so]
>>> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
>>> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>> 0x000000000000000f (RPATH) Library rpath: [/home/vagrant/miniconda3/envs/test-conda/lib:/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:]
>>> ...
>>>
>>> Which is wrong. Now the build tree binary will pick first any old version of
>>> the foo library installed in the environment instead of the version in the
>>> build tree.
>>>
>>> It seems that the Anaconda toolchain is setting the environment lib path into
>>> the RPATH by its own. It is also set when installing the binaries, even when
>>> CMake is not configured to set the install RPATH:
>>>
>>> $ make install
>>> ...
>>> Install the project...
>>> -- Install configuration: ""
>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/lib/libfoo.so
>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/include/foo.hpp
>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/bin/bar
>>> -- Set runtime path of "/home/vagrant/miniconda3/envs/test-conda/bin/bar" to ""
>>>
>>> $ bar
>>> Hello World!
>>>
>>> $ readelf -d $CONDA_PREFIX/bin/bar
>>> ...
>>> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
>>> 0x0000000000000001 (NEEDED) Shared library: [libxerces-c-3.2.so]
>>> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
>>> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>> 0x000000000000000f (RPATH) Library rpath: [/home/vagrant/miniconda3/envs/test-conda/lib]
>>> ...
>>>
>>> # deactivate the environment
>>> $ conda deactivate
>>>
>>> --------------------------------------------------
>>>
>>> TL;DR I cannot get CMake and the Anaconda compilers and packages working
>>> correctly.
>>>
>>> - On macOS, the Conda environment library path is not added to the build RPATH.
>>> - On Linux, the Conda environment library path is always added to the RPATH
>>> (in both build and install) independently of CMake.
>>>
>>> Any advice or workarounds?
>>>
>>> --
>>> Sebastian Mancilla
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>>
>>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>>
>>> CMake Support: http://cmake.org/cmake/help/support.html
>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> https://cmake.org/mailman/listinfo/cmake
--
Sebastian Mancilla Matta
CCTVal, UTFSM
Valparaíso, Chile
More information about the CMake
mailing list