[CMake] Issues trying to use the Anaconda compiler tools with CMake
Ray Donnelly
mingw.android at gmail.com
Tue Aug 14 02:43:15 EDT 2018
Why are you not using conda-build here? Anaconda Distro and
conda-forge build countless CMake projects. It handles so much extra
stuff on top of building (DSO verification, rpath rewriting, more). If
your end goal is not conda packages then you can untar most sets of
conda-packages to make working software installations (caveat for most
being that some software requires what we call 'prefix replacament'
and those aren't trivially relocatable).
IMHO CMake's handling of RPATHs is broken, as per my "determine
implicit linker directories" PR which there seems to be little to no
interest in instead preferring toolchain cmake files for something
that *clearly* should be queried from the link as it is imperative
when building software that CMake and the toolchain agree on things as
fundamental as this.
RPATH is never set by a toolchain except through being passed to it
(by CMake's logic, fixed by
https://gitlab.kitware.com/cmake/cmake/merge_requests/207).
I would again request that this pretty trivial, obvious and important
PR gets some consideration. Without it CMake asks anyone trying to
build a software distro with more than simple system prefix-based
compilation will run into needing this fix or needing to provide a
toolchain just to tell CMake what it should figure out for itself.
On Tue, Aug 14, 2018 at 2:12 AM, Sebastián Mancilla <smancill at jlab.org> wrote:
> Thanks for the links.
>
> The problem of setting CMAKE_BUILD_RPATH to $CONDA_PREFIX/lib is that
> for binaries and libraries in the build tree, the list of RPATH
> locations will be in the wrong order: the dependencies library path
> will be first, and then the build tree. This will break the unit tests
> if the library was installed previously (for example, to check the
> library by running other installed programs), and then modified as
> part of normal development, because the installed version will be
> loaded instead of the version in the build tree. CMake supports this
> use case out of the box and it works great (when not using Anaconda).
>
> Note that in Linux the $CONDA_PREFIX/lib seems to be added to the
> RPATH by the Anaconda toolchain itself, and it will appear first, so
> there is nothing that can be done about that with CMake.
>
> It seems that the workarounds when using Anaconda compilers, and doing
> development, are:
>
> - Always install before unit testing changes to the library being
> developed (and set CMAKE_BUILD_RPATH).
> - Never install the library (so the build tree not being first in the
> RPATH is not an issue, and set CMAKE_BUILD_RPATH), create a different
> environment to test installation.
> - Don't use the build RPATH and set the proper environment variables
> ([DY]LD_LIBRARY_PATH) with the right path order when running from the
> build tree.
> - Manipulate the RPATH manually (with patchelf or similar).
> - Don't use Anaconda compilers (will there be binary issues with
> Anaconda dependencies?)
>
> But they are not ideal.
>
>
> If the Conda environment is not being used for development and someone
> wants to just build and use some binary/library, setting
> CMAKE_INSTALL_RPATH to $CONDA_PREFIX/lib and then installing will work
> fine. And it looks that is not even necessary on Linux, because the
> RPATH is already set by the toolchain.
>
> El lun., 13 de ago. de 2018 a la(s) 13:35, Isaiah Norton
> (isaiah.norton at gmail.com) escribió:
>>
>> See also:
>>
>> https://gitlab.kitware.com/cmake/cmake/merge_requests/207
>>
>>
>> https://gitlab.kitware.com/cmake/cmake/issues/17483 (looks semi-related)
>>
>> Ray Donnelly, who filed those issues, maintains the Anaconda compiler toolchain -- so you are in good company hitting this issue. It looks like some conda recipes explicitly define INSTALL_RPATH, likely for this reason:
>>
>> https://github.com/AnacondaRecipes/vtk-feedstock/blob/791a1db9026fa2c81d171c38835b512adf221794/recipe/build.sh#L30
>>
>>
>>
>> On Mon, Aug 13, 2018 at 11:54 AM Isaiah Norton <isaiah.norton at gmail.com> wrote:
>>>
>>> The RPATH is not included when using Anaconda because CMake considers the Anaconda lib path to be an implicit link directory (as reported by the compiler). Look at `$build_dir/CMakeFiles/CMakeOutput.log` in the section about "implicit link info".
>>>
>>> The simplest work-around is to set BUILD_RPATH manually, and possibly also INSTALL_RPATH, depending on what conda-build's fixup routine expects.
>>>
>>> For example, adding this line near the top of your example CMakeList fixed the resulting binary, at least in the build directory:
>>>
>>> set(CMAKE_BUILD_RPATH "$ENV{CONDA_PREFIX}/lib")
>>>
>>>
>>> On Fri, Aug 10, 2018 at 5:55 PM Sebastián Mancilla <smancill at jlab.org> wrote:
>>>>
>>>> I am trying to use Conda as a package manager for isolated C++ development
>>>> environments. But unfortunately, when using CMake with the Anaconda-provided
>>>> compilers [1] (which are used to compile the binary packages in the Anaconda
>>>> repositories), things do not work as expected.
>>>>
>>>> I have a small test case available here [2], with an executable calling a
>>>> shared library and a third-party dependency installed with Conda.
>>>>
>>>> [1]: https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html
>>>> [2]: https://gist.github.com/smancill/b28ca07ac11fdf285b4d559545a1630b
>>>>
>>>> --------------------------------------------------
>>>>
>>>> First, when using the system compiler, all works fine (but I am not sure of the
>>>> binary compatibility with the Conda packages, that's why I want to use the
>>>> Anaconda compilers):
>>>>
>>>> # create the environment and install XercesC
>>>> $ conda create -n test-system xerces-c
>>>> ...
>>>> environment location: /Users/smancill/.local/share/miniconda3/envs/test-system
>>>> ...
>>>> The following NEW packages will be INSTALLED:
>>>>
>>>> icu: 58.2-h4b95b61_1
>>>> libcxx: 4.0.1-h579ed51_0
>>>> libcxxabi: 4.0.1-hebd6815_0
>>>> xerces-c: 3.2.1-h44e365a_0
>>>> ...
>>>>
>>>> # activate the environment
>>>> $ conda activate test-system
>>>>
>>>> $ mkdir build-osx-system
>>>> $ cd build-osx-system
>>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>>> -- The CXX compiler identification is AppleClang 9.0.0.9000039
>>>> -- Check for working CXX compiler: /usr/bin/c++
>>>> -- Check for working CXX compiler: /usr/bin/c++ -- works
>>>> ...
>>>> -- Found XercesC: /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib (found version "3.2.1")
>>>> -- Configuring done
>>>> -- Generating done
>>>> -- Build files have been written to: /Users/smancill/src/conda-test/build-osx-system
>>>>
>>>> $ make -j1 VERBOSE=1
>>>> ...
>>>> [100%] Linking CXX executable bar
>>>> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>>> /usr/bin/c++ -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.12 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/bar.dir/bar.cpp.o -o bar -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-system -Wl,-rpath,/Users/smancill/.local/share/miniconda3/envs/test-system/lib libfoo.dylib /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libxerces-c.dylib
>>>> ...
>>>>
>>>> The build directory (~/src/conda-test/build-osx-system) and the conda
>>>> environment lib directory (~/.local/share/miniconda3/envs/test-system/lib)
>>>> are correctly added to the RPATH in the build tree by the link command:
>>>>
>>>> $ ./bar
>>>> Hello, world!
>>>>
>>>> $ otool -L ./bar
>>>> ./bar:
>>>> @rpath/libfoo.dylib (compatibility version 1.0.0, current version 0.0.0)
>>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>>
>>>> $ otool -l ./bar | grep -A2 LC_RPATH
>>>> cmd LC_RPATH
>>>> cmdsize 56
>>>> path /Users/smancill/src/conda-test/build-osx-system (offset 12)
>>>> --
>>>> cmd LC_RPATH
>>>> cmdsize 80
>>>> path /Users/smancill/.local/share/miniconda3/envs/test-system/lib (offset 12)
>>>>
>>>> If I install the binary, it fails because I haven't configured CMake to set the install RPATH:
>>>>
>>>> $ make install
>>>> ...
>>>> Install the project...
>>>> -- Install configuration: ""
>>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/lib/libfoo.dylib
>>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/include/foo.hpp
>>>> -- Installing: /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar
>>>>
>>>> $ bar
>>>> dyld: Library not loaded: @rpath/libfoo.dylib
>>>> Referenced from: /Users/smancill/.local/share/miniconda4/envs/test-system/bin/bar
>>>> Reason: image not found
>>>> [1] 84611 abort bar
>>>>
>>>> $ otool -L $CONDA_PREFIX/bin/bar
>>>> /Users/smancill/.local/share/miniconda3/envs/test-system/bin/bar:
>>>> @rpath/libfoo.dylib (compatibility version 0.0.0, current version 0.0.0)
>>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>>
>>>> $ otool -l $CONDA_PREFIX/bin/bar | grep -A2 LC_RPATH
>>>> # empty
>>>>
>>>> # deactivate the environment to start again
>>>> $ conda deactivate
>>>>
>>>> The same can be observed on Linux. Everything works as it should.
>>>>
>>>> --------------------------------------------------
>>>>
>>>> If I try to use Anaconda compilers on macOS, the build RPATH is not set
>>>> properly anymore:
>>>>
>>>> # create the environment and install the Anaconda compiler for macOS, and XercesC
>>>> $ conda create -n test-conda clangxx_osx-64 xerces-c
>>>> ...
>>>> environment location: /Users/smancill/.local/share/miniconda3/envs/test-conda
>>>> ...
>>>> The following NEW packages will be INSTALLED:
>>>>
>>>> cctools: 895-h7512d6f_0
>>>> clang: 4.0.1-h662ec87_0
>>>> clang_osx-64: 4.0.1-h1ce6c1d_11
>>>> clangxx: 4.0.1-hc9b4283_0
>>>> clangxx_osx-64: 4.0.1-h22b1bf0_11
>>>> compiler-rt: 4.0.1-h5487866_0
>>>> icu: 58.2-h4b95b61_1
>>>> ld64: 274.2-h7c2db76_0
>>>> libcxx: 4.0.1-h579ed51_0
>>>> libcxxabi: 4.0.1-hebd6815_0
>>>> llvm: 4.0.1-hc748206_0
>>>> llvm-lto-tapi: 4.0.1-h6701bc3_0
>>>> xerces-c: 3.2.1-h44e365a_0
>>>> ...
>>>>
>>>> # activate the environment (which sets the variables to use the Anaconda compiler)
>>>> $ conda activate test-conda
>>>>
>>>> $ mkdir build-osx-conda
>>>> $ cd build-osx-conda
>>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>>> -- The CXX compiler identification is Clang 4.0.1
>>>> -- Check for working CXX compiler: /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++
>>>> -- Check for working CXX compiler: /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++ -- works
>>>> ...
>>>> -- Found XercesC: /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib (found version "3.2.1")
>>>> -- Configuring done
>>>> -- Generating done
>>>> -- Build files have been written to: /Users/smancill/src/conda-test/build-osx-conda
>>>>
>>>> $ make -j1 VERBOSE=1
>>>> ...
>>>> [100%] Linking CXX executable bar
>>>> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>>> /Users/smancill/.local/share/miniconda3/envs/test-conda/bin/x86_64-apple-darwin13.4.0-clang++ -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.12 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs CMakeFiles/bar.dir/bar.cpp.o -o bar -Wl,-rpath,/Users/smancill/src/conda-test/build-osx-conda libfoo.dylib /Users/smancill/.local/share/miniconda3/envs/test-conda/lib/libxerces-c.dylib
>>>> ...
>>>>
>>>> You can see that the environment lib path is not added to the RPATH by the link command,
>>>> which in turns result in the executable not running from the build tree anymore:
>>>>
>>>> $ ./bar
>>>> dyld: Library not loaded: @rpath/libxerces-c-3.2.dylib
>>>> Referenced from: /Users/smancill/src/conda-test/build-osx-conda/./bar
>>>> Reason: image not found
>>>> [1] 89350 abort ./bar
>>>>
>>>> $ otool -L ./bar
>>>> ./bar:
>>>> @rpath/libfoo.dylib (compatibility version 0.0.0, current version 0.0.0)
>>>> @rpath/libxerces-c-3.2.dylib (compatibility version 0.0.0, current version 0.0.0)
>>>> @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
>>>>
>>>> $ otool -l ./bar | grep -A2 LC_RPATH
>>>> cmd LC_RPATH
>>>> cmdsize 56
>>>> path /Users/smancill/src/conda-test/build-osx-conda (offset 12)
>>>>
>>>> # deactivate the environment
>>>> $ conda deactivate
>>>>
>>>> --------------------------------------------------
>>>>
>>>> If I try the Anaconda compilers on Linux, there are strange results too:
>>>>
>>>> # create the environment and install the Anaconda compiler for Linux, and XercesC
>>>> $ conda create -n test-conda gxx_linux-64 xerces-c
>>>> ...
>>>> environment location: /home/vagrant/miniconda3/envs/test-conda
>>>> ...
>>>> The following NEW packages will be INSTALLED:
>>>>
>>>> binutils_impl_linux-64: 2.28.1-had2808c_3
>>>> binutils_linux-64: 7.2.0-had2808c_27
>>>> gcc_impl_linux-64: 7.2.0-habb00fd_3
>>>> gcc_linux-64: 7.2.0-h550dcbe_27
>>>> gxx_impl_linux-64: 7.2.0-hdf63c60_3
>>>> gxx_linux-64: 7.2.0-h550dcbe_27
>>>> icu: 58.2-h9c2bf20_1
>>>> libgcc-ng: 7.2.0-hdf63c60_3
>>>> libstdcxx-ng: 7.2.0-hdf63c60_3
>>>> xerces-c: 3.2.1-hac72e42_0
>>>>
>>>> # activate the environment (which sets the variables to use the Anaconda compiler)
>>>> $ conda activate test-conda
>>>>
>>>> $ mkdir build-linux-conda
>>>> $ cd build-linux-conda
>>>> $ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
>>>> -- The CXX compiler identification is GNU 7.2.0
>>>> -- Check for working CXX compiler: /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++
>>>> -- Check for working CXX compiler: /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
>>>> ...
>>>> -- Found XercesC: /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so (found version "3.2.1")
>>>> -- Configuring done
>>>> -- Generating done
>>>> -- Build files have been written to: /vagrant/conda-test/build-linux-conda
>>>>
>>>> $ make -j1 VERBOSE=1
>>>> ...
>>>> [100%] Linking CXX executable bar
>>>> /usr/bin/cmake -E cmake_link_script CMakeFiles/bar.dir/link.txt --verbose=1
>>>> /home/vagrant/miniconda3/envs/test-conda/bin/x86_64-conda_cos6-linux-gnu-c++ -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now CMakeFiles/bar.dir/bar.cpp.o -o bar libfoo.so /home/vagrant/miniconda3/envs/test-conda/lib/libxerces-c.so -Wl,-rpath,/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:
>>>> ...
>>>>
>>>> You can see that the environment lib path is added to the RPATH by the link
>>>> command (unlike macOS):
>>>>
>>>> $ ./bar
>>>> Hello World!
>>>>
>>>> But when inspecting the dependencies, the environment lib path appears twice:
>>>>
>>>> $ readelf -d ./bar
>>>> ...
>>>> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libxerces-c-3.2.so]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>>> 0x000000000000000f (RPATH) Library rpath: [/home/vagrant/miniconda3/envs/test-conda/lib:/vagrant/conda-test/build-linux-conda:/home/vagrant/miniconda3/envs/test-conda/lib:]
>>>> ...
>>>>
>>>> Which is wrong. Now the build tree binary will pick first any old version of
>>>> the foo library installed in the environment instead of the version in the
>>>> build tree.
>>>>
>>>> It seems that the Anaconda toolchain is setting the environment lib path into
>>>> the RPATH by its own. It is also set when installing the binaries, even when
>>>> CMake is not configured to set the install RPATH:
>>>>
>>>> $ make install
>>>> ...
>>>> Install the project...
>>>> -- Install configuration: ""
>>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/lib/libfoo.so
>>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/include/foo.hpp
>>>> -- Installing: /home/vagrant/miniconda3/envs/test-conda/bin/bar
>>>> -- Set runtime path of "/home/vagrant/miniconda3/envs/test-conda/bin/bar" to ""
>>>>
>>>> $ bar
>>>> Hello World!
>>>>
>>>> $ readelf -d $CONDA_PREFIX/bin/bar
>>>> ...
>>>> 0x0000000000000001 (NEEDED) Shared library: [libfoo.so]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libxerces-c-3.2.so]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
>>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>>> 0x000000000000000f (RPATH) Library rpath: [/home/vagrant/miniconda3/envs/test-conda/lib]
>>>> ...
>>>>
>>>> # deactivate the environment
>>>> $ conda deactivate
>>>>
>>>> --------------------------------------------------
>>>>
>>>> TL;DR I cannot get CMake and the Anaconda compilers and packages working
>>>> correctly.
>>>>
>>>> - On macOS, the Conda environment library path is not added to the build RPATH.
>>>> - On Linux, the Conda environment library path is always added to the RPATH
>>>> (in both build and install) independently of CMake.
>>>>
>>>> Any advice or workarounds?
>>>>
>>>> --
>>>> Sebastian Mancilla
>>>> --
>>>>
>>>> Powered by www.kitware.com
>>>>
>>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>>>
>>>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>>>
>>>> CMake Support: http://cmake.org/cmake/help/support.html
>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> https://cmake.org/mailman/listinfo/cmake
>
>
>
> --
> Sebastian Mancilla Matta
> CCTVal, UTFSM
> Valparaíso, Chile
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
More information about the CMake
mailing list