[CMake] How can I control / debug linking arguments (Separable Cross-compile of CUDA 6.5 for ARMv7 linked with -std=c++11)?

[Gabriel Archacki Hare]

Since I figured out how to get the separable cross-compile to a library working I thought I would share.

To answer the “How can I use X to do Y” question (http://www.catb.org/~esr/faqs/smart-questions.html#idp42280576):
The FindCUDA package defines the CUDA_LIBRARIES list, which includes the problematic 64bit libraries.

The next step is to read through the FindCUDA source code. (The WRONG place to look is the GitHub repo for FindCUDA… the right place is the FindCUDA module that is installed.) In the source, it turns out that 2 variables are set that are not documented (https://cmake.org/cmake/help/v3.4/module/FindCUDA.html) whose state can be printed out. These are:
CUDA_rt_LIBRARY
CUDA_dl_LIBRARY
These variables in turn are set using find_library (https://cmake.org/cmake/help/v3.3/command/find_library.html) whose behavior is determined by variables including:
CMAKE_LIBRARY_ARCHITECTURE
CMAKE_PREFIX_PATH
The problem I had encountered was due to having CMAKE_LIBRARY_ARCHITECTURE = x86_64-linux-gnu, and CMAKE_PREFIX_PATH empty.

For my particular setup (Ubuntu 14.04 with cuda-cross-armhf installed) the I needed to change the above settings to have CMAKE_LIBRARY_ARCHITECTURE empty and CMAKE_PREFIX_PATH = /usr/arm-linux-gnueabihf.

P.S. In the course of trying to solve this I stumbled on a thread that seemed to me to be exactly the same issue and was NOT: (http://stackoverflow.com/questions/7476625/set-global-gcc-default-search-paths). Hopefully anyone reading this will avoid that particular rabbit hole (or identify it as their actual issue).


> On 6 Jan, 2016, at 10:02 AM, Gabriel Archacki Hare <Gabriel.Archacki.Hare at gmail.com> wrote:
> 
> I am trying to set up a cross-compile using FindCUDA to the TegraK1 architecture, which limits me to CUDA 6.5. Specifically, I have a subdirectory TestCUDA that compiles (using separable compilation) to a library libTestCUDA.a, which is then linked with code that is compiled using c++11 (which is not available in CUDA 6.5).
> 
> I can compile this by manually issuing build commands. And, I have set up the scripts so that in both OS X and Ubuntu compiling to the respective systems (not a cross-compile) works. However, when I try to cross compile, in the final linking stage 64 bit libraries are referenced, resulting in failure.
> 
> [100%] Linking CXX executable Test
> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/Test.dir/link.txt --verbose=1
> /usr/bin/arm-linux-gnueabihf-g++   -std=c++11 -O3 -DNDEBUG   CMakeFiles/Test.dir/Test.cxx.o  -o Test /home/ghare/Documents/autowiring-build-tegrak1/lib/libAutowiring.a TestCUDA/libTestCUDA.a /usr/local/cuda-6.5/targets/armv7-linux-gnueabihf/lib/libcudart_static.a -lpthread /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libdl.so 
> /usr/lib/x86_64-linux-gnu/librt.so: file not recognized: File format not recognized
> collect2: error: ld returned 1 exit status
> 
> However, the following command successfully completes the compiling:
> arm-linux-gnueabihf-g++ -mfloat-abi=hard -o Test  CMakeFiles/Test.dir/Test.cxx.o /home/ghare/Documents/autowiring-build-tegrak1/lib/libAutowiring.a TestCUDA/libTestCUDA.a -Xlinker --dynamic-linker=/lib/ld-linux-armhf.so.3 -L/usr/local/cuda-6.5/bin/../targets/armv7-linux-gnueabihf/lib/stubs -L/usr/local/cuda-6.5/bin/../targets/armv7-linux-gnueabihf/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl
> 
> All I need to do to have a complete cross-compile setup is to modify the linking…
> 
> -Gabriel
> 
> 
> P. S. In case it will help, verbose output from the linking state follows:
> Using built-in specs.
> COLLECT_GCC=/usr/bin/arm-linux-gnueabihf-g++
> COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/lto-wrapper
> Target: arm-linux-gnueabihf
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.8.2-16ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/arm-linux-gnueabihf/include/c++/4.8.2 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-armhf-cross --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
> Thread model: posix
> gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) 
> COMPILER_PATH=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/:/usr/lib/gcc-cross/arm-linux-gnueabihf/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/:/usr/lib/gcc-cross/arm-linux-gnueabihf/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/bin/
> LIBRARY_PATH=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/:/lib/:/usr/lib/
> COLLECT_GCC_OPTIONS='-v' '-std=c++11' '-O3' '-D' 'NDEBUG' '-o' 'Test' '-shared-libgcc' '-march=armv7-a' '-mfloat-abi=hard' '-mfpu=vfpv3-d16' '-mthumb' '-mtls-dialect=gnu'
> /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/collect2 --sysroot=/ --build-id --eh-frame-hdr -dynamic-linker /lib/ld-linux-armhf.so.3 -X --hash-style=gnu --as-needed -m armelf_linux_eabi -z relro -o Test /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/../lib/crt1.o /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/../lib/crti.o /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/crtbegin.o -L/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8 -L/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib CMakeFiles/Test.dir/Test.cxx.o /home/ghare/Documents/autowiring-build-tegrak1/lib/libAutowiring.a TestCUDA/libTestCUDA.a /usr/local/cuda-6.5/targets/armv7-linux-gnueabihf/lib/libcudart_static.a -lpthread /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libdl.so -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/crtend.o /usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/../../../../arm-linux-gnueabihf/lib/../lib/crtn.o
> /usr/lib/x86_64-linux-gnu/librt.so: file not recognized: File format not recognized
> collect2: error: ld returned 1 exit status