[CMake] Multiple occurrences of a library on linux (ldd)

[Thiago Crepaldi]

I might be getting close and the root cause might be related to a
conceptual question that I was saving for later: how transitive
linking should be done!

Returning to the build design: pytorch is compile if not installed in
the system, lib datareader_core consumes pytorch and standalone_gtests
consumes datareader_core
          (aka standalone_gtests --> lib_datareader.so --> libc10.so)

Here are the relevant snippets from all relevant cmake files:

***/CMakeLists.txt***
(...)
# pytorch will be compile here
add_subdirectory(external/upstream)

# datareader_core lib will consume torch during its build
ExternalProject_Add(${PROJECT_NAME}_core
   DEPENDS pytorch_external
   SOURCE_DIR (...)
   CMAKE_ARGS (...)
   CMAKE_CACHE_ARGS (...)
   BUILD_ALWAYS 1
)

# Consumes datareader_core, which consumes pytorch (libc10.so and libcaffe2.so)
ExternalProject_Add(${PROJECT_NAME}_core_test
   DEPENDS ${PROJECT_NAME}_core
   SOURCE_DIR (...)
   CMAKE_ARGS (...)
   CMAKE_CACHE_ARGS -DDATAREADER_CORE_ROOT:INTERNAL=${DATAREADER_CORE_ROOT}

-DDATAREADER_CORE_LIBRARYDIR:INTERNAL=${DATAREADER_CORE_LIBRARYDIR}

-DDATAREADER_CORE_INCLUDEDIR:INTERNAL=${DATAREADER_CORE_INCLUDEDIR}
   (...)
   BUILD_ALWAYS 1
)

***/external/upstream/CMakeLists.txt***
add_subdirectory(pytorch)

/external/upstream/CMakeLists.txt
find_package(Torch QUIET)
if(NOT Torch_FOUND)
   message(STATUS "Pytorch not found in the systemm. Looking in the
build environment")
   set(
      CMAKE_PREFIX_PATH "${EP_BASE}/Source/pytorch_external/torch/share/cmake"
      CACHE INTERNAL "Path to Pytorch cmake configs"
      FORCE
   )
   find_package(Torch QUIET)
endif()

if(Torch_FOUND)
   add_library(pytorch_external INTERFACE)
else()
   message(STATUS "Pytorch could not be located. Building instead")

   include(ExternalProject)
   ExternalProject_Add(pytorch_external (...)
endif()

***/datareader_src/CMakeLists.txt***
project(datareader_core)

   add_library(datareader SHARED ${DATAREADER_SRC})
   find_package(Torch REQUIRED)
   target_include_directories(datareader PUBLIC
${CMAKE_CURRENT_SOURCE_DIR}/include)
   target_link_libraries(datareader PUBLIC ${TORCH_LIBRARIES})

***/datareader_test_src/CMakeLists.txt***
project(datareader_core_test)

enable_testing()
add_executable(standalone_gtests test.cpp)
find_package(GTest REQUIRED)
find_package(Threads REQUIRED)
target_include_directories(standalone_gtests PUBLIC
${DATAREADER_CORE_INCLUDEDIR}
${GOOGLETEST_INCLUDEDIR}
${CMAKE_CURRENT_SOURCE_DIR}/../)

target_link_libraries(standalone_gtests PUBLIC
${TORCH_LIBRARIES}
${DATAREADER}
${GTEST_BOTH_LIBRARIES}
${CMAKE_THREAD_LIBS_INIT}
)

At this point, Pytorch is looked in the system by find_package(Torch),
if not found, it will be cloned and compiled. datareader_core lib
find_package(Torch) and link to it. Lastly, datareader_core_test app
will link to datareader_core lib.
If I ldd libdatareader.so, I can see libc10.so (from pytorch) is
properly linked. However, datareader_core_test compilation will fail
with an undefined reference to symbol inside libc10.so:
/usr/bin/ld: warning: libcaffe2.so, needed by
/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so,
not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libc10.so, needed by
/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so,
not found (try using -rpath or -rpath-link)
/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so:
undefined reference to `c10::Error::Error(c10::SourceLocation,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&)'
collect2: error: ld returned 1 exit status

My understanding was that if datareader_core did
target_link_libraries(datareader PUBLIC ${TORCH_LIBRARIES}), then any
other target that depends on datareader target would be properly
linked to its public dependences libraries.
Where is my mistake? To work this around, I have added
find_package(Torch REQUIRED) and
target_link_libraries(standalone_gtests PUBLIC ${TORCH_LIBRARIES}) to
standalone_gtests, although it doesn't directly consume anything from
pytorch. That change lead to the original issue of having
standalone_gtests having two entries of libc10.so in the ldd output:
one because of the indirect lib datareader dependency (not fond) and
another due tot he direct linking (working)

Am I linking Torch incorrectly to datareader_core or maybe
ExternalProject_Add(datareader_core) hides linking information from
another target inside another
ExternalProject_Add(datareader_core_test)?
I am a bit confused by the explanation at
https://cmake.org/cmake/help/latest/command/target_link_libraries.html
and this build and your help might help solidify the private, public
and interface concepts.

Thanks again
Thiago
On Thu, Feb 14, 2019 at 7:42 AM Thiago Crepaldi <dev at thiagocrepaldi.com> wrote:
>
> Thanks, Thompson, I will look into BUILD_RPATH and possibly INSTALL_RPATH.
>
> I just learned about `export LD_DEBUG=files` to debug linking issues
> on linux. It provides more detail on the ldd output, as below:
>
> 18843: file=libc10.so [0];  needed by
> /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so
> [0]
>                (no linking information)
> 18843: file=libdatareader.so [0];  needed by
> subprojects/Build/datareaders_core_test/standalone_gtests [0]
> 18843: file=libdatareader.so [0];  generating link map
>                 (linking information)
> (...)
> 18843: file=libc10.so [0];  needed by
> /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so
> [0]
> 18843: file=libc10.so [0];  generating link map
>                (linking information)
> 18843: file=libcaffe2.so [0];  needed by
> subprojects/Build/datareaders_core_test/standalone_gtests [0]
> 18843: file=libcaffe2.so [0];  generating link map
>                (linking information)
>
> Now I can see that the missing libc10.so is needed by
> `libdatareader.so`, which was linked against `standalone_gtests`.
>
> However, RUNPATH for both `libdatareader.so` and `standalone_gtests`
> seems to be correct and point to the same torch/lib folder that has
> libc10.so:
> libdatareader.so:
> RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib:
> standalone_gtests:
> RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib:/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib
>
> So, libdatareader.so links to libcaffe2.so which links to libc10.so
> successfully; standalone_gtests links to libcaffe2.so which links to
> libc10.so successully too; however, when standalone_gtests links to
> libdatareader.so, there is a transitive linking issue with
> libdatareader.so dependency on libc10.so, like the diagram below:
>
> libdatareader.so ---> libcaffe2.so ---> libc10.so (ok)
>
> standalone_gtests 0 ---> libcaffe2.so ---> libc10.so (ok)
>               |
>               -------------------> libdatareader.so ---> libc10.so (not found)
>
>
> Thanks again,
> Thiago
>
>
>
> On Thu, Feb 14, 2019 at 6:13 AM Thompson, KT <kgt at lanl.gov> wrote:
> >
> > Thiago,
> >
> >
> >
> > I haven’t see the double entry pattern that you mention below.  However, you might want to tell CMake to embed a BUILD_RPATH in your libraries.  This should get around the issue of manually setting LD_LIBRARY_PATH.
> >
> >
> >
> > https://cmake.org/cmake/help/latest/prop_tgt/BUILD_RPATH.html?highlight=rpath
> >
> >
> >
> > -kt
> >
> >
> >
> > From: CMake <cmake-bounces at cmake.org> On Behalf Of Thiago Crepaldi
> > Sent: Thursday, February 14, 2019 12:43 AM
> > To: cmake at cmake.org
> > Subject: [CMake] Multiple occurrences of a library on linux (ldd)
> >
> >
> >
> > Hello all,
> >
> >
> >
> > After reading CMake Cookbook I have written my first "complex" CMake build script based on the superbuild pattern. I am excited to heave a better understanding on CMake, but I definitely will learn much more from experience and your kind help.
> >
> > In summary, the standalone google test application `standalone_gtests` publicly links to `libdatareader.so` and to pytorch libraries (`libc10.so`,`libcafee2.so`, `libtorch.so`).
> >
> > `libdatareader.so` also publicly links to pytorch libraries (I have a theoretical question on why `standalone_gtests` had to link to pytorch libraries if `libdatareader.so` already did, but that can wait).
> >
> >
> >
> > Compilation finishes successfully, but when I try to run `standalone_gtests`, it aborts because it cant find `libc10.so`.
> >
> > After executing `ldd standalone_gtests`, the weird result was that there were two entries for `libc10.so`.
> >
> > The first one maps to "not found" while the second had the correct path to the library. `libcaffe2.so`, which is also a pytorch library, has a single occurrence with full path.
> >
> > If I add the (...)/pytorch_external/(...) (see ldd output below) path to LD_LIBRARY_PATH, then everything works, but I would like to avoid this, if possible.
> >
> >
> >
> > `ldd ./subprojects/Build/datareaders_core_test/standalone_gtests
> >
> > libdatareader.so => /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so
> >
> > libcaffe2.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so
> >
> > libc10.so => not found
> >
> > libc10.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libc10.so
> >
> > (...)`
> >
> >
> >
> > Have anyone seen multiple entries for the same library on ldd before? Why is that? Is it because `standalone_gtests` links to libc10.so and to `libdatareader.so`, which in turn also links to `libc10.so`?
> >
> > Both CMakeLists.txt (libdatareader and standalone_gtests) succeeds at find_package(Torch REQUIRED QUIET) commands (${TORCH_LIBRARY} returns the correct path).
> >
> >
> >
> >
> >
> > I run the same build on my Mac and everything works fine, so that is confined to linux environment. I have destroyed my conda environment and performed multiple clean builds in the process and no luck :(
> >
> > Hoping this was some sort of ldconfig issue, I tried `sudo ldconfig` and `sudo rm /etc/ld.so.cache`, but that doesn't fix it.
> >
> >
> >
> > Any ideas?
> >
> >
> >
> > best regards,
> >
> > Thiago
>
>
>
> --
> Thiago



-- 
Thiago