[CMake] Transitive linking

[Michael Hertling]

On 11/19/2011 08:02 AM, James Bigler wrote:
> On Fri, Nov 18, 2011 at 5:51 PM, Michael Hertling <mhertling at online.de>wrote:
> 
>> On 11/18/2011 10:03 PM, James Bigler wrote:
>>> 2011/11/18 Alexander Neundorf <a.neundorf-work at gmx.net>
>>>
>>>> On Friday 18 November 2011, James Bigler wrote:
>>>>> I thought CMake knew how to not drag all the dependent libraries once
>> you
>>>>> linked an executable module.
>>>>>
>>>>> add_library(A STATIC a.cpp)
>>>>> add_library(B SHARED b.cpp)
>>>>> target_link_libraries(B A)
>>>>> add_library(C SHARED c.cpp)
>>>>> target_link_libraries(C B)
>>>>>
>>>>> add_executable(run run.cpp)
>>>>> target_link_libraries(run C)
>>>>>
>>>>> At this point I'm seeing that C links against B and A when I think it
>>>>> should only link against B since A shouldn't be needed to link against
>> B.
>>>>> In addition when compiling run, it links against B and A.
>>>>>
>>>>> /usr/bin/c++   -dynamiclib -Wl,-headerpad_max_install_names   -o
>>>> libC.dylib
>>>>> -install_name
>> /Users/jbigler/tmp/code/cmake/translinking/build/libC.dylib
>>>>> CMakeFiles/C.dir/c.cpp.o libB.dylib libA.a
>>>>> /usr/bin/c++    -Wl,-search_paths_first
>> -Wl,-headerpad_max_install_names
>>>>> CMakeFiles/run.dir/run.cpp.o  -o run  libC.dylib libB.dylib libA.a
>>>>>
>>>>> Is this the expected behavior?
>>>>
>>>> Yes.
>>>> If you want to limit this, use target_link_libraries( C
>>>> LINK_INTERFACE_LIBRARIES ... ), with this you can specify the
>> transitively
>>>> linked libraries when linking against C.
>>>>
>>>> Alex
>>>>
>>>
>>> OK, so propagating the libraries is the default behavior.  I tried to use
>>> LINK_INTERFACE_LIBRARIES, but it only seemed to work for static
>> libraries:
>>>
>>> add_library(A STATIC a.cpp)
>>> add_library(B SHARED b.cpp)
>>> target_link_libraries(B A)
>>> target_link_libraries(B LINK_INTERFACE_LIBRARIES)
>>> add_library(C SHARED c.cpp)
>>> target_link_libraries(C B)
>>> target_link_libraries(C LINK_INTERFACE_LIBRARIES)
>>>
>>> add_executable(run run.cpp)
>>> target_link_libraries(run C)
>>>
>>> During build:
>>>
>>> Build B (this is fine)
>>> /usr/bin/c++   -dynamiclib -o libB.dylib CMakeFiles/B.dir/b.cpp.o libA.a
>>> Build C (this is fine too, no A in the list)
>>> /usr/bin/c++   -dynamiclib -o libC.dylib CMakeFiles/C.dir/c.cpp.o
>> libB.dylib
>>> Build run (this is weird, it linked both B and C)
>>> /usr/bin/c++    CMakeFiles/run.dir/run.cpp.o  -o run  libC.dylib
>> libB.dylib
>>>
>>> Did I specify something wrong or does this property only consider static
>>> libraries?
>>
>> On *nix, I can't confirm this:
>>
>> CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
>> PROJECT(TRANSLINK CXX)
>> SET(CMAKE_VERBOSE_MAKEFILE ON)
>> FILE(WRITE ${CMAKE_BINARY_DIR}/a.cxx "void a(){}\n")
>> ADD_LIBRARY(A STATIC a.cxx)
>> FILE(WRITE ${CMAKE_BINARY_DIR}/b.cxx "void b(){}\n")
>> ADD_LIBRARY(B SHARED b.cxx)
>> TARGET_LINK_LIBRARIES(B A)
>> TARGET_LINK_LIBRARIES(B LINK_INTERFACE_LIBRARIES)
>> FILE(WRITE ${CMAKE_BINARY_DIR}/c.cxx "void c(){}\n")
>> ADD_LIBRARY(C SHARED c.cxx)
>> TARGET_LINK_LIBRARIES(C B)
>> TARGET_LINK_LIBRARIES(C LINK_INTERFACE_LIBRARIES)
>> FILE(WRITE ${CMAKE_BINARY_DIR}/main.cxx "int main(){}\n")
>> ADD_EXECUTABLE(main main.cxx)
>> TARGET_LINK_LIBRARIES(main C)
>>
>> The main target's link command line reads:
>>
>> .../c++ .../main.cxx.o -o main ... libC.so ...  # No libB.so!
>>
>> Do you actually get different results with the above-noted project?
>>
>> In general, CMake's transitive handling of target dependencies causes
>> no harm since no library is loaded unnecessarily, although there may be
>> libraries which are specified unnecessarily. E.g., main would be linked
>> against libB.so without immediately referring to the latter, i.e. it is
>> formally overlinked, but whether libB.so is loaded on behalf of main or
>> on behalf of libC.so does not matter, and overlinking w.r.t. a static
>> library is not possible. Do you have particular requirements why you
>> want to reduce a target's references to the immediate dependencies?
>>
>> Regards,
>>
>> Michael
>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
>>
> 
> I tried your script, and it seems to exhibit the same behavior as my script:
> 
> /usr/bin/c++    -Wl,-search_paths_first -Wl,-headerpad_max_install_names
> CMakeFiles/main.dir/main.cxx.o  -o main  libC.dylib libB.dylib
> 
> Perhaps this is some Mac specific behavior that isn't showing up on other
> systems.
> 
> The main reason I wanted it was to prevent build failures.  I specified a
> library path (-L) in one subdirectory to correctly link in the set of
> external static libraries.  CMake tried to drag those libraries on to the
> "next" project which didn't have the library path and link failed.  This is
> why I started to look into why those static libraries were linked in the
> first place.  As far as I'm concerned once you link an executable module,
> you don't need its dependencies anymore.  Those dependencies should be
> satisfied within the library itself.  I'm not exactly sure why CMake
> implements this behavior by default, since it seems to go against the
> philosophy of shared library linkage.

Shared libraries can refer to other shared libraries - cf. ELF's DT_
NEEDED tag - and usually, the linker is smart enough to follow these
tags from library to library in order to resolve symbols, even if only
the immediate prerequisites are specified on the link command line. In
this regard, you are right that it would not be strictly necessary to
specify a mediate shared prerequisite, but which locations the linker
searches for libraries can be influenced in various ways, see the ld
manpage for more information. Thus, you are taking the same risks as
with the usage of LINK_DIRECTORIES() that the linker will look into
the wrong directories. It's even possible to mess up things with
project-internal libraries only, i.e. library targets, although
CMake tries very hard to avoid problems of this kind:

CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(TRANSLINK C)
SET(CMAKE_VERBOSE_MAKEFILE ON)
SET(CMAKE_SKIP_BUILD_RPATH TRUE)
FILE(WRITE ${CMAKE_BINARY_DIR}/a1.c "void a1(){}\n")
ADD_LIBRARY(A1 SHARED a1.c)
SET_TARGET_PROPERTIES(A1 PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY X LIBRARY_OUTPUT_NAME A)
FILE(WRITE ${CMAKE_BINARY_DIR}/b1.c "void b1(){a2();}\n")
ADD_LIBRARY(B1 SHARED b1.c)
SET_TARGET_PROPERTIES(B1 PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY X LIBRARY_OUTPUT_NAME B)
TARGET_LINK_LIBRARIES(B1 A2)
FILE(WRITE ${CMAKE_BINARY_DIR}/a2.c "void a2(){}\n")
ADD_LIBRARY(A2 SHARED a2.c)
SET_TARGET_PROPERTIES(A2 PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY Y LIBRARY_OUTPUT_NAME A)
FILE(WRITE ${CMAKE_BINARY_DIR}/b2.c "void b2(){a1();}\n")
ADD_LIBRARY(B2 SHARED b2.c)
SET_TARGET_PROPERTIES(B2 PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY Y LIBRARY_OUTPUT_NAME B)
TARGET_LINK_LIBRARIES(B2 A1)
FILE(WRITE ${CMAKE_BINARY_DIR}/c.c "void c(){b1();}\n")
ADD_LIBRARY(C SHARED c.c)
TARGET_LINK_LIBRARIES(C B1)
TARGET_LINK_LIBRARIES(C LINK_INTERFACE_LIBRARIES)
FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(){c();}\n")
ADD_EXECUTABLE(main main.c)
TARGET_LINK_LIBRARIES(main C)

The linker finds the correct libB.so but not the correct libA.so as
it is lured into the wrong location due to libB.so's absence on the
command line and the -rpath-link option's presence instead, and it
is the latter which leads to the wrong libA.so. If you disable the
offending TARGET_LINK_LIBRARIES(C LINK_INTERFACE_LIBRARIES), CMake
knows what has to be mentioned on the command line, and everything
is fine. Of course, the above-noted example is quite pathological,
but there're the weirdest setups out there, and with each external
library, i.e. not associated with a target, the risk that one will
hit the wrong one - or none at all - will even be higher. Thus, be
careful when cutting down the transitive dependencies by the usage
of LINK_INTERFACE_LIBRARIES, and *always* use full paths when you
refer to non-target-associated libraries for linking; *never* use
LINK_DIRECTORIES(), -L or the like - your problem's actual cause,
if I'm not mistaken.

W.r.t. static libraries, things are fundamentally different: Firstly,
they cannot refer to and cannot be referred to by other libraries, so
the linker can't track them by itself. Secondly, when they're created,
no code from any other library is transferred to them. Thus, they must
appear in the link command line if they're mediate prerequisites, i.e.
when linking against a binary containing unresolved symbols which are
defined in the said static libraries. This is particularly true when
static libraries are linked against other static libraries, but also
when static libraries are linked against shared libraries.

In the mixed case of a shared library C linked against a static one
B - supposedly the case your are talking about - you can't have un-
resolved symbols in C that are defined in B as the concerned object
code would be transferred from B to C during the latter's creation.
However, it might happen what Andreas has mentioned meanwhile, e.g.
C.h includes B.h and, thus, offers symbols from B it neither uses
nor defines, so B must be present when linking binaries that refer
to these symbols on behalf of C.h alone. Of course, CMake can not
recognize such subtleties, and one might argue if it's good style.

Basically, it's also possible that C, say, inherits an unresolved
symbol from B, e.g. if B itself is linked against a static A that
defines a symbol "x" referred to by B, "x" might get into C as un-
resolved if C is just linked against B and not against A. In this
case, A must be mentioned when linking a binary against C; i.e.,
transitive link dependencies on static libraries must not be cut
down. Note that CMake ensures this for project-internal library
targets, because one would need to cut B's dependency on A, but
LINK_INTERFACE_LIBRARIES does not apply to static libraries -
for exactly this reason, if I understand correctly.

In the end, you are right in as much as CMake sometimes mentions
static libraries in the link command that strictly don't need to
appear, but CMake simply plays safe in this regard, as David has
already outlined. OTOH, unnecessary static libraries in the link
command usually don't cause harm to the final binary; they might
prolong the link time a bit, but this is most often negligible.

> BTW, since the LINK_INTERFACE_LIBRARIES seems to work for static libraries
> it did solve my immediate problem, but the behavior for the shared
> libraries seems wrong to me.

Does the link command line work if you remove libB.dylib, i.e. is the
latter actually unnecessary for the link operation to succeed? If so,
this seems indeed to be an issue, since the user should be able to
control such things, IMO. Perhaps, that's worth a bug report?

Regards,

Michael

PS: CMake isn't the only program that tends to transitively pull in
link dependencies, see ld's --[no-]copy-dt-needed-entries options.