[CMake] Looking for advice on creating targets that use external packages

[Michael Hertling]

On 03/01/2011 04:52 PM, L. A. Pritchett-Sheats wrote:
> Thanks for the reply.
> 
> Just so I understand, is the following statement accurate: FIND_PACKAGE
> for XXX sets plural XXX_INCLUDE_DIRS and XXX_LIBRARIES which include all
> the needed include paths and libraries to compile and build anything using
> XXX.

Exactly. In addition, a find module or config file usually sets the
cached XXX_INCLUDE_DIR and XXX_LIBRARY variables to the XXX include
directory and library file - typically as results of FIND_PATH() and
FIND_LIBRARY() - to prevent multiple searchings if FIND_PACKAGE(XXX)
is invoked more than once or to preset the results in a GUI or on the
command line if FIND_PACKAGE(XXX) can't or shouldn't find the include
directory or library for whatever reasons. See [1] for further details.

> In my project that calls XXX
> 
> add_library(foo ${some_source_files})
> target_link_libraries(foo ${XXX_LIBRARIES})
> 
> resolves all the dependencies at link time.

Yes, w.r.t. XXX.

> We were looking into imported targets instead of explicit paths because we
> were concerned about the command line length and we have already seen on
> Mac OSX complaints from the compiler about repeated  *.dylib files. I'm
> assuming that if we define imported targets, that CMake would not repeat
> libraries in the compile line.

AFAIK, CMake repeatedly mentions libraries on the link line if they are
necessary to resolve known dependencies, e.g. with circularly dependent
static libraries. OTOH, CMake drops libraries from the link command if
they're discernibly unnecessary for dependency resolution, see [2] for
an example. However, IIRC, imported targets were mainly introduced due
to the flexibility they provide in regard of different build types or
configurations, respectively; e.g.

ADD_LIBRARY(XXX SHARED IMPORTED)
SET_TARGET_PROPERTIES(XXX PROPERTIES IMPORTED_LOCATION_CUSTOM ...)

would make the linker use a special version of XXX if the build type
equals "CUSTOM" which is hardly possible with the path-based approach.

> Here's the specific from my project. We have code (foo) using Boost, HDF5,
> NetCDF, ExodusII (Mesh database), MOAB (Mesh framework) and Trilinos (huge
> solver framework package). The dependencies are:
> NetCDF -> HDF5, ExodusII -> NetCDF, MOAB->NetCDF, MOAB->HDF5, foo->MOAB,
> foo->Boost, foo->Trilinos.
> We either written a FIndXXX module or are using a config file
> (Trilinos)for each of the external libraries.
> 
> HDF5 has been the difficult case here. The FindHDF5.cmake module usually
> returns more than one library in HDF5_LIBRARIES (hdf5,hdf5_hl,z,m,...). So
> in this case I would need to create an imported target for each library in
> HDF5_LIBRARIES, correct? [...]

Yes, for the HDF5 libraries if you want to give imported targets a try.
Without knowing HDF5, I'd roughly do the following in its find module:

FIND_PACKAGE(ZLIB)
FIND_LIBRARY(HDF5_LIBRARY hdf5 ...)
FIND_LIBRARY(HDF5_HL_LIBRARY hdf5_hl ...)
IF(NOT TARGET hdf5 AND NOT TARGET hdf5_hl)
    ADD_LIBRARY(hdf5 SHARED IMPORTED)
    ADD_LIBRARY(hdf5_hl SHARED IMPORTED)
ENDIF()
SET_TARGET_PROPERTIES(hdf5 PROPERTIES IMPORTED_LOCATION ${HDF5_LIBRARY})
SET_TARGET_PROPERTIES(hdf5_hl PROPERTIES IMPORTED_LOCATION
                                         ${HDF5_HL_LIBRARY})
SET(HDF5_LIBRARIES hdf5 hdf5_hl ${ZLIB_LIBRARIES} m)

However, if a package provides multiple libraries which can be used
selectively/independently, it is possibly worth to consider whether
the find module or configuration file can be made component-aware.

> [...] The other difficultly is Trilinos, which can have
> up to +30 libraries. Do I need to create an imported target for each of
> those libraries? [...]

Yes, if they're intended to be used explicitly by other projects. If
some of them are only needed by other libraries but are not meant to
be referred to by the package's users, you might mention the formers
in the IMPORTED_LINK_INTERFACE_LIBRARIES properties of the latters.
E.g., suppose the library XXX is linked against a library YYY, but
YYY does not have a public API; you might do in the config file:

IF(NOT TARGET XXX)
    ADD_LIBRARY(XXX SHARED IMPORTED)
ENDIF()
SET_TARGET_PROPERTIES(XXX PROPERTIES
    IMPORTED_LOCATION "path/to/libXXX.so"
    IMPORTED_LINK_INTERFACE_LIBRARIES "path/to/libYYY.so"
)

Then, if the package's users mention XXX in TARGET_LINK_LIBRARIES(),
libYYY.so will appear on the link command line, too, but YYY can't
be directly referred to as a target, so it's hidden from the users.

> [...] And if that is the case I can see why many projects do
> not do this and instead just use XXX_LIBRARIES as a list of libraries. I
> grep'd through the Modules directory in my CMake and only found one
> FindXXX.cmake module creating imported targets, FindQT.cmake.

If I'm not mistaken, most of the modules are already present since the
days of CMake 2.4, i.e. the pre-imported-target era, and many of them
do not have a maintainer, so they are rarely touched. However, if you
provide a configuration file instead of a find module, as you say for
Trilinos, this means the package is CMake-aware; usually, it is even
built with CMake. In that case, you might use the INSTALL(EXPORT ...)
command to have CMake generate the imported targets by itself, see [3].
Note that imported targets should be protected against redefinition, so
the ADD_LIBRARY/EXECUTABLE(... IMPORTED) commands should be encompassed
by IF(NOT TARGET ...) ... ENDIF(); otherwise, it's not possible to issue
multiple FIND_PACKAGE() calls for the same package within the same scope.
This aspect is is not taken into account in the abovementioned tutorials.

> Since we are starting our project from scratch and I am not that familiar
> with CMake, wanted to be sure the build system was following "best
> practices".

Although imported targets have been around for some time, they're used
somewhat hesitantly, IMO. Nevertheless, for new projects, they should
be absolutely the way to go due to their undeniable advantages.

> Thanks, again.

Regards,

Michael

[1] ${CMAKE_ROOT}/Modules/readme.txt
[2] http://www.mail-archive.com/cmake@cmake.org/msg34361.html
[3] http://www.cmake.org/Wiki/CMake/Tutorials#CMake_Packages

PS: Please reply to the ML, so others may benefit from the discussion.

>> On 02/24/2011 06:00 PM, L. A. Pritchett-Sheats wrote:
>>>
>>> I'm working on a software project that requires about a dozen external
>>> packages to build our libraries. What is the correct way to define the
>>> dependencies for our targets that depend on these external libraries?
>>>
>>> So far I have created FindXXX.cmake files for each external library.
>>> These
>>> files set XXX_INCLUDE_DIRS and XXX_LIBRARY or XXX_LIBRARIES. Should I
>>> make
>>> a target in our build for XXX?
>>>
>>> Example A:
>>> in the root CMakeLists.txt file:
>>> find_package(UnitTest)
>>> add_library(unittest++ STATIC IMPORTED)
>>> set_target_property(IMPORTED_LOCATION ${UnitTest_LIBRARY})
>>>
>>> then in a subdirectory foo/CMakeLists.txt
>>> add_executable(test_foo test/Main.cpp test/test_foo.cpp)
>>> add_test(foo test_foo)
>>> target_link_libraries(test_foo foo unittest++)
>>>
>>> or should each subdirectory that contains a CMakeLists.txt file that
>>> needs
>>> this library to link to an executable explicitly define the library in
>>> the
>>> target's target_link_libraries call? Like this....
>>>
>>> Example B:
>>> in foo/CMakeLists.txt:
>>> find_package(UnitTest)
>>> add_executable(test_foo test/Main.cpp test/test_foo.cpp)
>>> add_test(foo test_foo)
>>> target_link_libraries(test_foo foo ${UnitTest_LIBRARY})
>>>
>>>
>>> We've been following Example A, but have run into numerous link problems
>>> because we are using NetCDF, HDF5, ExodusII and a mesh framework package
>>> that depends on all 3. I've looked at two other projects that use CMake:
>>> Trilinos and Yarp. Both follow Example B and since we are having so much
>>> difficulty resolving links at run time, I wanted to know if we are
>>> defining these dependencies correctly.
>>>
>>> Thanks for your time.
>>
>> At first, your FindXXX.cmake files, provided XXX is a library, should
>> define XXX_LIBRARIES in any case since this variable is meant to hold
>> XXX_LIBRARY - usually the result of a FIND_LIBRARY() call - along with
>> XXX's prerequisite libraries. In other words, XXX_LIBRARIES contains
>> anything necessary to link against XXX. Mentioning UnitTest_LIBRARY
>> instead of UnitTest_LIBRARIES in TARGET_LINK_LIBRARIES() might be a
>> cause for the problems you report on.
>>
>> Furthermore, defining imported targets for external libraries should be
>> the find module's job; it's intended as a more flexible alternative to
>> the direct specification of a library's full path, e.g. The necessary
>> prerequisites are specified appropriately enough by target properties
>> IMPORTED_LINK_INTERFACE_LIBRARIES. As it were, dropping them might
>> result in unresolved dependencies at link time.
>>
>> Finally, if the target "foo" is sufficiently independent, particularly
>> if it can be configured and built by itself, I would tend to place the
>> needed FIND_PACKAGE() calls in foo's CMakeLists.txt as in example B
>> above, even if these calls appear in other CMakeLists.txt files, too.
>> In that way, this part of the project becomes more self-contained and
>> you do not need to rely on any FIND_PACKAGE() calls to be made before
>> somewhere in the current scope. Additionally, well written find modules
>> can be safely called multiple times and due to the cache, they do their
>> work usually once. OTOH, if "foo" is tightly connected with other parts
>> of the project, e.g. if it is a module which isn't useful on its own or
>> doesn't make sense without a main program, it's probably appropriate to
>> have a superordinate CMakeLists.txt call FIND_PACKAGE() and rely on the
>> results.
>>
>> In summary, I'd recommend to write FindXXX.cmake files for external
>> libraries which set up imported targets *with* their prerequisites,
>> i.e. IMPORTED_LOCATION and IMPORTED_LINK_INTERFACE_LIBRARIES target
>> properties. Note that the find modules need to protect the imported
>> targets from being defined twice; this is important when the same
>> FIND_PACKAGE() call is multiply issued within the same scope. The
>> imported targets, however, should be conveyed to the caller via the
>> XXX_LIBRARIES variables, so the calling CMakeLists.txt file can just
>> say TARGET_LINK_LIBRARIES(... ${XXX_LIBRARIES}) regardless whether the
>> FindXXX.cmake file uses imported targets or returns full paths directly.
>>
>> BTW, which link problems do you have exactly? You have also mentioned
>> a "difficulty resolving links *at run time*", so are you sure these
>> problems are caused by the find modules and the way you use them?
>>
>> Regards,
>>
>> Michael