[CMake] [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

[Craig Scott]

(dropping the cmake-developers list since this is really a user issue)


On Wed, Feb 20, 2019 at 2:34 AM Timothy Wrona <tjwrona1992 at gmail.com> wrote:

> Hi Craig,
>
> Thank you for the detailed description!
>
> To answer some of your questions:
>
>    - This project will not be incorporated into a Linux distribution,
>    however I would like to keep it cross platform and it should work on
>    Windows, Mac, and Linux.
>    - All of the pieces of the project that I am writing myself are using
>    CMake, but I do have some dependencies on external libs such as
>    "googletest" and "boost". Conan does seem to make using these external
>    dependencies very simple - especially since I am using MinGW to do my
>    compilation and "googletest" doesn't seem to compile out of the box on
>    MinGW without passing specific flags to the compiler. With Conan I get a
>    pre-compiled binary so it just works out of the box.
>    - At the current time, this project does not have many dependencies,
>    although it will likely grow quite large. I don't believe the external 3rd
>    party dependencies will need to update frequently, but all of the libraries
>    I am writing for the project will likely change quite a bit throughout
>    working on the project. I would also like to be able to compile these
>    libraries independently for reuse in other projects.
>    - I intend to support as many platforms/compilers as possible, I am
>    currently using Qt as my dev environment which allows me to
>    install/configure multiple compilers and try builds with each one - this
>    way I can at least do a build with both MinGW and msvc very quickly.
>    - I would like to support tools like sanitizers and code refactoring
>    tools at least within my own libraries (not necessarily with any of the 3rd
>    party libs)
>    - I would like past releases to be repeatable/rebuildable as much as
>    possible, although when the compiler is upgraded it is understandable that
>    past versions may have issues. I don't have much understanding of CI
>    systems at this point and it is something I have been trying to become more
>    familiar with. I'd like to avoid having multiple versions of the same
>    dependency, although I don't think having two versions of "googletest" for
>    two separate sub-projects that don't depend on each other would cause any
>    issues.
>       - I imagine Conan would make past releases more repeatable since
>       you can fetch binary packages instead of needing to rebuild from source and
>       package versions are always explicit.
>
> I am currently the only developer working on the project, but I would
> still like to find the most efficient method of managing packages for my
> particular situation since it will likely save me a lot of pain down the
> road if I get it right up-front. This project is in the situation where I
> have a set of different sub-projects (libraries) that all need to be able
> to be compiled independently for re-use, but I have a main project that
> will use them all. All of these projects have their own dependencies (some
> of them 3rd party, some of them written by me). The 3rd party libs will
> likely rarely change, but the ones written by me may change quite
> frequently.
>
> This is an approach I was thinking about taking:
>
>    - *So far I have tried this and it is working well* - Manage 3rd party
>    libs with Conan, since I don't need to see the source and it's quite
>    convenient to not need to recompile them this seems to work pretty well.
>    Using "FetchContent" on 3rd party libs and then attempting to compile them
>    yourself can sometimes be tricky (for example "googletest" requires special
>    compiler flags to be compiled with MinGW.) It also ensures that if two
>    projects ask for the same version of a dependency I get only one copy even
>    if the projects are built entirely independent of each other.
>    - *I haven't tried this yet, but I am hoping it will work well* - Put
>    my sub-projects (my own custom libraries) into their own independent git
>    repos and pull them into my main project using "FetchContent". Then when I
>    run "FetchContent" it will checkout the sub-projects and I will have all of
>    the source code available. I am hoping if I do this any changes I make to
>    the sub-projects can easily be committed and pushed back to their own
>    independent repositories.
>    - The alternative to this would be to put my own sub-projects into
>       their own Conan packages and use Conan to get them from the main project.
>       But I am thinking since they will change frequently, "FetchContent" may be
>       a better fit for this scenario.
>       - Maybe when the sub-project libraries reach a mature and stable
>       release they should be packaged into Conan and fetched in the main project
>       from Conan at that point?
>
> Let me know what you think! :)
>

Sounds like you are making progress exploring things already, you'll need
to make your own evaluations about which approach will work best for you.

As a data point, in my day job, most of the time I'm working with a project
hierarchy that spans about 40+ dependencies, most of which are internal
projects that can all be built standalone or as part of a larger whole.
Some are small and relatively trivial, others are much larger and have
varying age with the associated quirks that come with that. Most come from
git but some come from svn and a small number from release tarballs. Some
have code generators that also produce CMakeLists.txt files that other
parts of the build need to consume (see this stack overflow Q&A
<https://stackoverflow.com/q/36084785/1938798> for more on that). We use
both GoogleTest and Boost, plus various other third party projects (OpenSSL
is probably the one that gives us the most pain). We build for Linux,
Windows, Mac, Solaris, iOS and Android. We support our devs building with
Make, Ninja, Xcode or Visual Studio (we don't use MinGW). Many of the
projects are actively undergoing change across multiple teams and people,
most of whom don't have an interest in learning about CMake and build
systems and just want to focus on coding. The flexibility and robustness
that FetchContent has given us has been a critical factor in being able to
work effectively and efficiently. That and ccache. ;) We don't use a
package manager other than letting the OS provide some common things on a
few platforms. For some larger dependencies (e.g. NDK, Qt) we rely on
having pre-built packages installed at predictable locations and accept
that we will have to maintain these on our build slaves, although this is
an area that is still evolving.

I recently posted
<https://gitlab.kitware.com/cmake/cmake/issues/18831#note_509194> how I'm
currently working on bringing Boost into our build via a FetchContent-based
mechanism. That is still somewhat of a work-in-progress and has holes for
multi-config generators (i.e. Visual Studio and Xcode), but they don't look
insurmountable if those are important to you. It's by no means the only
approach, but it's a starting point if you want to explore it (once Boost
completes its move to CMake, I hope to be able to drop this and just bring
Boost into the build via add_subdirectory() like most of our other
dependencies).



>
>
> On Tue, Feb 19, 2019 at 5:56 AM Craig Scott <craig.scott at crascit.com>
> wrote:
>
>>
>>
>> On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <tjwrona1992 at gmail.com>
>> wrote:
>>
>>> I have been working on a new C++ project and I am trying to decide
>>> whether I should use CMake as my package management system or if I should
>>> use a dedicated package management tool such as Conan.
>>>
>>> For more information on Conan see: https://conan.io/
>>>
>>> I am trying to understand the main difference between using Conan to
>>> manage dependencies vs using CMakes "FetchContent" module. Are there any
>>> compelling reasons to prefer something like Conan?
>>>
>>
>> Excellent question, one that I think deserves a more detailed answer than
>> I can provide here, but I'll try to hit the main points as I see them.
>>
>> Personally, I think there is no "right answer" or "one size fits all"
>> when it comes to package management for a CMake project. What works well
>> for one situation, person or project may not be as convenient or suitable
>> for another. There are competing needs and views, some of which are
>> personal preferences, others are hard requirements from things like OS
>> distribution policies for packaging. Even just the maturity of a project
>> can have a big influence on how developers may prefer to handle its
>> dependencies and handle it as a dependency of other projects.
>>
>> The key thing for me is that the developer should ideally have choices
>> when it comes to this area. If a project hard-codes that its dependencies
>> must come from a particular provider (whether that be Conan, Hunter, vcpkg
>> or some other system), this might not be compatible with what the
>> developer's situation allows. You would need to weigh up whether it makes
>> sense to lock the developer into a particular package manager if they want
>> to use your project or not. An inappropriate choice here can mean lower
>> adoption of the project as some may reject it for consideration based on
>> this point alone.
>>
>> If instead a project relies only on find_package() to find its
>> dependencies, then it is up to the developer to ensure they are all
>> available. This could be done using whatever package manager the developer
>> finds convenient, or they could build the dependency projects from source
>> individually or they might set up a superbuild parent project that builds
>> the dependencies in the required order and makes dependees available to
>> dependers. This gives good flexibility at the cost of more responsibility
>> on the developer than perhaps some would want (again, it will be highly
>> situation-dependent).
>>
>> A drawback with find_package() is that it assumes you actually have a
>> packageable project. For a variety of reasons, this may not be the case.
>> Consider a large, complex project in its early stages and where multiple
>> teams are working on different subprojects which all get combined into some
>> larger whole. Each of the subprojects may need to be able to build on their
>> own with their own smaller subset of dependencies, but they also need to be
>> able to be incorporated into a larger build (think of different teams
>> working on core toolkits, rendering engines, different algorithm
>> strategies, multiple GUI applications, backend components, etc). No-one may
>> know yet how it should get packaged up and everyone might be focused on
>> just getting a minimal viable prototype up and running as a technical
>> demonstrator. For a case like that, neither find_package() nor a package
>> manager really fits the workflow. In this situation though, FetchContent is
>> a perfect fit, since it doesn't require any packaging to already be in
>> place, it needs no external tools other than CMake and it gives each
>> project precise control over its dependencies down to the individual commit
>> to bring in for each one.
>>
>> With the above in mind, perhaps the following few questions may be
>> helpful in clarifying what your constraints are and maybe steering you more
>> toward one way or the other:
>>
>>    - Will the project be incorporated into a Linux distribution at some
>>    point (not just be installed on Linux, but be part of the actual Linux
>>    distribution as provided by its own native package manager)? If so, I would
>>    expect this would pretty much eliminate using any package manager and
>>    instead require that you use find_package() to find all dependencies.
>>    - Are any of the dependencies of the project using a build system
>>    other than CMake? If so, they tend to take a bit more work to incorporate
>>    into a build via ExternalProject or FetchContent. If you can assume the
>>    developer provides those somehow, bringing them in by find_package() shifts
>>    responsibility for building them from the project to the developer (or
>>    whatever package manager they choose to use). This might be anywhere from
>>    entirely appropriate to entirely problematic depending on who your intended
>>    audience is.
>>    - How many dependencies does the project have and what is the
>>    maturity of each one? Will the project need to update any of those
>>    dependencies often (are they also being actively developed, do you need to
>>    follow recent work in them)? What will be the impact on developers working
>>    on the project when these dependencies need to be updated (how easy is it
>>    for them to update, what assumptions are you making about their development
>>    environment and tools)?
>>    - What breadth of platforms, compilers and CMake generators do you
>>    want to support? The bigger this set, the more it may drive you towards
>>    building dependencies from source rather than relying on having pre-built
>>    packages available to you.
>>    - Do you want developers to be able to use tools like sanitizers,
>>    perform code refactoring across projects or have source code visibility
>>    into dependencies within their IDE tools? FetchContent supports all of
>>    these requirements quite naturally, but doing so for the other methods it
>>    may be more difficult.
>>    - How will you manage repeatability of building past releases? Will
>>    you be able to build a year old release again if you update a build slave?
>>    In particular, what assumptions are you making about a CI system's build
>>    slaves and what dependencies they have installed (can you have multiple
>>    versions of any given dependency available at once, whether that be at a
>>    known path or via whatever package manager(s) you choose to use)?
>>
>>
>> On a side note, this is an area I'm increasingly thinking about these
>> days (I'm the author of the FetchContent module). I'm interested in hearing
>> peoples' views on dependency management and building an understanding of
>> what works for people and what doesn't. If you want to send feedback my
>> way, do feel free to get in touch.
>>
>
-- 
Craig Scott
Melbourne, Australia
https://crascit.com

Get the hand-book for every CMake user: Professional CMake: A Practical
Guide <https://crascit.com/professional-cmake/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20190220/a885fa73/attachment-0001.html>