[CMake] Building third party libraries along with normal targets

[Miller Henry]

I have found the following projects which all seem to do some variation of a meta build so that you can build multiple projects that depend on each other and manage dependencies.  (there are a couple others that seem to not be maintained as well)

https://gradle.org/
https://bazel.build/ 
https://github.com/LLNL/spack 
https://github.com/ruslo/hunter 
http://www.biicode.com 
https://conan.io/
https://conda.io/

Unfortunately I have never found anyone who has actually compared even two of these. None of the projects have good marketing: it appears they somehow solve similar problems, but none actually have defined the problem or their solution.  It is like everyone assumes that everyone in the world has their exact same problem and the solution is obvious so the only thing left is the details of implementing it.  This of course tells me nothing about if they handle cross compiling (not a common use case but it is yours and mine), what packages they create, what compromises they make, what they expect of my environment...  These are important questions: I'm pretty sure that I could eliminate several just by comparing my needs to their features.

I'm currently using an in house system that builds everything in a Docker which lets me ensure nobody is accidentally using the wrong compiler. (we cross compile for a x86 target - 90% of the time if you build with gcc for the local system everything will work just fine, the other 10% of the time our system has an imcompatible version of some library and things blow up when you try to use some uncommon feature). I'm thinking about moving to one of the above, but I haven't actually evaluated anything.

If you do evaluate any of the above please document your experience and in particular what is good/bad about the things you look at.

This is getting off-topic for the cmake mailing list, but I don't know where else to move the conversation.


-----Original Message-----
From: CMake [mailto:cmake-bounces at cmake.org] On Behalf Of Robert Dailey
Sent: Wednesday, March 29, 2017 9:39 PM
To: Florent Castelli <florent.castelli at gmail.com>
Cc: CMake <cmake at cmake.org>
Subject: Re: [CMake] Building third party libraries along with normal targets

On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli <florent.castelli at gmail.com> wrote:
> On 30/03/2017 03:54, Robert Dailey wrote:
>>
>> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli 
>> <florent.castelli at gmail.com> wrote:
>>>
>>> This is known as "super build".
>>> Yes, this is exactly why I made my Boost CMake build scripts, which 
>>> you use unless you changed your mind today :)
>>
>> You mean this?
>> https://github.com/Orphis/boost-cmake
>>
>> It's on the drawing board, for sure. It's the best solution I've 
>> found. It's also part of the reason I'm asking this question to begin 
>> with.
>>
>> Although, this specific method of building boost doesn't use 
>> superbuild, it uses normal add_subdirectory() with target.
>
> It doesn't "super build" its dependencies, you have to provide them 
> externally, but it is a possible addition.
> But it is designed to be part of a super build environment and is used 
> as such at some large companies shipping to millions of people.
>
>>
>>> It can be done for other projects as well. Sometimes, they even 
>>> provide CMake build scripts you can use directly with 
>>> "add_subdirectory()" so you don't have to write CMake scripts or use 
>>> "ExternalProject_Add()" (which isn't all great since it doesn't 
>>> propagate all your current project settings).
>>
>> Long term, I'm thinking a project like hunter[1] would be the best 
>> choice. The problem with hunter is that it doesn't separate concerns 
>> between host machine and target host. Specifically, when cross 
>> compiling.
>>
>> All the superbuild commands assume linux command syntax, simply 
>> because I specified a toolchain utilizing android NDK. For this 
>> reason it requires a lot of TLC to get working.
>>
>> [1]: https://github.com/ruslo/hunter
>
> There are those issues and also ABI changing flags that can become an 
> issue as they aren't always propagated from what I can see.
>
>>> You can use ccache or its Windows variants to make it faster. But 
>>> you also shouldn't need "ninja clean" most of the time. Possibly, 
>>> you could just clean a specific target "ninja -t clean foo".
>>> If you declare all your dependencies properly, then you could just 
>>> always run "ninja" and the build will just be correct.
>>> If your purpose is to see the compiler output again (to fix 
>>> warnings), it is acceptable to clean and rebuild (with ccache it 
>>> should be fast enough).
>>> Personally, I just have a very long history in my terminal and 
>>> scroll back or pipe the build content to a file to look at it later. 
>>> Some IDEs will also record all the compilation output and make it 
>>> available later (Xcode does it), then it's less of an issue.
>>
>> Never used ccache before, what does it do? Also if even if you ninja 
>> -t to clean, it will also clean dependencies. So if I specify those 
>> in-project dependencies that actually build third party libs, those 
>> will be cleaned too.
>
>
> It's a program you use to record compilation outputs and save them to 
> a disk cache. If you build a file again, instead of compiling it, it 
> will just fetch the outputs from the cache and return the same 
> warnings. It's usually a safe optimization in a build pipeline and is 
> quite easy to use with CMake and Ninja generator.
>
> It works well with GCC or Clang and there are some alternative 
> versions for the Microsoft compiler too.

At the end of the day, I've solved the Boost problem but I have many other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be challenging but I think that's the proper way to go in my specific situation (multi platform). It'll be less maintenance than rebuilding binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github to work with superbuild? And do you prefer your superbuild execution to happen during cmake generate (basically cmake invokes itself for the superbuild) or do you prefer a separate external run, independent of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could hand write my own, but seems better to piggyback on someone else's efforts.

Thanks for your feedback!
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake