[CMake] [cmake-developers] Need ideas/opinions on third party library management

[Ruslan Baratov]

On 17-Aug-16 04:29, Florent Castelli wrote:
>
>>> The Boost source download is cached outside of the build directory 
>>> in a unique folder.
>>> So it’s effectively only done once for all platforms and then reused.
>> This is true for local machines and for custom build servers like 
>> your personal Jenkins. For Travis/AppVeyor you have to create root 
>> folder with 3rd parties from scratch for each build (at least for 
>> free public accounts).
> Yes. If you're using a free shared service, that's not something you 
> can count on.
> If you host your CI, you can do neat tricks like this, use ccache or 
> other similar techs.
What if you don't need "tricks" and make everything plain and simple?

>
>>
>>>
>>> You’ll also have symbols and the sources available for debugging 
>>> properly, and they’re
>>> not always available with a binary distribution.
>> Just to clarify: with Hunter you're creating binaries from sources, 
>> so everything you install with `cmake --build _builds --target 
>> install` (or `./b2 install` for Boost) is available.
> So you build each dependency separately, install them, and then use 
> them in your top level dependency.
No, in shared root folder.

> That works, but you have the extra overhead of the instrumentation for 
> the whole build.
> Even a no-op build will have to recurse in the build folder for each 
> dependency and run again, which is slow.
> This is why I prefer Ninja to the Makefile generator: it has a global 
> view of everything and you get a quick no-op build.
Not sure I understand that. What is the overhead?

>
>>
>>>
>>>
>>>> Of course building from source is not an option for such monsters 
>>>> like Qt or OpenCV. Ready-to-use binaries is something critical for 
>>>> real life applications. There is no way to test everything on 
>>>> Travis/AppVeyor without this feature.
>>>
>>> Well, you don’t have to use Travis or AppVeyor.
>> It's the most powerful and easy to setup services I know. If you know 
>> any better solutions please share.
> Well, you don't have a full control on the environment, so I wouldn't 
> say it's the most powerful.
Please share your solution. I have worked with Jenkins before and will 
not say that it's something easy customizable. Shareable folder - yes, 
good, but other stuff is a complete pain.
You can setup Travis in a seconds, add AppVeyor and you got Linux, OSX, 
Windows testing. How much time will take to start and tune Jenkins 
master and connect several slaves with different OSes on them?
Then add bunch of projects and tune them, then create dev branches for 
testing and so on.

> Convenient for sure. It probably fits smaller structures very well.
> Bigger companies have the resources to host their own service most of 
> the time and requirements that
> force them to do so. Those are probably the ones that will have the 
> manpower to handle a super-build type
> build system.
Why do not have both? Hunter can share downloads/root/cache on local 
folder for such CIs like Jenkins/custom so you don't need to rebuild 
everything.
At the same time binaries from server can be downloaded for 
"build-from-scratch" environment like Travis.

Anyway what about users? So you think it's okay to spend several hours 
of building from sources to just to run simple example of your product?

>
>>
>>> Spotify isn’t at the same scale as most
>>> projects hosted there and we have different requirements and resources.
>>> Admittedly, Spotify doesn’t use Qt anymore, so this isn’t a problem 
>>> for us.
>> It's not about Qt, it's about expandability. Use 20 of smaller 
>> libraries and you will have quite the same issues.
> As I said before, if I have build scripts for 20 small libraries and I 
> want to update a build flag affecting the ABI, I don't have to do 
> anything but just change the flag once.
> In your case, you'll have to tweak the 20 build scripts for each 
> library to reflect it.
> The dependencies are only intermediate products you use for the final 
> one. I don't want to deal with them constantly.
It's not true, I don't need to tweak 20 scripts, I just need to tweak 
one toolchain file.

>
>>
>>>
>>> Note that by integrating everything in the same project, you also 
>>> have proper dependencies
>>> and you will only build what you need. You may save some time by 
>>> doing that.
>>> And caching is important, when done in the right way!
>> With Hunter you're installing only what you need too, it does respect 
>> options like FOO_WITH_OPENSSL or FOO_WITH_QT, it download only 
>> binaries for toolchain you're currently working on, etc.
>>
>> Don't want to make a discussion too broad. You said that it's hard to 
>> manage binaries for a lot of configuration, I'm saying that it's 
>> possible and is very handy.
> I'm not saying it's impossible. I'm saying the overhead of managing 
> binaries is just a burden we shouldn't have to accept in the C/C++ world.
> If you can build build everything from source all the time in a 
> super-build, why wouldn't you do it?
Because it's not practical. I have such experience with Gentoo, I prefer 
do something useful instead of watching on "emerge world" progress. 
Super-build doesn't scale, what if you have two super-build projects how 
will you merge them?

> I'm lazy, I don't want to have to configure everything separately and 
> maintain them.
> You seem to have a lot of issues with Hunter anyway such as this one: 
> https://github.com/ruslo/hunter/issues/454
I don't understand how is this issue specific to Hunter? You will have 
same issues with super-build, Python-based build, brew, whatever...

> Just so you know, I shared the CMake file for Boost with some 
> employees who wanted to use it for a personal project running on both 
> iOS and Android: they had no configuration to do and it actually worked.
> It's one anecdote and doesn't mean much. but it's an approach that works.
I like approach of CMakefying Boost but without dependency manager it 
will not work smooth. So you have CMakeLists.txt for all Boost libraries 
or you have CMakeLists.txt for each component. I don't see the link to 
the code.

> It's not about saying which solution if better or worse. Each have 
> benefits and cons. If you want a traditional dependency management, go 
> for it.
> If you want to use your software Hunter, go for it.
>
> I'm merely explaining that with the solution we have in place, 
> everything is working great for us and we don't have any issue now 
> managing builds for many libraries.
> Robert asked about feedback and options, I'm just giving one more to him.
That's the problem with C++ community - everybody use tool that "works 
for me!". So package A use one build tool, package B decide that tool 
for A is not powerful enough and use custom tool. Package C want to use 
A and B so it have to ignore A/B system or create "integrator". Package 
D use C but want to use custom A so I have to sneak into C build process 
and patch it. Building Tower of Babel, it will not end well...

Real world example: Boost user b2, Boost.iostreams depends on ZLIB, ZLIB 
use CMake, I have to run CMake to install ZLIB then run ./b2 with custom 
flags to show where ZLIB is located. Now I want to add OpenCV to my 
project, and OpenCV has an internal copy of ZLIB, so I have to sneak 
into OpenCV build and teach it to use my version of ZLIB.

Ruslo