[CMake] [cmake-developers] Need ideas/opinions on third party library management

[Ruslan Baratov]

On 16-Aug-16 17:04, Florent Castelli wrote:
>
>> On 16 Aug 2016, at 15:29, Ruslan Baratov <ruslan_baratov at yahoo.com 
>> <mailto:ruslan_baratov at yahoo.com>> wrote:
>>
>> On 16-Aug-16 13:52, Florent Castelli wrote:
>>> For example, Boost is used by 5 platforms: Windows, OSX, Linux, 
>>> Android and iOS.
>>> Each platform has a different CPU target (or many 32/64bit, x86/ARM).
>>> Each platform has many compilers.
>>> Some platforms have instrumentation options (Debug / Release, ASan, 
>>> MSan…) and really need
>>> to be compiled properly, otherwise you’ll end up with false positives.
>>> The matrix of builds is REALLY hard to track. Each time we update 
>>> Boost, we had to update
>>> a lot of things.
>> Not a problem for Hunter.
>>
>> Linux, OSX, Windows, iOS 9.3, iOS 8.2, Android, GCC, Clang, ASan, 
>> LeakSan, ThreadSan, Static Analyzer,  libstdc++, libc++, MinGW, 
>> Visual Studio 2008-2015:
>> * https://travis-ci.org/ingenue/hunter/builds/140317830
>> * https://ci.appveyor.com/project/ingenue/hunter/build/1.0.665
>>
>> This list is not even full, I guess I can add more toolchains in 
>> future (GCC variations and C++ standards).
>>
>> To test all matrix I need to push one commit to pkg.boost branch, to 
>> upload binaries to server I need to push one commit to upload.boost 
>> branch (upload ALL toolchains at one shot). To reuse all updates 
>> users just need to set new URL/SHA1 of HunterGate module: 
>> https://github.com/ruslo/hunter/releases
>>
>>> Overall, building boost takes 10s on our developers’ machines. The 
>>> sources aren’t changed often,
>>> so the cost is pretty low.
>> What kind of hardware do they have? And what libraries you mean? It 
>> takes about 20 seconds on my Linux machine just to unpack 80 MB of 
>> Boost release archive. It's even worse on Windows, it takes several 
>> minutes for some strange reason even on SSD + Core i7. Using binaries 
>> in such cases is a huge time saver because there is no need to 
>> compile anything and there is no a lot of junk that they put into 
>> release archive (if you remove docs and tests 80 MB became 15 MB).
>
> I consider the time to download Boost isn’t part of the build.
I'm not counting that. It takes 20 seconds just to unpack archive that 
already downloaded. Anyway I'm just wondering what is possible to do 
with Boost for 10 seconds. Install header-only libraries? Build 1-2 
libraries? All of them? :)

> The Boost source download is cached outside of the build directory in 
> a unique folder.
> So it’s effectively only done once for all platforms and then reused.
This is true for local machines and for custom build servers like your 
personal Jenkins. For Travis/AppVeyor you have to create root folder 
with 3rd parties from scratch for each build (at least for free public 
accounts).

>
> You’ll also have symbols and the sources available for debugging 
> properly, and they’re
> not always available with a binary distribution.
Just to clarify: with Hunter you're creating binaries from sources, so 
everything you install with `cmake --build _builds --target install` (or 
`./b2 install` for Boost) is available.

>
>
>> Of course building from source is not an option for such monsters 
>> like Qt or OpenCV. Ready-to-use binaries is something critical for 
>> real life applications. There is no way to test everything on 
>> Travis/AppVeyor without this feature.
>
> Well, you don’t have to use Travis or AppVeyor.
It's the most powerful and easy to setup services I know. If you know 
any better solutions please share.

> Spotify isn’t at the same scale as most
> projects hosted there and we have different requirements and resources.
> Admittedly, Spotify doesn’t use Qt anymore, so this isn’t a problem 
> for us.
It's not about Qt, it's about expandability. Use 20 of smaller libraries 
and you will have quite the same issues.

>
> Note that by integrating everything in the same project, you also have 
> proper dependencies
> and you will only build what you need. You may save some time by doing 
> that.
> And caching is important, when done in the right way!
With Hunter you're installing only what you need too, it does respect 
options like FOO_WITH_OPENSSL or FOO_WITH_QT, it download only binaries 
for toolchain you're currently working on, etc.

Don't want to make a discussion too broad. You said that it's hard to 
manage binaries for a lot of configuration, I'm saying that it's 
possible and is very handy.

Ruslo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20160816/56632192/attachment-0001.html>