[CMake] ExternalProject, continuous integration and caching builds

[Isaiah Norton]

>
> CMake notices the downloaded tarball is up-to-date and
> doesn't download it again, but it still extracts it again


>From what I can tell, the 'check|download tarball' and 'extract tarball'
commands are independent parts of the "download step": as long as the
download step runs at all, it will re-extract the tarball, even if it
skipped re-downloading. So the issue (design questions aside) is what
triggers the download step. The minimal dependency for that step looks like
"LIBNAME-gitinfo.txt" file somewhere in the stamp directory, so you could
check the mtime preservation there. I don't know if there are other
dependencies added for the download target added separately.

On Wed, Sep 5, 2018 at 8:06 AM Antoine Pitrou <antoine at python.org> wrote:

>
> Hello,
>
> On our project (Apache Arrow - https://arrow.apache.org/) we're using
> CMake for the C++ source tree and have many external dependencies
> fetched using ExternalProject.  In turn building those dependencies can
> make up a significant portion of build times on CI services, especially
> AppVeyor.  So I've been looking for a solution to cache those
> third-party builds from one CI run to the other.
>
> Right now, what I'm trying to do is to set EP_BASE to a well-known base
> directory and ask AppVeyor to cache and restore that directory in each
> new build.  The AppVeyor caching seems to work fine (the EP_BASE
> directory is saved and restored).  However, it seems that nevertheless
> CMake will rebuild all those projects again, despite the cached build
> results.
>
> This is with CMake 3.12.1 on Windows.
>
> Here is the log for an example build step, here the zstd library:
>
> https://ci.appveyor.com/project/pitrou/arrow/build/1.0.700/job/i4tj6tifp4xq1mjn?fullLog=true#L803
>
> As you can see, CMake notices the downloaded tarball is up-to-date and
> doesn't download it again, but it still extracts it again (why?) and
> builds the source code anew.  Yet the entire EP_BASE directory (here
> "C:/Users/appveyor/arrow-externals") is cached and restored by AppVeyor.
>
> Did someone manage to make this work, and/or is there another solution?
>
> Thank you
>
> Regards
>
> Antoine.
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180906/e4e4b510/attachment.html>