[CMake] One build, multiple compilers and packages

[Craig Scott]

Thanks for the suggestions Clinton. The idea of using a custom target to
invoke CMake's --build feature on a sub-project was the piece I needed.
When combined with ExternalProject, this gives me the behavior I was after.
I use ExternalProject to define a separate build for each compiler and some
additional custom targets ensure a build from the top level picks up
changes to files in the sub projects. It's still a little bit awkward, but
definitely workable.

The packaging part was best handled by defining each package in its own
ExternalProject too and making use of CPACK_INSTALL_CMAKE_PROJECTS to
define the components to pull in from the various builds. This meant I
could keep the structure of everything relatively close to being as though
it was all part of a single unified build and not have to bother with
importing/exporting targets, etc.




On Mon, Aug 26, 2013 at 11:19 PM, <clinton at elemtech.com> wrote:

>
>
> ------------------------------
>
> Hi. First, apologies for the length. If you are not interested in mixing
> different compilers and generating multiple packages all within one build,
> you can probably stop reading now.
>
> After trying various approaches and not being entirely satisfied with any,
> I thought I'd seek the collective wisdom on this list. The details below
> define the scenario, the main problems faced and some things that have been
> tried to address them. Comments, ideas or suggestions would be most welcome.
>
> SCENARIO:
>
> We have a large code base for which different components need to be built
> with different compilers. Some parts are desktop software (applications,
> libraries) and others are firmware for embedded devices. All are pulled
> together into a small number of packages using CPack. Some of those
> packages share common items and there is no correlation between compilers
> and packages. Most items are included in more than one of the packages. For
> instance, firmware might be included in all packages, a particular desktop
> application or library in a subset of the packages, etc. On a given
> platform, some packages will have one format and other packages will have a
> different format (eg a mix of RPM and tarball). We create packages for at
> least Windows and Linux.
>
> Most of the source code is in a single git repository and after testing
> different strategies, this is an arrangement we want to keep. There are a
> couple of components being built using CMake's ExternalProject feature,
> either because they use a different build system that is currently too
> complex for us to change to CMake or the source code comes from a 3rd party.
>
>
> PROBLEMS:
>
>    1. CMake assumes you only want to use one compiler for an entire build
>    tree.
>    2. CPack assumes you are only building one package. Some package
>    generators allow you to do a component-based setup where multiple packages
>    can be generated, but this is not supported for all package generators (eg
>    the NSIS package generator won't produce multiple packages).
>
>
> APPROACHES TRIED (feedback welcome):
>
> The first of the above two issues is the more problematic. I've spent
> considerable time experimenting with ExternalProject as a potential way of
> addressing various issues, including the "one compiler per build tree"
> restriction. What I found was that dependency handling and integration with
> CPack becomes much more complicated, and in some cases has deficiencies not
> present in a single unified CMake build. For example, once you have
> successfully built an external project, the top level build won't rebuild
> it again unless something about its configuration changes. Thus, if you
> subsequently make changes to a source file, simply building the top level
> project won't work because it still thinks the sub project is up to date.
> This is because ExternalProject uses time stamps rather than forwarding the
> build request to the sub project to let it decide if anything needs to be
> built. This kinda makes sense, since an ExternalProject is meant to
> encompass everything from download through to install, but this is not how
> we are trying (and needing) to use it. ExternalProject starts to become
> less useful when applied to source code you are actively working on.
>
> Effectively, what we really need is the ability to let add_subdirectory()
> create a whole new build with a potentially different compiler, but still
> have it aware of all the other targets and dependencies and also to
> propagate all its own the targets, dependencies, etc. up to the parent. We
> experimented with exporting targets from the ExternalProject's using the
> export features of the install command, but this was messy and ultimately
> still suffers from the "isn't aware of changes to the source" problem. It
> also has implications for stripping of targets (for an example, see
> http://public.kitware.com/Bug/view.php?id=14157 ). Packaging also gets
> much more cumbersome when ExternalPackage is involved.
>
> For the second problem, my current workaround is to create a custom target
> which re-configures and re-builds the project for each package I need to
> create. The targets only need to get built once, but the package details
> change each time and a different package gets built. In essence, the build
> calls itself with a set of different configurations, which seemed a bit
> mind-bending at first, but was fairly simple to implement and didn't seem
> to have much in the way of negative side effects. The approach works
> reliably, but it feels like a roundabout solution to something that
> CMake/CPack should be able to handle much more cleanly. In a similar vein
> to the suggestion above, it would be really nice if add_subdirectory() had
> the ability to define a package that was not affected or did not affect
> other directories. Thus, you could have one package per sub directory. Any
> common variables could be defined at the parent level and would propagate
> down to the sub directories. Alas, this is not something that
> add_subdirectory() currently supports.
>
> I'd be interested in hearing what other people have done or would do to
> address the above problems for the specified scenario. I'm happy to discuss
> in more detail things I've tried if people are interested, but hopefully
> the above will stimulate some ideas or discussion about how these two
> problems can be better handled with CMake/CPack. These are great tools for
> which I'm sure these limitations can be overcome, either through methods I
> haven't considered or improvements to the tools themselves.
>
>
> This idea may or may not work well for you.
>
> To replace your ExternalProject, with a method that has deep build
> dependencies, you can try this:
>
> Do an execute process so the configure step of the sub project happens at
> the same time as the configure of the parent project:
>
> set(sub_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/sub" CACHE INTERNAL "")
> set(sub_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/sub" CACHE INTERNAL "")
>
> execute_process(WORKING_DIRECTORY ${sub_BINARY_DIR}
> COMMAND ${CMAKE_COMMAND} -G "Unix Makefiles"
> # standard cmake options
> -D CMAKE_C_COMPILER:FILEPATH=${CMAKE_C_COMPILER}
> -D CMAKE_OSX_ARCHITECTURES:STRING=${CMAKE_OSX_ARCHITECTURES}
> -D CMAKE_C_FLAGS:STRING=${CMAKE_C_FLAGS}
> -D CMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
> # 3rd party options
> .....
> # the source tree
> "${sub_SOURCE_DIR}"
> )
>
> Then the build step can be handled like this ( you may need to fill out
> dependencies or configuration type ):
>
> add_custom_target(sub ALL
> COMMAND "${CMAKE_COMMAND}"
> --build "${sub_BINARY_DIR}"
> COMMENT "Building sub..."
> VERBATIM
> )
>
>
>
> For the packaging part, you can create multiple cpack config files so you
> do not have to regenerate a single config file.
> To use a specific config file, use the --config option to cpack.  To
> select parts of the software to go into certain packages, you can use
> components, and each cpack config file can list which components to include
> in that package.
>
>
> To include the disconnected sub projects in packaging, I am guessing you
> can try something like this:
> install(CODE "
>  include(\"${CMAKE_CURRENT_BINARY_DIR}/sub/cmake_install.cmake\")
> ")
>
> Clint
>
>


-- 
Craig Scott
Melbourne, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20130828/159da155/attachment-0001.htm>