[CMake] One build, multiple compilers and packages

[clinton at elemtech.com]

----- Original Message -----

> Hi. First, apologies for the length. If you are not interested in mixing
> different compilers and generating multiple packages all within one build,
> you can probably stop reading now.

> After trying various approaches and not being entirely satisfied with any, I
> thought I'd seek the collective wisdom on this list. The details below
> define the scenario, the main problems faced and some things that have been
> tried to address them. Comments, ideas or suggestions would be most welcome.

> SCENARIO:

> We have a large code base for which different components need to be built
> with different compilers. Some parts are desktop software (applications,
> libraries) and others are firmware for embedded devices. All are pulled
> together into a small number of packages using CPack. Some of those packages
> share common items and there is no correlation between compilers and
> packages. Most items are included in more than one of the packages. For
> instance, firmware might be included in all packages, a particular desktop
> application or library in a subset of the packages, etc. On a given
> platform, some packages will have one format and other packages will have a
> different format (eg a mix of RPM and tarball). We create packages for at
> least Windows and Linux.

> Most of the source code is in a single git repository and after testing
> different strategies, this is an arrangement we want to keep. There are a
> couple of components being built using CMake's ExternalProject feature,
> either because they use a different build system that is currently too
> complex for us to change to CMake or the source code comes from a 3rd party.

> PROBLEMS:

> 1. CMake assumes you only want to use one compiler for an entire build tree.
> 2. CPack assumes you are only building one package. Some package generators
> allow you to do a component-based setup where multiple packages can be
> generated, but this is not supported for all package generators (eg the NSIS
> package generator won't produce multiple packages).

> APPROACHES TRIED (feedback welcome):

> The first of the above two issues is the more problematic. I've spent
> considerable time experimenting with ExternalProject as a potential way of
> addressing various issues, including the "one compiler per build tree"
> restriction. What I found was that dependency handling and integration with
> CPack becomes much more complicated, and in some cases has deficiencies not
> present in a single unified CMake build. For example, once you have
> successfully built an external project, the top level build won't rebuild it
> again unless something about its configuration changes. Thus, if you
> subsequently make changes to a source file, simply building the top level
> project won't work because it still thinks the sub project is up to date.
> This is because ExternalProject uses time stamps rather than forwarding the
> build request to the sub project to let it decide if anything needs to be
> built. This kinda makes sense, since an ExternalProject is meant to
> encompass everything from download through to install, but this is not how
> we are trying (and needing) to use it. ExternalProject starts to become less
> useful when applied to source code you are actively working on.

> Effectively, what we really need is the ability to let add_subdirectory()
> create a whole new build with a potentially different compiler, but still
> have it aware of all the other targets and dependencies and also to
> propagate all its own the targets, dependencies, etc. up to the parent. We
> experimented with exporting targets from the ExternalProject's using the
> export features of the install command, but this was messy and ultimately
> still suffers from the "isn't aware of changes to the source" problem. It
> also has implications for stripping of targets (for an example, see
> http://public.kitware.com/Bug/view.php?id=14157 ). Packaging also gets much
> more cumbersome when ExternalPackage is involved.

> For the second problem, my current workaround is to create a custom target
> which re-configures and re-builds the project for each package I need to
> create. The targets only need to get built once, but the package details
> change each time and a different package gets built. In essence, the build
> calls itself with a set of different configurations, which seemed a bit
> mind-bending at first, but was fairly simple to implement and didn't seem to
> have much in the way of negative side effects. The approach works reliably,
> but it feels like a roundabout solution to something that CMake/CPack should
> be able to handle much more cleanly. In a similar vein to the suggestion
> above, it would be really nice if add_subdirectory() had the ability to
> define a package that was not affected or did not affect other directories.
> Thus, you could have one package per sub directory. Any common variables
> could be defined at the parent level and would propagate down to the sub
> directories. Alas, this is not something that add_subdirectory() currently
> supports.

> I'd be interested in hearing what other people have done or would do to
> address the above problems for the specified scenario. I'm happy to discuss
> in more detail things I've tried if people are interested, but hopefully the
> above will stimulate some ideas or discussion about how these two problems
> can be better handled with CMake/CPack. These are great tools for which I'm
> sure these limitations can be overcome, either through methods I haven't
> considered or improvements to the tools themselves.

This idea may or may not work well for you. 

To replace your ExternalProject, with a method that has deep build dependencies, you can try this: 

Do an execute process so the configure step of the sub project happens at the same time as the configure of the parent project: 

set(sub_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/sub" CACHE INTERNAL "") 
set(sub_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/sub" CACHE INTERNAL "") 

execute_process(WORKING_DIRECTORY ${sub_BINARY_DIR} 
COMMAND ${CMAKE_COMMAND} -G "Unix Makefiles" 
# standard cmake options 
-D CMAKE_C_COMPILER:FILEPATH=${CMAKE_C_COMPILER} 
-D CMAKE_OSX_ARCHITECTURES:STRING=${CMAKE_OSX_ARCHITECTURES} 
-D CMAKE_C_FLAGS:STRING=${CMAKE_C_FLAGS} 
-D CMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE} 
# 3rd party options 
..... 
# the source tree 
"${sub_SOURCE_DIR}" 
) 

Then the build step can be handled like this ( you may need to fill out dependencies or configuration type ): 

add_custom_target(sub ALL 
COMMAND "${CMAKE_COMMAND}" 
--build "${sub_BINARY_DIR}" 
COMMENT "Building sub..." 
VERBATIM 
) 

For the packaging part, you can create multiple cpack config files so you do not have to regenerate a single config file. 
To use a specific config file, use the --config option to cpack. To select parts of the software to go into certain packages, you can use components, and each cpack config file can list which components to include in that package. 

To include the disconnected sub projects in packaging, I am guessing you can try something like this: 
install(CODE " 
include(\"${CMAKE_CURRENT_BINARY_DIR}/sub/cmake_install.cmake\") 
") 

Clint 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20130826/eea32b93/attachment-0001.htm>