[CMake] Cmake Dependency Generation with Compiler Specific Plugins

[Brad King]

On 07/27/2010 06:48 PM, Russell Harmon wrote:
> We're looking to use cmake on a project which has some dependencies
> which are not properly picked up by cmake's dependency scanner. I'm not
> looking to fix the scanner, because I feel the scanner is fundamentally
> broken:

Yes, it certainly has some limitations.  I agree that the solution is
not to fix the current scanner.  I've considered using the "wave" c++
preprocessor library but it is problematic to locate all the implicit
include directories and preprocessor definitions for each compiler.
Using the compiler itself to get dependencies is much better.

FYI, our Fortran dependency scanner does do some limited preprocessing
to pick out correct Fortran 90 "use" and "module" directives.  This is
necessary just to generate a correct build.

> In order to properly scan a file for dependencies, you need to either
> use the compiler specific dependency scanner [1][2], or implement a
> fully featured c preprocessor. You can't do a partial job like the
> current dependency scanner does.

An ancient (1990's) predecessor to CMake actually did full preprocessing
of every translation unit and scanned for the "#line" directives or
similar constructs in the output.  The trade-off for such perfect
dependencies is that it takes almost twice as long to build because
every translation unit needs to be preprocessed twice.

We purposely chose to use a limited scanner in CMake which is very
fast and works well for many projects.  Since the set of #include
lines one can "grep" from a source file does not depend on the
preprocessing state we can share/cache results from scanning each
file.  This allows us to scan all dependencies in a target a few
orders of magnitude faster than the compiler takes to build it.
For non-developer (end-user or packager) builds that do not need
to rebuild this is very nice.

The opposing trade-off is that projects that use conditional and
macro-based includes do not get perfect dependencies.  This has not
been a problem for our needs in practice.  I'm not opposed to
adding support for more advanced dependency scanning in projects
that are willing to pay the cost of additional scanning time for
better dependencies.  See below.

> I'd encourage you to read "Recursive Make Considered Harmful" [3]

Ironically it is impossible to do make-time implicit dependency
scanning without recursive make (or include directive reload magic
extensions in GNU make).  After generating dependencies make needs
to invoke another make recursively to load them.  When we throw
generated source files into the mix then we cannot even scan them
until the sources have been generated.

> 1. #define ARCH x86
>    #define ASM_INCLUDE <ARCH/asm.h>
>    #include ASM_INCLUDE
>    cmake adds a dependency on ASM_INCLUDE instead of x86/asm.h

A simple test example shows me that CMake skips this dependency
rather than generating a broken build.

> 2. #ifdef DEBUG
>    #include <debug_config.h>
>    #else
>    #include <config.h>
>    #endif
>    cmake adds a dependency on both debug_config.h and config.h
>    In a worst-case scenario here (one which our project has), this could
> generate circular dependencies which cause the build to fail.

The dependency we generate is between an object file and the
headers.  The dependency lines never have .h files on the left
or .o files on the right, so there cannot be circular dependencies.

Are you actually seeing this problem with CMake?

> 3. #if 0
>    #include <nonexistantfile.h>
>    #endif
>    cmake adds a dependency on nonexistantfile.h when it shouldn't have

A simple test example shows me that CMake skips this dependency
because it cannot find the header's location.  This is true of
system headers in some cases too.

> My suggestion for a solution to this problem is to use the compiler
> specific dependency generation methods.

This is a good solution, but it should be enabled optionally
for the performance reasons I describe above.

> This could be done cleanly by converting dependency generation
> into a plugin based system with plugins for all the supported
> compilers.

Yes.  Howerver they should not be plugins in the dynamic loader sense.
We do not want to support an SDK, require users to install extra
components, or depend on the target toolchain to be capable of
building a plugin that is ABI-compatible with the running CMake.

This can probably be done with a simple command-line interface.
We just need a way to tell CMake how to invoke a third-party
tool and load dependencies from the results.  I've considered
doing something similar for implicit dependencies of rules
defined by add_custom_command, but that is much harder because
the VS IDE does not provide an easy way to implement it.  This
is not a problem for C preprocessor dependencies though.

I envision this capability being activated by a target property
which is initialized from a CMake variable, much like the
RUNTIME_OUTPUT_DIRECTORY property.  This makes it easy to
enable project-wide or in specific targets.

> I would have (tried to) made these changes to cmake already, but since
> this is both a non-trivial change and an architectural change, I felt it
> necessary to get an ok from you before I changed anything.

I'm interested in seeing a solution to this problem.  Let's
discuss the design here first though.  What interface do you
propose?

Thanks!
-Brad