[cmake-developers] Generating buildsystem metadata from CMake

[Tobias Hunger]

Hi Stephen,

sorry for being late to the party again:-)

I am just reading your documentation.

Some nits:

* In the "Introduction" you say CMAKE_GENERATE_METADATA needs to be
ON, in "Generating Metadata" it needs to be set to a version number.

* In "Metadata Contents/Optional Properties" you propose to have
several language specific properties. Would it make sense to group
those together into one "language" property?

* In "Target Properties/Conditional Properties", "backtrace": Which
frame is the current one? The top one or the bottom one? I've seen
both approaches in the wild:-)

* In "Target Properties/Conditional Properties and optional
properties": Would it make sense to have a list of target_files, each
with a filepath and a type? That would be more similar to the targets
which also are a list with name/type. Are these paths in the build or
in the install location?

* In "Target Properties/Conditional Optional Properties": Again,
wouldn't a set of sources arrays with a list of filepaths and all the
related meta information for this set of files be nicer? That way all
the "language fallback properties" and "source file properties" would
be in one place.

  Somewhat like this (going with JSON-ish syntax here for ease of writing):

   sources: [ {
       language: "C++",
       type: sources,
       defines: "FOO=1",
       include_paths: "/usr/include",
       files: [ "/full/path/a.cpp", "/full/path/b.cpp" ]
   },
   {
       language: "C++",
       type: headers,
       files: [ "/full/path/a.h" ]
   },
   {
       language: "C++",
       type: generated,
       files: [ "/full/path/config.h" ]
   }]

  This can express defines set for individual files, which I think you
can have in CMake, can't you? I do not see how to express that within
your proposal.

The rest of my replies are inline:

On Wed, Mar 11, 2015 at 11:10 AM, Stephen Kelly <steveire at gmail.com> wrote:
<snip>
> I expect to require a few iterations to figure out what the metadata files
> should contain in the end.  Note that there are already some differences
> between my design and Aleix's implementation, such as that my design
> proposes one metadata file per config. There are also some things
> missing like location, because it is not yet clear to me whether build
> or install locations are needed etc.

These configurations are only relevant to generators that can support
e.g. having debug and release builds in the same build directory or
did I misunderstand this?

How are we supposed to handle a set of files for these configurations?
How can we notice when one gets added and removed?

Having everything in one file would probably be easier, as that is
just one file what needs watching:-) Could we maybe just tag the
targets with the configurations they apply to and maybe also add the
same tag to the source file groups proposed above?

> The content of the metadata file is determined by the build properties, and
> is necessarily similar to the compile-related content created when
> generating the actual buildsystem.  It additionally contains information
> about the output locations of build artifacts and information relating to
> the cmake description itself.

Output location of build artifacts: That is the location inside the
build directory, isn't it? What about install locations?

> Goals include:
>
> * Make it possible for IDEs to access the compile-related information for
>   autocompletion and code navigation etc purposes.
>
> * Remove the need for IDEs to parse generated Makefiles or Ninja files to
>   access compile-related information.  The structure of those files is not
>   'stable', while the content of the metadata file is stable.
>     http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/48412
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11081

YES!

> * Remove the need for users to create a new build directory and new build
>   in order to use or switch IDEs.  QtCreator requires that
>   the C::B 'extra generator is used as it parses compile information from
>   that.  Other 'extra generators' such as for eclipse, sublime, kate etc
>   also require fresh/new build directories, although the actual buildsystem
>   they create is identical (assuming using all Makefile based or
>   all Ninja based 'extra generators')

That would indeed be a nice side-effect:-)

> * Make it possible for editors/IDEs to allow specifying the configuration
>   at build-time, where the IDE has that feature, and where a multi-config
>   generator is available.  That is, QtCreator provides user-interface for
>   choosing debug/release config to build.  Currently it can't offer that
>   when using cmake, because it only allows the use of Makefile or Ninja
>   generators, in order to make use of the C::B file.  QtCreator should be
>   able to use the Xcode or Visual Studio generators, generate the metadata
>   file(s), and invoke `cmake --build . --config ${CONFIG}` as
>   appropriate.  Eclipse, Sublime and other editors have similar abilities
>   to invoke config-specific builds after generation.

Yes again:-)

> * Provide a list of 'build targets', which can be listed and invoked in
>   IDE/editor user interface.  Build targets for all linked binaries
>   and utilties are provided.  The tooling is expected to perform filtering
>   on the target types to show only executables and utilities for
>   execution, for example.

OK.

> * Provide a list of source files per target per type of source file, eg
>   object sources, header files, generated files, files excluded from the
>   active configuration/platform/compiler, non-compiled files.

Nice.

> * Make it more easy for an IDE to support actions such as 'remove file
>   from the project', which requires removing it from the CMakeLists.txt
>   at the appropriate place, and 'add new file/class to target', which
>   involves adding code to the CMakeLists.txt file at the appropriate
>   place.  Most likely the easiest way to do the latter is using the
>   target_sources() command, and to support the former, the location of
>   the declaration of the target, and all target_sources() calls would
>   need to be recorded.  Even that is not enough because of transitive
>   consumption of source files through the link interface, but that is
>   likely irrelevant.

That would be the icing on the cake, but getting a full description of
the project is what I care for at this point.

> * Provide information about the entire build graph of link-dependencies
>   for visulization and dependency analysis
>     http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/4042

Nice.

> * Provide enough information about runtime link dependencies for IDEs to
>   be able to properly invoke targets with debuggers, profilers and other
>   tools.
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11011

Yes.

> The CMAKE_EXPORT_COMPILE_COMMANDS variable already exists to generate
> compile command lines, but that is a file whose format is defined by
> Clang, not by CMake, and other tools in the Clang ecosystem also support
> it.  It is not extensible or versioned in the way that a CMake-defined
> file format is, so I don't propose using it as a starting point.

Fully agree with this:-)

<snip>
> * In the documentation and the test I didn't mention that the metadata file
>   is JSON.  I'm not opposed to it, but I think there should be some kind of
>   schema defining how to read the metadata file.  If we go with JSON
> (instead of xml, say), then we'd have to define our own schema format, which
>   is not impossible.  My proposal is to generate the schema beside the metadata
>   file instead of writing a metadata version into the metadata file
> itself.  Does
>   anyone feel strongly about the file being JSON?  I just want to record the
>   reason for everything in this design phase - we don't have to spend a lot
>   of time on it.

I do not care too much on the format, but I am strongly in favor of
anything that I can conveniently parse with Qt. So XML and JSON are
thus both fine with me:-)

I would prefer JSON, simply because I find that way more human-readable.

> * I didn't document the location or directory.  I'm not clear on whether
>   it is supposed to be the build location, or the install location(s!),
>   or all of those.

I see four options to place this file:

* Source directory: Nah, let's not pollute the source dir more than necessary.

* Build directory: Probably a good place as it contains per-build
information if I understood this correctly.

* Install directory: Probably not around by the time you start a build
and I do not see any reason to have IDE support for installed files.

* Some configuration directory:
~/.config/cmake/builds/whatever.config. That would make it simple to
find all builds of a project, but will make it hard to keep the
configuration in sync with the build directories (e.g. when somebody
removes build directories).

The only practical place seems the build directory to me.

> * I don't generate 'dependencies' (actually the list of files which the
>   buildsystem re-generation depends on) as Aleix did, because there is no
>   well-defined usefulness for that list yet.

How is the IDE supposed to know that it needs to re-run cmake then? If
a file changes that may very well change the list of files that the
IDE needs to display, so re-running cmake only at the next build is
not going to work.

> * I documented the 'name' for the targets, and the TARGET_FILE_NAME
>   etc information, and can push an implementation which generates them.
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12619

Great.

> * If it is useful to preprocess/compile/assemble individual files from
>   IDEs, as made possible by the Makefiles and Ninja generators, we'll need
>   to design that.
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12429

That would be nice to have, but let's get the basics done first.

At the same time being able to build only the src/ directory, etc.
would be useful, too.

> * Some more information from project() may be relevant, but it's not clear
>   yet. We will likely know more when we have decided the file format and
>   generated some 'interesting' metadata files.

That is definitely needed! What is this file about(full path to top
level CMakeLists.txt that was used to generate this build directory as
well as the configuration that  was used!) is essential to answer:-)

I would love to have something like ccmake built into Qt Creator at
some point, and having this information would spare me poking around
CMakeCache.txt and whatnot:-)

Creator also offers to import existing builds and with your stated
goal of standardizing on this one file for several IDEs this import
features becomes even more important.

For that to work information on project this build directory is for as
well as which Qt/compiler/whatnot was used in the project is essential
for Creator to properly configure the code model and the rest of
Creator.

Having the full configuration available would also remove the need to
have special flags in the "Metadata Contents" section for the compiler
used.

> * We might need backtraces for not only all add_library and add_executable
>   calls, but also all target_sources() calls.

I do not see any obvious use for a full backtrace, but then it is
probably interesting to have to generate the build system structure we
display in the projects view in Qt Creator... Please keep it:-)

Apropos project view: IIRC cmake allows to define lists of files to be
displayed together in visual studio, doesn't it? Would it be possible
to make this data available to other IDEs, too?

> * We list the Generator used to configure the build.  That way IDEs know
>   which build output parser to use.

Good!

> * I propose to generate one metadata file per configuration active in the
>   buildsystem.  That means that the Makefile and Ninja generators would
>   generate only one file, and the multi-config generators would generate
>   multiple files.  When changing the 'active configuration', IDEs would
>   then read whatever metadata file is relevant to the active configuration.

How can I get the list of files that are relevant? Will there be one
master file that lists all the available configurations and their
configuration description file?

>   This also means that conditions don't need to be added inside the metadata
>   file for configurations in order to show files 'excluded' by being part of
>   a non-active configuration.  The code implementing discovery of excluded
>   files is in the generate-metadata branch in my clone.
>
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11323
>     "At least Qt Creator does not need information on which conditions to
>      be met for a file to become part of the current build."

OK.

> * I propose generating language-specific source lists, because CMake can
>   build files with compilers which do not match their file extension.

Makes a lot of sense, even though I am not too happy with the syntax.
See the nits above.

> * I propose generating language-specific compile properties in the metadata
>   file if there a language specific variant groups.  For example if the
>   compile options themselves are language-specific (the feature enabling that
>   was merged to CMake master this week), then a property for each language
>   is specified.  Otherwise only a language-agnostic property is generated. This
>   can vary per-configuration for a multi-config generator, so each metadata
>   file for a multi-config generator could have different properties present.
>   Consumers read the language specific property if it is present, and read
>   the language agnostic version otherwise.

I do not fully get where you are going here.

> * Generating metadata only (without generating buildsystem files) is not
>   currently in scope.  This was requested several times, but it is not
>   clear why.

Not sure... When opening a CMakeLists.txt, ideally I would ask the
user for a build directory and a cmake generator to be used and run
cmake once.

Afterwards I would have a project description that I can generate a
ccmake-like config UI from. If the user changes that I run cmake again
and afterwards the project is fully set up and documented in the --
now updated -- project description file.

Afterwards I will need to regenerate the configuration description
each time something changed. That should be as fast as possible, as
the data I display in the UI is stale at that point. If not generating
the complete build-system makes this step faster, then I am all for
that:-) OTOH, it probably does not matter too much if the description
is created first and the build system is updated only afterwards in
the background.

Unrelated question: Can I query the available generators in a
machine-readable format with user friendly names and their name from
cmake somehow?

> * How much information does tooling need about installation?  Targets
>   can use different include directories and compile definitions in their
>   install locations compared to their build locations.  If IDEs want to
>   provide some user interface related to the project files in their
>   install location, perhaps a separate solution based on cmExportFile*
>   is needed.  For future investigation.

Why is that necessary?

My goal is to have solid support for stand-alone cmake-based projects
in Creator. If such projects reference installed libraries I would
expect that to be reflected in the compiler and linker flags. Why
would that need special support for include directories and compile
definitions?

Creator needs to know where all files will end up on install since we
might need to copy them to a remote machine (e.g. a phone, or a Linux
box) to run/debug there. Installing everything with some temporary
directory prefixed to all paths does work, though. We can then rsync
the temporary directory to a device. But even then we will still need
to know where the executables end up on the device to run/debug.

Best Regards,
Tobias

PS: I hope there is not too much redundant stuff in here that is
already discussed in the follow up mails I only skimmed over so far:-)