[cmake-developers] Generating buildsystem metadata from CMake

[Tobias Hunger]

Hi Stephen,

On Thu, Mar 19, 2015 at 2:22 AM, Stephen Kelly <steveire at gmail.com> wrote:
>> * In "Target Properties/Conditional Properties and optional
>> properties": Would it make sense to have a list of target_files, each
>> with a filepath and a type? That would be more similar to the targets
>> which also are a list with name/type.
>
> What semantics would the 'type' have? I don't really understand what you
> propose here.

See the examples with the sources below:

>>    sources: [ {
>>        language: "C++",
>>        type: sources,
>>        defines: "FOO=1",
>>        include_paths: "/usr/include",
>>        files: [ "/full/path/a.cpp", "/full/path/b.cpp" ]
>>    },
>>    {
>>        language: "C++",
>>        type: headers,
>>        files: [ "/full/path/a.h" ]
>>    },
>>    {
>>        language: "C++",
>>        type: generated,
>>        files: [ "/full/path/config.h" ]
>>    }]

The type is used to distinguish between the different kinds of files.
Your proposal uses different keys for all the different kinds of
files, my is to use the same key for all sources and distinguish
between them by giving a type.

<snip>

>> These configurations are only relevant to generators that can support
>> e.g. having debug and release builds in the same build directory or
>> did I misunderstand this?
>
> Yes, your understanding is correct.
>
>> How are we supposed to handle a set of files for these configurations?
>
> Yes, that's how I see it working...
>
>> How can we notice when one gets added and removed?
>
> I'm not experienced enough with those generators to know. I'm not even
> certain whether it is possible to add to the list of configurations for
> those generators, or if the list is hardcoded.
>
> Something to look into or get information from someone else on this list if
> they know.
>
> I think in the worst case, you'd watch the CMakeCache.txt and read the
> CMAKE_CONFIGURATION_TYPES from there. I know you want to care about only one
> file, but maybe two is not so bad :), and there is potentially other stuff
> in the cache which might be useful, and which might not lend itself to the
> json metadata file. We'll have to see.

So we are back to parsing random files in internal formats again:-)

Could you add have a "configuration list" key in the top level of each
file that lists all available configurations (including itself)? That
might be the lowest overhead solution that does not require parsing
extra files that I can think of right now.

I definitely will not hardcode cmake generator-A has configurations X
and Y while generator-A supports configurations X, Z and D into the
cmake plugin. I do prefer not to support configurations at all to
doing that.

<snip>

>>> * Make it more easy for an IDE to support actions such as 'remove file
>>>   from the project', which requires removing it from the CMakeLists.txt
>>>   at the appropriate place, and 'add new file/class to target', which
>>>   involves adding code to the CMakeLists.txt file at the appropriate
>>>   place.  Most likely the easiest way to do the latter is using the
>>>   target_sources() command, and to support the former, the location of
>>>   the declaration of the target, and all target_sources() calls would
>>>   need to be recorded.  Even that is not enough because of transitive
>>>   consumption of source files through the link interface, but that is
>>>   likely irrelevant.
>>
>> That would be the icing on the cake, but getting a full description of
>> the project is what I care for at this point.
>
> Can you expand on what 'a full description of the project' means to you? Do
> you mean something different than 'how each object is compiled'?

When you open a project in Creator we offer to "import existing
builds" (at least for qmake-based projects, none of the other build
systems supports this yet). That is very convenient when you blew away
the .user file we put into the sources directory with all the
information on how you built stuff where.

What we do there is to check the default project build location for
folders that contain a build of the current sources.

To do so we need to know which code was built in that directory and
which kit (think set of compiler, Qt, some other settings) were used
to in the built in that directory. We also need to figure out the
exact configuration (parameters passed to the "configure" step of the
build system) that was built there.

Since this is a feature our qmake users love I would also like to make
that possible with cmake. It would be great if I could get most
answers from the json file.

>>> * I don't generate 'dependencies' (actually the list of files which the
>>>   buildsystem re-generation depends on) as Aleix did, because there is no
>>>   well-defined usefulness for that list yet.
>>
>> How is the IDE supposed to know that it needs to re-run cmake then? If
>> a file changes that may very well change the list of files that the
>> IDE needs to display, so re-running cmake only at the next build is
>> not going to work.
>
> Ok, I didn't realize that's the scenario we're talking about for that
> content. Are you saying that you would
>
>   Set up a filesystem watcher for each of the (very many) files.
>   If one of the files changes:
>     Run `cmake .` in the build dir
>   If the metadata file changes:
>     Reload the data it contains

Yes, that's the scenario.

> Is it a problem if the value of 'very many' above is in the hundreds?

I would probably trim down the list to those actually in the project
or build directories, but I would prefer having too much information
than too little:-)

In addition I will probably have a "Run cmake" option in the menu to
force the metadata to regenerate.

<snip>

>> I would love to have something like ccmake built into Qt Creator at
>> some point, and having this information would spare me poking around
>> CMakeCache.txt and whatnot:-)
>
> If you mean you would like a fully-featured cache editor in Creator, then
> reading CMakeCache.txt is unavoidable, but you'd probably fork some of the
> Source/QtDialog code as a starting point.

Is CMakeCache.txt considered to be a implementation detail of cmake,
or is it expected that external applications will parse it?

>> For that to work information on project this build directory is for as
>> well as which Qt/compiler/whatnot was used in the project is essential
>> for Creator to properly configure the code model and the rest of
>> Creator.
>
> For CMake, there's nothing particularly special about Qt - we'd probably
> also have to provide information about which Boost etc, but actually that
> information is in the CMakeCache.txt. We might be better off creating a
> stable interface to reading that file somehow. Eg
>
>  $ cmake -E read_cache Qt5Core_DIR Boost_ROOT KF5Archive_DIR ZLIB_LIBRARY
>  {
>    "Qt5Core_DIR": "/opt/qt5/...",
>    "BOOST_ROOT": "/opt/boost/1.58/...",
>    "KF5Archive_DIR": "/opt/kf5/karchive/...",
>    "ZLIB_LIBRARY": "/opt/zlib/..."
>  }

Yes, that would be nice, but once you add "cmake -E list_cache" you
could just as well just add the whole config into the json file and
let its users worry whether or not to actually read the section.

<snip>

>> Having the full configuration available would also remove the need to
>> have special flags in the "Metadata Contents" section for the compiler
>> used.
>
> I don't know what you mean. You mean '-g', '-O3' etc? Or are you referring
> to what I wrote about the pattern of the compile command line, and the order
> of where includes definitions and compile flags appear?

You documented a couple of keys in the top level that deal with the
compiler (See "Metadata Contents/Optional Properties", e.g.
"<lang>_compiler").

I will need more information than just the compiler used: The Qt
version used is obviously important to us,  as are all settings
related to cross-compilation. We have a couple of specialized Qt
Creator versions floating around that are specialized for certain
environments and those usually also need access to special flags of
the build system.

Basically I need access to the complete configuration. With the
current proposal I will need to get that information from
CMakeCache.txt (if I understood that right). If that is intended, then
I do not see the need to copy the compiler settings into the json
file. A quick google search did not turn up whether this file is
considered a implementation detail of cmake or whether it is meant to
be consumed by e.g. IDEs.

Ideally I would get away with just the json file (which would require
copying all the configuration into it), but if I am required to parse
CMakeCache.txt anyway, then I do not see the need to copy settings
that are readily available there.

>> I do not see any obvious use for a full backtrace, but then it is
>> probably interesting to have to generate the build system structure we
>> display in the projects view in Qt Creator...
>
> In the unit test, I show that things like loop controls and functions/macros
> appear in the backtraces. Those don't seem useful to the idea you describe
> here. In fact they'd probably get in the way.

Well, I am mostly interested in the tree of CMakeLists.txt files and I
think I can re-generate that from the backtrace:-)

The "Projects" view in Qt Creator is meant to visualize the structure
of the build system to allow the user to edit that. Currently I would
use the backtraces to generate that view. If dependency information
(the list of all files included by cmake into the build system) became
available that is what I would use instead.

>> Apropos project view: IIRC cmake allows to define lists of files to be
>> displayed together in visual studio, doesn't it? Would it be possible
>> to make this data available to other IDEs, too?
>
> Yes, probably. I don't have any first hand experience with what people use
> that for. The command is source_group() I think.

Maybe we could have a tree of "source objects" in the targets stead of a list?

Something along this line:

<some json leading up and into a target omitted>

sources: {
      language: "C++",
      defines: "FOO=1",

      sources: [
          {
              type: "sources",
              name: "Critical source files",
              include_paths: "/some/path",
              files: [ "/full/path/a.cpp", "/full/path/b.cpp" ]
          },
          {
             type: headers,
             files: [ "/full/path/a.h" ]
         },
         {
            type: generated,
            files: [ "/full/path/config.h" ]
         },
         {
              type: "sources",
              name: "Tests",
              include_paths: "/some/path",
              files: [ "/full/path/a.cpp", "/full/path/b.cpp",
"/full/path/test.cpp" ]
          }
      ]
}

<more json omitted>

where the "name" keys are taken from the source_group?

>>> * I propose generating language-specific compile properties in the
>>> metadata
>>>   file if there a language specific variant groups.  For example if the
>>>   compile options themselves are language-specific (the feature enabling
>>>   that was merged to CMake master this week), then a property for each
>>>   language
>>>   is specified.  Otherwise only a language-agnostic property is
>>>   generated. This can vary per-configuration for a multi-config
>>>   generator, so each metadata file for a multi-config generator could
>>>   have different properties present. Consumers read the language specific
>>>   property if it is present, and read the language agnostic version
>>>   otherwise.
>>
>> I do not fully get where you are going here.
>
> If you are dealing with a cxx file and there is a "cxx_compile_options"
> property, then you read that and you're done. Otherwise, you read
> "compile_options".

I agree with Anton: This will required all IDEs to duplicate the logic
for mapping a file to a type 'X' and then get the settings for
'X_compile_options' and fall back to 'compile_options' if that is
unset.

Can't we go for a syntax where the files and everything that applies
to them are grouped so that there is no need to reimplement the logic
found in cmake (with added bugs:-)?

>>> * Generating metadata only (without generating buildsystem files) is not
>>>   currently in scope.  This was requested several times, but it is not
>>>   clear why.
>>
>> Not sure... When opening a CMakeLists.txt, ideally I would ask the
>> user for a build directory and a cmake generator to be used and run
>> cmake once.
>
> Yes, so far this is possible, and doesn't carry a requirement to not
> generate an actual buildsystem...

I agree with Anton that it would be nice not to have to ask for a
build directory and generator first thing. Many users just want to
browse some project and an IDE should support that use case as well as
possible. Nagging those users about a build directory or generator is
not the best user experience.

>> Afterwards I would have a project description that I can generate a
>> ccmake-like config UI from. If the user changes that I run cmake again
>> and afterwards the project is fully set up and documented in the --
>> now updated -- project description file.
>>
>> Afterwards I will need to regenerate the configuration description
>> each time something changed. That should be as fast as possible, as
>> the data I display in the UI is stale at that point. If not generating
>> the complete build-system makes this step faster, then I am all for
>> that:-) OTOH, it probably does not matter too much if the description
>> is created first and the build system is updated only afterwards in
>> the background.
>
> Yes, if speed is the concern, then we'll have to see what the gains are of
> not generating the buildsystem (probably much more significant on Windows
> actually).
>
>> Unrelated question: Can I query the available generators in a
>> machine-readable format with user friendly names and their name from
>> cmake somehow?
>
> Not yet. We can add that though with something like
>
>  $ cmake -E list_generators
>  {
>    "generators": ["Unix Makefiles", "Ninja", "Xcode"]
>  }
>
> etc.

That would be *really* nice to have once this metadata file is in place:-)

<snip>

>> Creator needs to know where all files will end up on install since we
>> might need to copy them to a remote machine (e.g. a phone, or a Linux
>> box) to run/debug there. Installing everything with some temporary
>> directory prefixed to all paths does work, though. We can then rsync
>> the temporary directory to a device. But even then we will still need
>> to know where the executables end up on the device to run/debug.
>
> The CMAKE_STAGING_PREFIX is designed for this purpose. You would specify it
> when running cmake and copy the files from there.
>
>  http://www.cmake.org/cmake/help/git-next/variable/CMAKE_STAGING_PREFIX.html
>  http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/8363/focus=8629
>
> It's basically the same as extprefix in the Qt configure script.

I am aware of that:-)

Creator would still need to know where the files went that the user
wants to run/debug though. So at least for the binaries a way to map
them from the build to the install location is needed for our
use-case.

Best Regards,
Tobias