[CMake] CMake 3.9 change to dependencies of object compilation

[Puetz Kevin A]

Thanks for the reply, questions/clarifications below.

> -----Original Message-----
> From: Ben Boeckel [mailto:ben.boeckel at kitware.com]
> Sent: Thursday, August 03, 2017 10:59 AM
> To: cmake at cmake.org
> Cc: Robert Maynard <robert.maynard at kitware.com>; Puetz Kevin A
> <PuetzKevinA at JohnDeere.com>
> Subject: Re: CMake 3.9 change to dependencies of object compilation
> 
> > . The Ninja generator has loosened the dependencies of object
> > compilation. Object compilation now depends only on custom targets and
> > custom commands associated with libraries on which the object's target
> > depends and no longer depends on the libraries themselves. Source
> > files in dependent targets may now compile without waiting for their
> > targets' dependencies to link.
> 
> Correct.
> 
> > We have a few cases where the object compilation really does depend on
> > the TARGET_FILE itself, e.g.
> > 1. An RC compiler embedding the resulting file of a custom target (I
> > think this one may still work, since custom targets appear to have
> > been exempted from the change)
> 
> Correct, though this issue:
> 
>     https://gitlab.kitware.com/cmake/cmake/issues/17097
> 
> requests that that be fixed as well (though that is backwards compatible
> since the solution will likely involve ).

Sentence cut off? I assume you meant "will likely involve a new keyword"?

> > 2. MSVC's #import construct which needs the indirect dependencies
> > (dependencies of the #import-ed dependency) be registered, which is
> > handled as part of the target using add_custom_command(TARGET foo
> > POST_BUILD COMMAND ...)
> 
> So there's an issue here that there's a dependency between your build rules
> which CMake doesn't know about (though I don't know #import well
> enough, the docs don't state where the information *goes*).

#import will load a COM typelib during preprocessing, possibly following registry keys to locate other typelibs which the specified one refers to. It will have the byproduct of creating .tlh/.tli files next to the other compiler outputs (e.g. .o file) Arguably the .tlh/.tli files should be listed in OBJECT_OUTPUTS, but I can't because I don't know their location; CMake doesn't have a variable/property/generator expression that reveals where it's going to place the object files (i.e. /Fo$out), so I don't know where they will end up. Luckily the .tlh/.tli files aren't important to list for dependency resolution anyway, because the #import also automatically #includes the just-generated headers, (though this is not mentioned in /showIncludes). So CMake is at least *consistently* unaware of these files, and they get regenerated any time they would have been read so it doesn't really need to know.

The important missing dependency is the one between creating/regstering the typelib (we'll call this target COMServer) and the #import that will read it in a source file in another target (we'll call it COMClient).  I have a call add_custom_command(TARGET COMServer POST_BUILD COMMAND regsvr32 $<TARGET_FILE: COMServer>), which will create the registry keys under HKEY_CLASSES_ROOT. This needs to happen before the source file in COMClient can preprocess the #import successfully. Prior to CMake 3.9, I could inform CMake of this by just using add_dependencies(COMClient COMServer) to tell CMake that it couldn't build (any of) Client until Server had been built (and thus its POST_BUILD had also run to register it). But in 3.9, add_dependencies has changed in meaning; although the documentation still says "to ensure that they build before <target> does", in practice this now only means "to ensure that they build before <target> *links*"; these edges do not apply to object compilation or add_custom_command rules.

add_custom_command is no problem; it already had a DEPENDS argument that allows target-level dependencies, and arguably such dependencies needed to be stated there anyway since an add_custom_command output can get reused by multiple targets in the same subdir. But object compilation is a problem because there's nowhere to add them per-source, and add_dependencies doesn't work anymore to add them per-target.

> When adding
> this custom command, you may use the `BYPRODUCTS` argument
> (introduced in 3.2.0) to let CMake know what's going on here. It only affects
> Ninja, but the other generators do target-level dependencies anyways. That
> output can then be depended on via `OBJECT_DEPENDS` and the
> dependency should link up properly.

There is not an explicit file output, though I could do the usual workaround of a stamp/witness file listed in BYPRODUCTS to the add_custom_command(TARGET ... POST_BUILD ...). But I don't think that will work with most generators, since CMake doesn't generally allow file-level depends to set the order in which targets are built. I suppose it might work out in practice for ninja since that writes a monolithic set of rules, but conditional code where I have to peek at CMAKE_GENERATOR and use BYPRODUCTS/OBJECT_DEPENDS for ninja and add_dependencies for other generators seems like the sort of thing this list would tell me not to do :-)

And even for ninja I think I'd have to be making undocumented assumptions about the binary dir layout to refer to my witness file that was generated in a different subdir's CMakelists.txt.

> If it instead gets registered somewhere in the aether (as far as CMake is
> concerned), adding support for generator expressions to `OBJECT_DEPENDS`
> so that `$<TARGET_FILE:tgt>` may be used there would be the next solution.

Yes, the dependency in question for #import is on information "somewhere in the aether" (or rather the Win32 registry).

Supporting $<TARGET_FILE> does in OBJECT_DEPENDS would be a great solution for my first use case of a embedding that file in a resource. But I don't think that helps with #import, since I don't actually want to read the $<TARGET_FILE>, I just want the post-build that registers it to have run. Also, in the cases of .tlb files that are *not* embedded in DLL resources, the target in question is going to be an add_custom_target from another subdirectory; the .tlb file is built by an add_custom_command(OUTPUT...) but this rule gets emitted in an add_custom_target that depends on this file to build it and then registers it. If each subdir had the add_custom_command instead of using an intermediate target, multiple targets would each end up with their own copy of the rule to build the .tlb file, leading to race conditions where they all try to build it at once and get file-in-use errors (they can't just build individual copies, because it has to end up with a unique key referencing the .tlb path in the win32 registry). 

You're not currently allowed to use $<TARGET_FILE:x> on UTILITY targets even if the LOCATION property has been set (it's blocked in TargetFilesystemArtifact::Evaluate with "Target <x> is not an executable or library"). Maybe that could be changed as well (which would be nice), but it seems like if one is adding support for $<TARGET_FILE> generator expressions in OBJECT_DEPENDS (which implies supporting the generator context and context->DependTargets), it seems like you as well go the rest of the way and just treat them completely the same as the DEPENDS argument to add_custom_command, allowing both file and target dependencies to be listed in the first place.

> Making `POST_BUILD` write out a stamp file would also work and then using
> `OBJECT_DEPENDS` on that would also work.

No, as above I don't think that would be legal across subdirs, at least in the context of a CMake file that's supposed to work with various generators. Feel free to correct me if I'm wrong about that...

> --Ben