[CMake] CMake 3.9 change to dependencies of object compilation

[Ben Boeckel]

On Fri, Aug 04, 2017 at 17:35:53 +0000, Puetz Kevin A wrote:
> Thanks for the reply, questions/clarifications below.
> > requests that that be fixed as well (though that is backwards compatible
> > since the solution will likely involve ).
> 
> Sentence cut off? I assume you meant "will likely involve a new keyword"?

Yep, sorry.

> > > 2. MSVC's #import construct which needs the indirect dependencies
> > > (dependencies of the #import-ed dependency) be registered, which is
> > > handled as part of the target using add_custom_command(TARGET foo
> > > POST_BUILD COMMAND ...)
> > 
> > So there's an issue here that there's a dependency between your build rules
> > which CMake doesn't know about (though I don't know #import well
> > enough, the docs don't state where the information *goes*).
> 
> #import will load a COM typelib during preprocessing, possibly
> following registry keys to locate other typelibs which the specified
> one refers to. It will have the byproduct of creating .tlh/.tli files
> next to the other compiler outputs (e.g. .o file) Arguably the
> .tlh/.tli files should be listed in OBJECT_OUTPUTS, but I can't
> because I don't know their location; CMake doesn't have a
> variable/property/generator expression that reveals where it's going
> to place the object files (i.e. /Fo$out), so I don't know where they
> will end up. Luckily the .tlh/.tli files aren't important to list for
> dependency resolution anyway, because the #import also automatically
> #includes the just-generated headers, (though this is not mentioned in
> /showIncludes). So CMake is at least *consistently* unaware of these
> files, and they get regenerated any time they would have been read so
> it doesn't really need to know.

OK, a genex for where object outputs may be useful anyways. I think
there's something along those lines with Cuda's PTX file generation?

> The important missing dependency is the one between
> creating/regstering the typelib (we'll call this target COMServer) and
> the #import that will read it in a source file in another target
> (we'll call it COMClient).  I have a call add_custom_command(TARGET
> COMServer POST_BUILD COMMAND regsvr32 $<TARGET_FILE: COMServer>),
> which will create the registry keys under HKEY_CLASSES_ROOT. This
> needs to happen before the source file in COMClient can preprocess the
> #import successfully. Prior to CMake 3.9, I could inform CMake of this
> by just using add_dependencies(COMClient COMServer) to tell CMake that
> it couldn't build (any of) Client until Server had been built (and
> thus its POST_BUILD had also run to register it). But in 3.9,
> add_dependencies has changed in meaning; although the documentation
> still says "to ensure that they build before <target> does", in
> practice this now only means "to ensure that they build before
> <target> *links*"; these edges do not apply to object compilation or
> add_custom_command rules.
> 
> add_custom_command is no problem; it already had a DEPENDS argument
> that allows target-level dependencies, and arguably such dependencies
> needed to be stated there anyway since an add_custom_command output
> can get reused by multiple targets in the same subdir. But object
> compilation is a problem because there's nowhere to add them
> per-source, and add_dependencies doesn't work anymore to add them
> per-target.

It sounds like the logic may need fixing then. Do you have an example
case where add_dependencies doesn't work anymore in Ninja?

> > When adding
> > this custom command, you may use the `BYPRODUCTS` argument
> > (introduced in 3.2.0) to let CMake know what's going on here. It only affects
> > Ninja, but the other generators do target-level dependencies anyways. That
> > output can then be depended on via `OBJECT_DEPENDS` and the
> > dependency should link up properly.
> 
> There is not an explicit file output, though I could do the usual
> workaround of a stamp/witness file listed in BYPRODUCTS to the
> add_custom_command(TARGET ... POST_BUILD ...). But I don't think that
> will work with most generators, since CMake doesn't generally allow
> file-level depends to set the order in which targets are built. I
> suppose it might work out in practice for ninja since that writes a
> monolithic set of rules, but conditional code where I have to peek at
> CMAKE_GENERATOR and use BYPRODUCTS/OBJECT_DEPENDS for ninja and
> add_dependencies for other generators seems like the sort of thing
> this list would tell me not to do :-)

Well, other generators are generally target-ordered anyways. Ninja is
the oddball here (which is why it's the only one to get the feature). I
don't know the effect it'd have in other generators, but I feel like I'd
be surprised if it *broke* them since excess dependencies (usually) only
result in either slower builds or circular dependency loops and Ninja
complains loudly about the latter. And since BYPRODUCTS only affects
Ninja, if BYPRODUCTS is used, other generators shouldn't care anyways.

> And even for ninja I think I'd have to be making undocumented
> assumptions about the binary dir layout to refer to my witness file
> that was generated in a different subdir's CMakelists.txt.

There's nothing stopping the witness files couldn't all be under a
single directory (such as ${CMAKE_BINARY_DIR}/CMakeFiles/tlb).

> > If it instead gets registered somewhere in the aether (as far as CMake is
> > concerned), adding support for generator expressions to `OBJECT_DEPENDS`
> > so that `$<TARGET_FILE:tgt>` may be used there would be the next solution.
> 
> Yes, the dependency in question for #import is on information
> "somewhere in the aether" (or rather the Win32 registry).
> 
> Supporting $<TARGET_FILE> does in OBJECT_DEPENDS would be a great
> solution for my first use case of a embedding that file in a resource.
> But I don't think that helps with #import, since I don't actually want
> to read the $<TARGET_FILE>, I just want the post-build that registers
> it to have run.

POST_BUILD rules are attached to the target, so depending on the target
also guarantees that the POST_BUILD command(s) have run as well.

> Also, in the cases of .tlb files that are *not* embedded in DLL
> resources, the target in question is going to be an add_custom_target
> from another subdirectory; the .tlb file is built by an
> add_custom_command(OUTPUT...) but this rule gets emitted in an
> add_custom_target that depends on this file to build it and then
> registers it. If each subdir had the add_custom_command instead of
> using an intermediate target, multiple targets would each end up with
> their own copy of the rule to build the .tlb file, leading to race
> conditions where they all try to build it at once and get file-in-use
> errors (they can't just build individual copies, because it has to end
> up with a unique key referencing the .tlb path in the win32 regist>
> ry). 

Yeah, there should be just one .tlb rule writer. Usually I handle that
by collecting information in global properties and writing a rule at the
end to handle all of them.

> You're not currently allowed to use $<TARGET_FILE:x> on UTILITY
> targets even if the LOCATION property has been set (it's blocked in
> TargetFilesystemArtifact::Evaluate with "Target <x> is not an
> executable or library"). Maybe that could be changed as well (which
> would be nice), but it seems like if one is adding support for
> $<TARGET_FILE> generator expressions in OBJECT_DEPENDS (which implies
> supporting the generator context and context->DependTargets), it seems
> like you as well go the rest of the way and just treat them completely
> the same as the DEPENDS argument to add_custom_command, allowing both
> file and target dependencies to be listed in the first place.

That sounds like a likely path to follow when supporting genexes in
OBJECT_DEPENDS.

> > Making `POST_BUILD` write out a stamp file would also work and then using
> > `OBJECT_DEPENDS` on that would also work.
> 
> No, as above I don't think that would be legal across subdirs, at
> least in the context of a CMake file that's supposed to work with
> various generators. Feel free to correct me if I'm wrong about that...

Experiments would be more useful. add_custom_* have some of the most
complicated interaction semantics in CMake. I can't keep all of them
straight all the time (usually I rediscover them when necessary; I
should probably write up some docs next time I need to do so).

--Ben