[CMake] CMake 3.9 change to dependencies of object compilation

[Robert Maynard]

You can find the location for object files by using
$<TARGET_OBJECTS:${name}>. This can be used as the DEPENDS of a custom
command. I would not try using OBJECT_OUTPUTS as IIRC that is only
used by the makefile generator.

On Fri, Aug 4, 2017 at 1:55 PM, Ben Boeckel <ben.boeckel at kitware.com> wrote:
> On Fri, Aug 04, 2017 at 17:35:53 +0000, Puetz Kevin A wrote:
>> Thanks for the reply, questions/clarifications below.
>> > requests that that be fixed as well (though that is backwards compatible
>> > since the solution will likely involve ).
>>
>> Sentence cut off? I assume you meant "will likely involve a new keyword"?
>
> Yep, sorry.
>
>> > > 2. MSVC's #import construct which needs the indirect dependencies
>> > > (dependencies of the #import-ed dependency) be registered, which is
>> > > handled as part of the target using add_custom_command(TARGET foo
>> > > POST_BUILD COMMAND ...)
>> >
>> > So there's an issue here that there's a dependency between your build rules
>> > which CMake doesn't know about (though I don't know #import well
>> > enough, the docs don't state where the information *goes*).
>>
>> #import will load a COM typelib during preprocessing, possibly
>> following registry keys to locate other typelibs which the specified
>> one refers to. It will have the byproduct of creating .tlh/.tli files
>> next to the other compiler outputs (e.g. .o file) Arguably the
>> .tlh/.tli files should be listed in OBJECT_OUTPUTS, but I can't
>> because I don't know their location; CMake doesn't have a
>> variable/property/generator expression that reveals where it's going
>> to place the object files (i.e. /Fo$out), so I don't know where they
>> will end up. Luckily the .tlh/.tli files aren't important to list for
>> dependency resolution anyway, because the #import also automatically
>> #includes the just-generated headers, (though this is not mentioned in
>> /showIncludes). So CMake is at least *consistently* unaware of these
>> files, and they get regenerated any time they would have been read so
>> it doesn't really need to know.
>
> OK, a genex for where object outputs may be useful anyways. I think
> there's something along those lines with Cuda's PTX file generation?
>
>> The important missing dependency is the one between
>> creating/regstering the typelib (we'll call this target COMServer) and
>> the #import that will read it in a source file in another target
>> (we'll call it COMClient).  I have a call add_custom_command(TARGET
>> COMServer POST_BUILD COMMAND regsvr32 $<TARGET_FILE: COMServer>),
>> which will create the registry keys under HKEY_CLASSES_ROOT. This
>> needs to happen before the source file in COMClient can preprocess the
>> #import successfully. Prior to CMake 3.9, I could inform CMake of this
>> by just using add_dependencies(COMClient COMServer) to tell CMake that
>> it couldn't build (any of) Client until Server had been built (and
>> thus its POST_BUILD had also run to register it). But in 3.9,
>> add_dependencies has changed in meaning; although the documentation
>> still says "to ensure that they build before <target> does", in
>> practice this now only means "to ensure that they build before
>> <target> *links*"; these edges do not apply to object compilation or
>> add_custom_command rules.
>>
>> add_custom_command is no problem; it already had a DEPENDS argument
>> that allows target-level dependencies, and arguably such dependencies
>> needed to be stated there anyway since an add_custom_command output
>> can get reused by multiple targets in the same subdir. But object
>> compilation is a problem because there's nowhere to add them
>> per-source, and add_dependencies doesn't work anymore to add them
>> per-target.
>
> It sounds like the logic may need fixing then. Do you have an example
> case where add_dependencies doesn't work anymore in Ninja?
>
>> > When adding
>> > this custom command, you may use the `BYPRODUCTS` argument
>> > (introduced in 3.2.0) to let CMake know what's going on here. It only affects
>> > Ninja, but the other generators do target-level dependencies anyways. That
>> > output can then be depended on via `OBJECT_DEPENDS` and the
>> > dependency should link up properly.
>>
>> There is not an explicit file output, though I could do the usual
>> workaround of a stamp/witness file listed in BYPRODUCTS to the
>> add_custom_command(TARGET ... POST_BUILD ...). But I don't think that
>> will work with most generators, since CMake doesn't generally allow
>> file-level depends to set the order in which targets are built. I
>> suppose it might work out in practice for ninja since that writes a
>> monolithic set of rules, but conditional code where I have to peek at
>> CMAKE_GENERATOR and use BYPRODUCTS/OBJECT_DEPENDS for ninja and
>> add_dependencies for other generators seems like the sort of thing
>> this list would tell me not to do :-)
>
> Well, other generators are generally target-ordered anyways. Ninja is
> the oddball here (which is why it's the only one to get the feature). I
> don't know the effect it'd have in other generators, but I feel like I'd
> be surprised if it *broke* them since excess dependencies (usually) only
> result in either slower builds or circular dependency loops and Ninja
> complains loudly about the latter. And since BYPRODUCTS only affects
> Ninja, if BYPRODUCTS is used, other generators shouldn't care anyways.
>
>> And even for ninja I think I'd have to be making undocumented
>> assumptions about the binary dir layout to refer to my witness file
>> that was generated in a different subdir's CMakelists.txt.
>
> There's nothing stopping the witness files couldn't all be under a
> single directory (such as ${CMAKE_BINARY_DIR}/CMakeFiles/tlb).
>
>> > If it instead gets registered somewhere in the aether (as far as CMake is
>> > concerned), adding support for generator expressions to `OBJECT_DEPENDS`
>> > so that `$<TARGET_FILE:tgt>` may be used there would be the next solution.
>>
>> Yes, the dependency in question for #import is on information
>> "somewhere in the aether" (or rather the Win32 registry).
>>
>> Supporting $<TARGET_FILE> does in OBJECT_DEPENDS would be a great
>> solution for my first use case of a embedding that file in a resource.
>> But I don't think that helps with #import, since I don't actually want
>> to read the $<TARGET_FILE>, I just want the post-build that registers
>> it to have run.
>
> POST_BUILD rules are attached to the target, so depending on the target
> also guarantees that the POST_BUILD command(s) have run as well.
>
>> Also, in the cases of .tlb files that are *not* embedded in DLL
>> resources, the target in question is going to be an add_custom_target
>> from another subdirectory; the .tlb file is built by an
>> add_custom_command(OUTPUT...) but this rule gets emitted in an
>> add_custom_target that depends on this file to build it and then
>> registers it. If each subdir had the add_custom_command instead of
>> using an intermediate target, multiple targets would each end up with
>> their own copy of the rule to build the .tlb file, leading to race
>> conditions where they all try to build it at once and get file-in-use
>> errors (they can't just build individual copies, because it has to end
>> up with a unique key referencing the .tlb path in the win32 regist>
>> ry).
>
> Yeah, there should be just one .tlb rule writer. Usually I handle that
> by collecting information in global properties and writing a rule at the
> end to handle all of them.
>
>> You're not currently allowed to use $<TARGET_FILE:x> on UTILITY
>> targets even if the LOCATION property has been set (it's blocked in
>> TargetFilesystemArtifact::Evaluate with "Target <x> is not an
>> executable or library"). Maybe that could be changed as well (which
>> would be nice), but it seems like if one is adding support for
>> $<TARGET_FILE> generator expressions in OBJECT_DEPENDS (which implies
>> supporting the generator context and context->DependTargets), it seems
>> like you as well go the rest of the way and just treat them completely
>> the same as the DEPENDS argument to add_custom_command, allowing both
>> file and target dependencies to be listed in the first place.
>
> That sounds like a likely path to follow when supporting genexes in
> OBJECT_DEPENDS.
>
>> > Making `POST_BUILD` write out a stamp file would also work and then using
>> > `OBJECT_DEPENDS` on that would also work.
>>
>> No, as above I don't think that would be legal across subdirs, at
>> least in the context of a CMake file that's supposed to work with
>> various generators. Feel free to correct me if I'm wrong about that...
>
> Experiments would be more useful. add_custom_* have some of the most
> complicated interaction semantics in CMake. I can't keep all of them
> straight all the time (usually I rediscover them when necessary; I
> should probably write up some docs next time I need to do so).
>
> --Ben