[CMake] avoid rebuilding targets depending on generated source files

[Michael Hertling]

On 03/05/2012 01:59 AM, Ajay Panyala wrote:
> I use cmake version 2.8.3.
> 
> If I use CMakeLists from your previous reply, it avoids overwriting files
> when X=0.
> I have attached the output of your script.

Actually, this is exactly what I was talking about: The second "make
X=0" invocation rewrites the source files - i.e., they receive a new
timestamp - but their contents do not change. Thus, the destination
files in the parser directory are *not* touched by the second "make
X=0", and this can be seen clearly by reference to the timestamps.
It's the subsequent "make X=1" which provides for a new content of
the source files and makes "cmake -E copy_if_different" copy the
sources to the destinations. That's why I said everything works
as expected, but you wrote:

>>> It does not work. The files are still overwritten.

So, my question again: What exactly does not work?

I.e., does "cmake -E copy_if_different"

- copy a file although the destination exists
  and has the same content as the source, or

- not copy a file although the destination does not
  exist or does not have the same content as the source?

> It works for my CMakeLists as well now. What I did now is
> 
> if(${GrammarSource}/test.g IS_NEWER_THAN ${PROJECT_BINARY_DIR}/test.g)
>     ADD_CUSTOM_TARGET(...)
>     ADD_CUSTOM_COMMAND(...)
> 
> i.e I am totally avoiding the process of rebuilding the test.g file and
> copying the generated files to build/parser
> *IF* test.g has not been modified.

...and if test.g is once modified, you won't have a custom target which
regenerates the source files and copies them to the parser directory -
I'm pretty sure this is not what you intend. Recall the difference
between configuration time and build time in CMakified projects.

> PS: Could you set up your e-mail client so that it marks quoted
>    text in some way? Currently, one can't distinguish between
>    the lines you are writing and the lines written by others.
> 
> Sorry. I did not realize that. I use gmail and when I checked my sent
> emails,
> quoted text appears to be clearly marked. Is the quoted text in this email
> marked properly.

Yes, it is.

Regards,

Michael

> On Sun, Mar 4, 2012 at 4:05 PM, Michael Hertling <mhertling at online.de>wrote:
> 
>> On 03/04/2012 08:02 PM, Ajay Panyala wrote:
>>> The following project is a boiled-down version of yours but doesn't
>>> need any programs except for CMake - that's what I actually meant
>>> with "minimal but complete" as I don't have the org.antlr.Tool
>>> Java program:
>>>
>>> I am sorry. I was mainly trying to cleanup the big CMakeLists file
>>> I had and removed about 10 different targets - all of which were
>>> rebuilt because the 5 files (test*.*) were overwritten.
>>>
>>> If you want to try running the test project on your system
>>>
>>> test.g is at http://pastie.org/private/agzor3ibzoa5pom6q31qq
>>>
>>> org.antlr.Tool is at www.antlr.org/download/antlr-3.4-complete.jar
>>>
>>> After configuration, you can run the target by "make X=0" and check the
>>> timestamps by "ls -l --full-time test.tokens parser src". Issuing "make
>>> X=0" again reveals that the copied files aren't rewritten as it happens
>>> after "make X=1". Thus, AFAICS, everything works as expected. Could you
>>> check if the above-noted example also works on your system?
>>>
>>> It does not work. The files are still overwritten.
>>
>> Could you run the following shell script in a build tree configured
>> with the CMakeLists.txt from my previous reply and post the output?
>>
>> # check.sh:
>> make X=0 2>&1 > /dev/null
>> echo "After make X=0 (1):"
>> echo "-------------------"
>> echo ""
>> ls --full-time test.tokens src parser
>> echo -ne "\n\n"
>> sleep 3
>> make X=0 2>&1 > /dev/null
>> echo "After make X=0 (2):"
>> echo "-------------------"
>> echo ""
>> ls --full-time test.tokens src parser
>> echo -ne "\n\n"
>> sleep 3
>> make X=1 2>&1 > /dev/null
>> echo "After make X=1:"
>> echo "---------------"
>> echo ""
>> ls --full-time test.tokens src parser
>> # End of check.sh
>>
>> BTW, which CMake version do you use?
>>
>> Regards,
>>
>> Michael
>>
>> PS: Could you set up your e-mail client so that it marks quoted
>>    text in some way? Currently, one can't distinguish between
>>    the lines you are writing and the lines written by others.
>>
>>> What exactly does not work with your example? You wrote:
>>>
>>>>>> I have 4 cmake -E copy_if_different commands, one for each file.
>>>>>> Only the last file is not copied (if similar). [...]
>>>
>>> Does this mean that the last file out of four - in fact, your example
>>> handles five files - is not copied *although* the source file and the
>>> destination file are different, i.e. similar but not equal?
>>>
>>> Yes the file test.tokens is not copied overwritten since they are
>>> exactly (diff) similar files. This is the case with the other 4 files as
>>> well,
>>> but they are still copied over and rewritten.
>>>
>>>
>>> You wrote further:
>>>
>>>>>> [...] The others are copied
>>>>>> even if they are the same.
>>>>>>
>>>>>> I verfied that they are the same with a diff.
>>>
>>> Does this mean that source files are copied *although* they are equal
>>> to their respective destination file? How do you determine that they
>>> have been copied? Do you check the timestamps? With --full-time?
>>>
>>> Yes, I do check with ls -l --full-time. Except test.tokens, all the other
>>> files
>>> are copied over (rewritten) even though they are exactly the same (diff
>>> same I mean).
>>>
>>> This is what is confusing me about the behavior of copy_if_different.
>>> that is why it works only with test.tokens and not others.
>>>
>>> PS: Does org.antlr.Tool write to the source tree? If so: Don't do that.
>>>
>>> Yes, it generates the files in the source dir itself (where test.g is
>>> present)
>>> I now modified CMakeLists to copy test.g to the project build folder and
>>> run it there. The new CMakeLists is at
>>>
>>> http://pastie.org/private/p1yi0l8so9cqimqlywfmhw
>>>
>>>
>>> Thank You
>>> Ajay
>>>
>>> On Sun, Mar 4, 2012 at 12:52 AM, Michael Hertling <mhertling at online.de
>>> wrote:
>>>
>>>> On 03/04/2012 01:06 AM, Ajay Panyala wrote:
>>>>> Please provide a minimal but complete example for this issue.
>>>>>
>>>>> Please find it in the following link
>>>>> http://pastie.org/private/pd13u33s9xpfihf2dbzc1q
>>>>
>>>>
>>>
>>>
>>>> The following project is a boiled-down version of yours but doesn't
>>>> need any programs except for CMake - that's what I actually meant
>>>> with "minimal but complete" as I don't have the org.antlr.Tool
>>>> Java program:
>>>>
>>>> CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
>>>> PROJECT(P NONE)
>>>> SET(CMAKE_VERBOSE_MAKEFILE ON)
>>>>
>>>> SET(GrammarSource ${PROJECT_BINARY_DIR}/src)
>>>> FILE(MAKE_DIRECTORY ${PROJECT_BINARY_DIR}/src)
>>>>
>>>> ADD_CUSTOM_TARGET(GrammarBuild ALL
>>>>    COMMAND ${CMAKE_COMMAND} -E echo "\${X}"
>>>>        > ${GrammarSource}/testLexer.h
>>>>    COMMAND ${CMAKE_COMMAND} -E echo "\${X}"
>>>>        > ${GrammarSource}/testLexer.c
>>>>    COMMAND ${CMAKE_COMMAND} -E echo "\${X}"
>>>>        > ${GrammarSource}/testParser.h
>>>>    COMMAND ${CMAKE_COMMAND} -E echo "\${X}"
>>>>        > ${GrammarSource}/testParser.c
>>>>    COMMAND ${CMAKE_COMMAND} -E echo "\${X}"
>>>>        > ${PROJECT_BINARY_DIR}/test.tokens
>>>> )
>>>>
>>>> ADD_CUSTOM_COMMAND(TARGET GrammarBuild POST_BUILD
>>>>    COMMAND ${CMAKE_COMMAND} -E copy_if_different
>>>>        ${GrammarSource}/testLexer.h
>>>>        ${PROJECT_BINARY_DIR}/parser/testLexer.h
>>>>    COMMAND ${CMAKE_COMMAND} -E copy_if_different
>>>>        ${GrammarSource}/testLexer.c
>>>>        ${PROJECT_BINARY_DIR}/parser/testLexer.c
>>>>    COMMAND ${CMAKE_COMMAND} -E copy_if_different
>>>>        ${GrammarSource}/testParser.h
>>>>        ${PROJECT_BINARY_DIR}/parser/testParser.h
>>>>    COMMAND ${CMAKE_COMMAND} -E copy_if_different
>>>>        ${GrammarSource}/testParser.c
>>>>        ${PROJECT_BINARY_DIR}/parser/testParser.c
>>>>    COMMAND ${CMAKE_COMMAND} -E copy_if_different
>>>>        ${PROJECT_BINARY_DIR}/test.tokens
>>>>        ${PROJECT_BINARY_DIR}/parser/test.tokens
>>>> )
>>>>
>>>> After configuration, you can run the target by "make X=0" and check the
>>>> timestamps by "ls -l --full-time test.tokens parser src". Issuing "make
>>>> X=0" again reveals that the copied files aren't rewritten as it happens
>>>> after "make X=1". Thus, AFAICS, everything works as expected. Could you
>>>> check if the above-noted example also works on your system?
>>>>
>>>> What exactly does not work with your example? You wrote:
>>>>
>>>>>>> I have 4 cmake -E copy_if_different commands, one for each file.
>>>>>>> Only the last file is not copied (if similar). [...]
>>>>
>>>> Does this mean that the last file out of four - in fact, your example
>>>> handles five files - is not copied *although* the source file and the
>>>> destination file are different, i.e. similar but not equal?
>>>>
>>>> You wrote further:
>>>>
>>>>>>> [...] The others are copied
>>>>>>> even if they are the same.
>>>>>>>
>>>>>>> I verfied that they are the same with a diff.
>>>>
>>>> Does this mean that source files are copied *although* they are equal
>>>> to their respective destination file? How do you determine that they
>>>> have been copied? Do you check the timestamps? With --full-time?
>>>>
>>>> Regards,
>>>>
>>>> Michael
>>>>
>>>> PS: Does org.antlr.Tool write to the source tree? If so: Don't do that.
>>>>
>>>>> On Sat, Mar 3, 2012 at 3:54 PM, Michael Hertling <mhertling at online.de
>>>>> wrote:
>>>>>
>>>>>> On 03/04/2012 12:14 AM, Ajay Panyala wrote:
>>>>>>> No, it wouldn't; check it out:
>>>>>>>
>>>>>>> % touch a
>>>>>>> % rm -f b
>>>>>>> % ls b
>>>>>>> ls: cannot access b: No such file or directory
>>>>>>> % cmake -E copy_if_different a b
>>>>>>> % ls b
>>>>>>> b
>>>>>>> % cksum a b
>>>>>>> 4294967295 0 a
>>>>>>> 4294967295 0 b
>>>>>>>
>>>>>>> It works with one file, but I have 4 files that are generated.
>>>>>>> I have 4 cmake -E copy_if_different commands, one for each file.
>>>>>>> Only the last file is not copied (if similar). The others are copied
>>>>>>> even if they are the same.
>>>>>>>
>>>>>>> I verfied that they are the same with a diff.
>>>>>>>
>>>>>>> Any idea what might be happening here ?
>>>>>>
>>>>>> Please provide a minimal but complete example for this issue.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>>> On Sat, Mar 3, 2012 at 2:47 PM, Michael Hertling <
>> mhertling at online.de
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On 03/03/2012 10:36 PM, Ajay Panyala wrote:
>>>>>>>>> Try "cmake -E copy_if_different ..."
>>>>>>>>>
>>>>>>>>> cmake -E copy_if_different build/test1.c build/tests/test1.c
>>>>>>>>>
>>>>>>>>> That would work when make is run atleast once.
>>>>>>>>> When running make for the 1st time test1.c was never
>>>>>>>>> copied to build/tests before. So I would be comparing a file with
>>>>>>>>> another non-existant file and that would result in an error halting
>>>>>>>>> the make process.
>>>>>>>>
>>>>>>>> No, it wouldn't; check it out:
>>>>>>>>
>>>>>>>> % touch a
>>>>>>>> % rm -f b
>>>>>>>> % ls b
>>>>>>>> ls: cannot access b: No such file or directory
>>>>>>>> % cmake -E copy_if_different a b
>>>>>>>> % ls b
>>>>>>>> b
>>>>>>>> % cksum a b
>>>>>>>> 4294967295 0 a
>>>>>>>> 4294967295 0 b
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>> On Sat, Mar 3, 2012 at 1:20 PM, Hendrik Sattler <
>>>>>> post at hendrik-sattler.de
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Am Samstag, 3. März 2012, 21:41:49 schrieb Ajay Panyala:
>>>>>>>>>>> I have a custom target which runs a command to generate
>>>>>>>>>>> a C source file say test1.c
>>>>>>>>>>>
>>>>>>>>>>> ADD_CUSTOM_TARGET(TestGen ALL
>>>>>>>>>>> COMMAND genExec ${PROJECT_SOURCE_DIR}/Main.java
>>>>>>>>>>> DEPENDS ${PROJECT_SOURCE_DIR}/Main.java
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> And I have a custom command that moves the generated *test1.c *
>>>>>>>>>>> to a new directory inside the build directory.
>>>>>>>>>>>
>>>>>>>>>>> ADD_CUSTOM_COMMAND(
>>>>>>>>>>> TARGET TestGen
>>>>>>>>>>> POST_BUILD
>>>>>>>>>>> COMMAND mv
>>>>>>>>>>> ARGS ${PROJECT_BINARY_DIR}/test1.c ${PROJECT_BINARY_DIR}/tests/
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> Each time I run make, the custom target is run (since custom
>>>> targets
>>>>>>>> are
>>>>>>>>>>> always
>>>>>>>>>>> out-of-date). But I want to avoid moving the new test1.c
>> generated
>>>>>> each
>>>>>>>>>>> time if build/test1.c is the same as build/tests/test1.c since
>>>> there
>>>>>>>> are
>>>>>>>>>>> other targets
>>>>>>>>>>> like add_executable and add_library later in the CMakelists file
>>>> that
>>>>>>>> are
>>>>>>>>>>>  re-built
>>>>>>>>>>> each time since they depend on test1.c
>>>>>>>>>>
>>>>>>>>>> Try "cmake -E copy_if_different ..."
>>>>>>>>>>
>>>>>>>>>> HS