[CMake] Generated CMakeLists.txt

[Michael Hertling]

On 11/09/2011 04:27 PM, Łukasz Tasz wrote:
> Hi Michael, Hi all,
> 
> The workflow you described works fine, with one remark,
> cmake phase will be triggered each time when input for generator is
> changing, it means reconfiguration will happen even in cases it's not
> needed.
> 
> The issue is that as I wrote:
>>> The setup looks like: generator has some input, and depending on this
>>> input files are generated.
> In most cases changes in input are because some structure changes,
> interface changes and so, number of generated files is not changing,
> but there of course are cases that developer will add / remove some
> file and those are the cases configuration must be reexecuted.
> 
> 
> In case of small project you can rerun reconfiguration as much as you
> want, but if configuration parses lot of cmake files, imports lot of
> modules,
> then it should not be triggered when not needed. That's why I ask for
> kind of hook to cmake --check-build-system action which is
> prerequisite to each make phase.

OK, I see. CMake - or Make, respectively, to be exact - can *not*
distinguish if a change in the generator's input affects the set of
generated files, i.e. if files are added/removed/renamed, or if just
the contents of the generated files will be different. Thus, a simple
dependency of the CMakeLists.txt file on the generator's input file is
not desirable as this would invalidate the whole project each time the
input file is touched. OTOH, the project must be reconfigured if the
set of generated files has changed in order to correctly set up the
generated files' lists. To address this issue, you might:

(1) Separate the generator's input in a file-set related part and a
file-contents related one; the former must not change if solely the
generated files' contents are changing. Of course, you won't have the
generator's input in a single file anymore which might be undesirable.

(2) Introduce an intermediate CMake step; see the following example

# CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(RECONF C)
SET(CMAKE_VERBOSE_MAKEFILE ON)
IF(NOT EXISTS ${CMAKE_BINARY_DIR}/srcs.cmake)
    FILE(WRITE ${CMAKE_BINARY_DIR}/srcs.cmake "")
ENDIF()
INCLUDE(${CMAKE_BINARY_DIR}/srcs.cmake)
IF(NOT EXISTS ${CMAKE_BINARY_DIR}/main.c)
    FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(void){return 0;}\n")
ENDIF()
ADD_EXECUTABLE(main0 EXCLUDE_FROM_ALL main.c ${SRCS})
ADD_CUSTOM_TARGET(main ALL
    COMMAND ${CMAKE_COMMAND}
        -DINPUT=${CMAKE_SOURCE_DIR}/input.txt
        -DDIR=${CMAKE_BINARY_DIR}
        -DVAR=SRCS
        -DSRC=${CMAKE_SOURCE_DIR}/srcs.cmake.in
        -DDST=${CMAKE_BINARY_DIR}/srcs.cmake
        -P ${CMAKE_SOURCE_DIR}/generate.cmake
    COMMAND ${CMAKE_COMMAND}
        --build ${CMAKE_BINARY_DIR}
        --target main0)

# generate.cmake:
FILE(STRINGS ${INPUT} LINES)
UNSET(SRCS)
FOREACH(i IN LISTS LINES)
    STRING(REGEX REPLACE "^(.*)=.*$" "\\1" NAME ${i})
    STRING(REGEX REPLACE "^.*=(.*)$" "\\1" VALUE ${i})
    FILE(WRITE ${DIR}/${NAME}.c.in
        "int ${NAME}(void){return ${VALUE};}\n")
    CONFIGURE_FILE(${DIR}/${NAME}.c.in ${DIR}/${NAME}.c COPYONLY)
    LIST(APPEND SRCS ${DIR}/${NAME}.c)
ENDFOREACH()
LIST(SORT SRCS)
CONFIGURE_FILE(${SRC} ${DST} @ONLY)

# srcs.cmake.in:
SET(@VAR@ @SRCS@)

with an input.txt file:

a=3
b=2
c=1

Each line in the latter file describes a generated file containing a
function which returns the value to the right of the equal sign. The
generate.cmake script generates source files for these functions and
configures the srcs.cmake file with the list of source files. AFAICS,
this roughly models the use case you have outlined. The basic idea is
that the INCLUDE() command establishes a dependency of CMakeLists.txt
on srcs.cmake with the list of generated files, so a reconfiguration
takes place if the latter changes, and the targets are built in an
indirect manner, see the main target: This is a custom target that
first invokes the generate.cmake script and then builds the actual
target main0 via CMake's --build switch - that's the intermediate
CMake step mentioned above. If the set of generated files changes,
e.g. by adding "d=0" to input.txt, generate.cmake will change the
srcs.cmake file and, thus, invalidate CMakeLists.txt, so --build
will trigger a reconfiguration before it rebuilds the project. If
just the generated files' contents have changed, e.g. a=3 -> a=30
in input.txt, the srcs.cmake file remains unaffected, and --build
rebuilds a.c and relinks main0. Note that the CONFIGURE_FILE() in
generate.cmake takes care to not touch the respective output file
if this already exists and would not change. So, a minimum number
of compilations should take place whenever the generator's input
is changed. The downside is that you must provide your own, say,
target forwarding, e.g. main -> main0, but code generators some-
times pose special challenges in CMakified projects.

'hope that helps.

Regards,

Michael

> My very simple example is attached in generator.tar file.
> - CMakeLists.txt
> - model.xml - imput for model,
> - genfiles.py - simple&dummy generaor
> 
> 
> testcase is:
> mkdir test
> cp ....generator.tar test
> tar xvf generator.tar
> mkdir obj && cd obj
> cmake ../cmake_issue
> make -j
> then add new structure in model.xml e.g test.abc
> make -j
> #linking error will occurs since configuration is
> make -j
> # cmake phase will fix configuration and linking will be fine.
> 
> Please remind that dummy model.xml in most cases is changed because of
> internal structure changes.
> I think you got my question about attaching to cmake
> --check-build-system commnad genfiles.py --cmake-only ...
> 
> thanks in advance
> Lukasz
> 
> 
> 2011/11/8 Michael Hertling <mhertling at online.de>:
>> On 11/08/2011 03:30 PM, Łukasz Tasz wrote:
>>> Hi All,
>>>
>>> I have one issue connected with generated code.
>>>
>>> What is recommended way to proceed with generators which provides
>>> source which is dynamic?
>>>
>>> The setup looks like: generator has some input, and depending on this
>>> input files are generated.
>>> From cmake point of view I need only this list of source files to
>>> create library.
>>>
>>> Generator is providing simple command to generate such a list.
>>>
>>>
>>> During cmake phase, I'm calling generator to generate list.cmake with
>>> list of source file,
>>> then such a list is included, and configuration is generated. Source
>>> file are marked as generated, and appropriate
>>> ADD_CUSTOM_COMMAND(OUTPUT list) is registerd.
>>>
>>>
>>> In lifecycle cmake -> make -> edit-input_files -> make -> edit ... make
>>>
>>> everything works smooth with one remark, If you will edit input files
>>> in a way that new source file will be introduced, then regeneration is
>>> not enough, I need to regenerate configuration,
>>> with current approach it looks like:
>>> edit-input-adding-new-file ->
>>>      make-step1-generate: will regenerate the code + regenerate
>>> cmakelist - since new file is there
>>>      make-step2-compile-sources:
>>>      make-step3-linking - will fail since new file is not added to
>>> configuration,
>>>
>>> with next run of make, cmake --check-build-system will recognize
>>> changes in cmakelist and configuration will be regenerated, and build
>>> will be successfull (if code changes was smart enough:) )
>>>
>>> The question is if there are some hooks to cmake --check-build-system
>>> that also additionaly my generate --cmake action will be called? (to
>>> regenerate cmakelist which holds list of files)
>>>
>>> Or maybe my approach is wrong?
>>>
>>> thanks in advance
>>> Lukasz
>>
>> If I understand correctly, you need to establish a dependency of the
>> CMakeLists.txt file on the input for the code generator. You might
>> use a CONFIGURE_FILE() command for this purpose, e.g.
>>
>> CONFIGURE_FILE(input.txt input.txt.bak COPYONLY)
>>
>> with input.txt being the code generator's input file. In this way, a
>> change of input.txt outdates the CMakeLists.txt file, and the entire
>> project is reconfigured and rebuilt the next time Make is called. If
>> the machinery works by now when you reconfigure the project by hand,
>> this should do the trick. If not, please come up with a minimal but
>> complete example to show exactly what you are doing at the moment,
>> what does not work and what you'd like to see instead.
>>
>> Regards,
>>
>> Michael