[CMake] Parallel build & test problem

[Marcel Loose]

> Look at the following project for an
example:                                                                
        
>                                                                       
                                               
> #
CMakeLists.txt:                                                         
                                           
> CMAKE_MINIMUM_REQUIRED(VERSION 2.8
FATAL_ERROR)                                                            
          
> PROJECT(PARALLEL
C)                                                                      
                            
> FILE(WRITE ${CMAKE_BINARY_DIR}/generate.txt "0
\n")                                                                    
>
ADD_CUSTOM_COMMAND(                                                     
                                             
>     OUTPUT
${CMAKE_BINARY_DIR}/generate.txt                                        
                                  
>     COMMAND
${CMAKE_COMMAND}                                                        
                                 
>     -DGENERATE=
${CMAKE_BINARY_DIR}/generate.txt                                        
                              
>     -P
${CMAKE_SOURCE_DIR}/generate.cmake)                                     
                                      
> FILE(WRITE ${CMAKE_BINARY_DIR}/f.c "void
f(void){}\n")                                                           
    
> FILE(WRITE ${CMAKE_BINARY_DIR}/g.c "void
g(void){}\n")                                                           
    
> ADD_LIBRARY(f SHARED f.c
${CMAKE_BINARY_DIR}/generate.txt)                                       
                    
> ADD_LIBRARY(g SHARED g.c
${CMAKE_BINARY_DIR}/generate.txt)                                       
                    
>                                                                       
                                               
> #
generate.cmake:                                                         
                                           
> IF(EXISTS
${GENERATE})                                                            
                                   
>   FILE(STRINGS ${GENERATE}
VAR)                                                                    
                  
>   MATH(EXPR VAR
${VAR}+1)                                                               
                             
>   FILE(WRITE ${GENERATE}
"${VAR}\n")                                                             
                    
>
ELSE()                                                                  
                                             
>   FILE(WRITE ${GENERATE} "1
\n")                                                                    
                  
>
ENDIF()                                                                 
                                             
>                                                                       
                                               
> After configuring, enter the following
command:                                                                
      
>
> while true; do
> (make clean; make -j1) | grep "generate\.txt";
> echo "generate.txt: $(cat generate.txt)";
> done
>
> You'll endlessly see "... Generating generate.txt" followed by
> "generate.txt: 1" which is expected. Now, switch to parallel:
>
> while true; do
> (make clean; make -j2) | grep "generate\.txt";
> echo "generate.txt: $(cat generate.txt)";
> done
>
> On my system, findings are:
>
> 1. Two messages "... Generating generate.txt". This means that both
> Make processes run the custom command, i.e. none of these processes
> finds the generated.txt file already generated by the other process.
> 2. The generated.txt file's content varies between 1 and 2: If the
> IF(EXISTS) command of one process is executed after the FILE(WRITE)
> command of the other process, the result in generate.txt will be 2,
> but if the one's IF(EXISTS) command is executed between the other's
> IF(EXISTS) and FILE(WRITE) commands, both processes will find the
> generate.txt file absent and write it with a content of 1. That's
> a typical race condition among the Make processes with j2 or more.
> 3. If I replace '| grep "generate\.txt"' with '> /dev/null' in the
> above-noted command, the following error occurs from time to time:
>
> > CMake Error at .../generate.cmake:3 (MATH):
> >   math cannot parse the expression: "+1": syntax error, unexpected
exp_PLUS,
> >   expecting exp_OPENPARENT or exp_NUMBER (1)
>
> Supposedly, the reason is that the FILE(STRINGS) command of one
process
> is executed during the other's FILE(WRITE) command, i.e. between
open()
> and close() when the generate.txt file is open for writing but not yet
> closed and, thus, empty, so the VAR variable will be empty, too, and
> the MATH() command will fail. That's also a typical race condition.
>
> This example shows that CMake-generated Makefiles might rate targets
as
> sufficiently independent to be built in parallel although these
targets
> are coupled by a custom command which - due to its construction - is
> sensitive to parallel execution and gives rise to a race condition.
> So, the warning w.r.t. building multiple targets in parallel must
> be taken absolutely seriously.
>
> However, David, things can even get worse: Enhance the above-noted
> PARALLEL project's CMakeLists.txt with the following three lines:
>
> FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(void){return 0;}\n")
> ADD_EXECUTABLE(main main.c)
> TARGET_LINK_LIBRARIES(main f g)
>
> Most certainly, this is a quite common configuration, and IMO, it's
> perfectly legal to build with "make -j2 main", i.e. explicitly only
> one target in parallel. When I enter the following lines
>
> while true; do
> (make clean; make -j2 main) > /dev/null;
> echo "generate.txt: $(cat generate.txt)";
> done
>
> I see plenty of "generate.txt: 2" messages, i.e. the custom command is
> still executed twice and usually in sequence, but sometimes there's a
> "generate.txt: 1" message. Obviously, the race condition hasn't gone.
>
> What does this mean in regard to parallel building a single target?
> Are independent targets the only ones that can be reliably built in
> parallel, i.e. do I have to build f,g and main individually in the
> correct order by hand? That would be strange, IMO. Maybe, someone
> can shed light upon this issue.
>

Hi Michael,

Nice example. Do you know, by any chance, if this only happens with
custom targets/commands. I get the feeling that race condition only seem
to happen with these user-defined thingies.

Maybe this should be put in the issue tracker? 

I haven't had any issues with race condition when building just one
target in parallel, but I *do* have occasional broken builds when
building multiple targets in parallel. In that case, one of the targets
is a custom target. I therefore recommended my colleague to build one
target at a time when doing parallel builds.

Regards,
Marcel Loose.