MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0015968CMakeCMakepublic2016-02-10 12:132016-06-10 14:21
ReporterChris 
Assigned ToBrad King 
PriorityhighSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformApple MacOSOS XOS Version10.4.10
Product VersionCMake 3.4.3 
Target VersionCMake 3.6Fixed in VersionCMake 3.6 
Summary0015968: Ninja generator is non-deterministic
DescriptionCMake's Ninja generator is non-deterministic on our project (LLVM). This is trivially reproducible by configuring llvm into a clean directory multiple times and comparing the results.

This is a very real problem for our project because the nondeterminism in the generated ninja build seems to be the cause of non-determinism in our build outputs. One of the important methods of compiler testing is what we call a 3-stage build. The process is basically:

(1) build clang
(2) use clang from step 1 to build clang again
(3) use clang from step 2 to build clang again
(4) ensure clang from step 2 and clang from step 3 are binary identical

Today this works using the CMake Unix Makefile generator, but not using the Ninja generator. This issue reproduces on CMake 3.4.x and CMake 3.5.0 rc1. I assume it probably goes back quite a long ways too.
Steps To Reproducemkdir scratch
cd scratch
cmake -G Ninja <path to llvm>
cp build.ninja ..
rm -rf
cmake -G Ninja <path to llvm>
diff build.ninja ../build.ninja
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2016-02-10 12:13ChrisNew Issue
2016-02-10 14:12Brad KingNote Added: 0040467
2016-02-10 17:13ChrisNote Added: 0040468
2016-02-10 17:26ChrisNote Added: 0040469
2016-02-11 08:56Brad KingNote Edited: 0040467bug_revision_view_page.php?bugnote_id=40467#r2023
2016-02-11 08:59Brad KingNote Added: 0040471
2016-02-11 16:21ChrisNote Added: 0040476
2016-02-12 08:58Brad KingNote Added: 0040477
2016-02-12 19:27ChrisNote Added: 0040480
2016-02-16 09:45Brad KingNote Added: 0040493
2016-02-16 09:45Brad KingAssigned To => Brad King
2016-02-16 09:45Brad KingStatusnew => resolved
2016-02-16 09:45Brad KingResolutionopen => fixed
2016-02-16 09:45Brad KingFixed in Version => CMake 3.6
2016-02-16 09:45Brad KingTarget Version => CMake 3.6
2016-06-10 14:21Kitware RobotNote Added: 0041252
2016-06-10 14:21Kitware RobotStatusresolved => closed

Notes
(0040467)
Brad King   
2016-02-10 14:12   
(edited on: 2016-02-11 08:56)
Thanks. Please try this fix:

 Ninja: Fix non-determinism in generated build statement order
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=59ade844 [^]
(0040468)
Chris   
2016-02-10 17:13   
Even with the applied patch I'm still seeing differences in the build.ninja files. A lot less differences than previously, but target orders are still different.

I don't expect that those should impact the build output. I have a test build running.

I applied your patch on top of CMake 3.5.0 rc2.
(0040469)
Chris   
2016-02-10 17:26   
My build completed. While the ninja files are not identical, the generated binaries are.

Thanks for the rapid fix. Any chance this can get nominated for 3.5.0?
(0040471)
Brad King   
2016-02-11 08:59   
Re 0015968:0040469: The development window for 3.5 is closed:

 https://cmake.org/pipermail/cmake-developers/2016-February/027637.html [^]

We typically don't add non-regression fixes after rc1 because every change risks a new regression and therefore undermines rc testing already done by others.

Meanwhile we should work further to eliminate the non-determinism you're seeing. I just generated a LLVM build tree with "-G Ninja" twice and got identical build.ninja files. What differences do you see?
(0040476)
Chris   
2016-02-11 16:21   
I'm still seeing a lot of dependencies being re-ordered.

With your patch I'm no longer seeing randomization in the order that targets appear in the file.

One example diff I'm seeing is the first run has:

build cmake_order_depends_target_llvm-tblgen: phony || lib/libLLVMSupport.a lib/libLLVMTableGen.a utils/TableGen/obj.llvm-tblgen

The second run has:

build cmake_order_depends_target_llvm-tblgen: phony || utils/TableGen/obj.llvm-tblgen lib/libLLVMSupport.a lib/libLLVMTableGen.a

This is less important than the other diffs because it doesn't impact the final built product.
(0040477)
Brad King   
2016-02-12 08:58   
Re 0015968:0040476: Thanks. Here is a fix for at least some such cases:

 Ninja: Fix non-determinism in generated target dependency order
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=46fa9583 [^]
(0040480)
Chris   
2016-02-12 19:27   
With that patch the ninja files are identical. I tested by comparing 3 subsequent builds.

Thanks for the rapid fix!
(0040493)
Brad King   
2016-02-16 09:45   
Great, thanks for testing!
(0041252)
Kitware Robot   
2016-06-10 14:21   
This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.