View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0015968CMakeCMakepublic2016-02-10 12:132016-06-10 14:21
ReporterChris 
Assigned ToBrad King 
PriorityhighSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformApple MacOSOS XOS Version10.4.10
Product VersionCMake 3.4.3 
Target VersionCMake 3.6Fixed in VersionCMake 3.6 
Summary0015968: Ninja generator is non-deterministic
DescriptionCMake's Ninja generator is non-deterministic on our project (LLVM). This is trivially reproducible by configuring llvm into a clean directory multiple times and comparing the results.

This is a very real problem for our project because the nondeterminism in the generated ninja build seems to be the cause of non-determinism in our build outputs. One of the important methods of compiler testing is what we call a 3-stage build. The process is basically:

(1) build clang
(2) use clang from step 1 to build clang again
(3) use clang from step 2 to build clang again
(4) ensure clang from step 2 and clang from step 3 are binary identical

Today this works using the CMake Unix Makefile generator, but not using the Ninja generator. This issue reproduces on CMake 3.4.x and CMake 3.5.0 rc1. I assume it probably goes back quite a long ways too.
Steps To Reproducemkdir scratch
cd scratch
cmake -G Ninja <path to llvm>
cp build.ninja ..
rm -rf
cmake -G Ninja <path to llvm>
diff build.ninja ../build.ninja
TagsNo tags attached.
Attached Files

 Relationships

  Notes
(0040467)
Brad King (manager)
2016-02-10 14:12
edited on: 2016-02-11 08:56

 Thanks. Please try this fix:

 Ninja: Fix non-determinism in generated build statement order
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=59ade844 [^]
(0040468)
Chris (reporter)
2016-02-10 17:13

 Even with the applied patch I'm still seeing differences in the build.ninja files. A lot less differences than previously, but target orders are still different.

I don't expect that those should impact the build output. I have a test build running.

I applied your patch on top of CMake 3.5.0 rc2.
(0040469)
Chris (reporter)
2016-02-10 17:26

 My build completed. While the ninja files are not identical, the generated binaries are.

Thanks for the rapid fix. Any chance this can get nominated for 3.5.0?
(0040471)
Brad King (manager)
2016-02-11 08:59

 Re 0015968:0040469: The development window for 3.5 is closed:

 https://cmake.org/pipermail/cmake-developers/2016-February/027637.html [^]

We typically don't add non-regression fixes after rc1 because every change risks a new regression and therefore undermines rc testing already done by others.

Meanwhile we should work further to eliminate the non-determinism you're seeing. I just generated a LLVM build tree with "-G Ninja" twice and got identical build.ninja files. What differences do you see?
(0040476)
Chris (reporter)
2016-02-11 16:21

 I'm still seeing a lot of dependencies being re-ordered.

With your patch I'm no longer seeing randomization in the order that targets appear in the file.

One example diff I'm seeing is the first run has:

build cmake_order_depends_target_llvm-tblgen: phony || lib/libLLVMSupport.a lib/libLLVMTableGen.a utils/TableGen/obj.llvm-tblgen

The second run has:

build cmake_order_depends_target_llvm-tblgen: phony || utils/TableGen/obj.llvm-tblgen lib/libLLVMSupport.a lib/libLLVMTableGen.a

This is less important than the other diffs because it doesn't impact the final built product.
(0040477)
Brad King (manager)
2016-02-12 08:58

 Re 0015968:0040476: Thanks. Here is a fix for at least some such cases:

 Ninja: Fix non-determinism in generated target dependency order
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=46fa9583 [^]
(0040480)
Chris (reporter)
2016-02-12 19:27

 With that patch the ninja files are identical. I tested by comparing 3 subsequent builds.

Thanks for the rapid fix!
(0040493)
Brad King (manager)
2016-02-16 09:45

 Great, thanks for testing!
(0041252)
Kitware Robot (administrator)
2016-06-10 14:21

 This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2016-02-10 12:13 Chris New Issue
2016-02-10 14:12 Brad King Note Added: 0040467
2016-02-10 17:13 Chris Note Added: 0040468
2016-02-10 17:26 Chris Note Added: 0040469
2016-02-11 08:56 Brad King Note Edited: 0040467
2016-02-11 08:59 Brad King Note Added: 0040471
2016-02-11 16:21 Chris Note Added: 0040476
2016-02-12 08:58 Brad King Note Added: 0040477
2016-02-12 19:27 Chris Note Added: 0040480
2016-02-16 09:45 Brad King Note Added: 0040493
2016-02-16 09:45 Brad King Assigned To => Brad King
2016-02-16 09:45 Brad King Status new => resolved
2016-02-16 09:45 Brad King Resolution open => fixed
2016-02-16 09:45 Brad King Fixed in Version => CMake 3.6
2016-02-16 09:45 Brad King Target Version => CMake 3.6
2016-06-10 14:21 Kitware Robot Note Added: 0041252
2016-06-10 14:21 Kitware Robot Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team