View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0015525CMakeModulespublic2015-04-21 02:482016-06-10 14:31
ReporterNils Gladitz 
Assigned ToKitware Robot 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusclosedResolutionmoved 
PlatformMacOSOS XOS VersionYosemite 10.10.2
Product VersionCMake 3.2.1 
Target VersionFixed in Version 
Summary0015525: FindCUDA: Sporadic build failures in parallel make on OS X
DescriptionI sporadically get build errors like:

  CMake Error at foobar.cu.o.cmake:231 (message):
    Error generating
    /Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir//./foobar.cu.o

This only seems to happen in my OS X build; Linux and Windows builds seem to be unaffected.
TagsNo tags attached.
Attached Files

 Relationships
related to 0014550closedJames Bigler FindCUDA: source files shared between multiple targets results in linker using objects from both targets 

  Notes
(0038586)
Nils Gladitz (developer)
2015-04-23 02:38

 I locally extended the cuda_execute_process macro to output stdout/stderr of the failing command.

This gives me something like e.g. the following on stderr:

CMake Error at /Applications/CMake.app/Contents/share/cmake-3.2/Modules/FindCUDA/make2cmake.cmake:38 (file):
  file failed to open for reading (No such file or directory):

    /Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir/foobar.cu.o.NVCC-depend
(0038587)
Nils Gladitz (developer)
2015-04-23 03:01

 /Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir/foobar.cu.o.depend seems to exist.

Given that this is a full (non-incremental) build with a fresh build directory and that foobar.cu.o.depend is created based on foobar.cu.o.NVCC-depend I guess the same dependencies file is being generated twice and in parallel.

e.g. both processes generate foobar.cu.o.NVCC-depend at roughly the same time but the first process consumes and deletes it before the second process gets a chance to.
(0038588)
Nils Gladitz (developer)
2015-04-23 04:01

 Sorry I think I stepped into 0014550 again.

The project has the following broken workaround:
set(CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE OFF)
find_package(CUDA 4.2 REQUIRED)

Due to CUDA_ATTACH_VS_BUILD_RULE being a cache variable initialized to ON by FindCUDA.cmake this leads to the variable being ON on the first run and OFF in subsequent runs; hence making it ineffective for the first run.
(0038589)
James Bigler (developer)
2015-04-23 12:14

 How does the CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE affect a build on OSX?
(0038590)
Nils Gladitz (developer)
2015-04-23 12:30

 Multiple custom commands with the same MAIN_DEPENDENCY can lead to the same command running more than once in parallel builds.

0015434 diagnoses this now but I was still running 3.2.1.

I am not sure why I did not hit this on the other platforms I build for but the issue is inherently sporadic.
(0038591)
James Bigler (developer)
2015-04-23 12:33

 The MAIN_DEPENDENCY should have no affect beyond DEPENDENCY on makefile generators.

What generator are you using?
(0038593)
Nils Gladitz (developer)
2015-04-24 09:53

 I used the "Unix Makefiles" generator.

Take e.g.:
  cmake_minimum_required(VERSION 3.2)

  project(DependencyTest CXX)

  find_package(CUDA REQUIRED)

  cuda_add_executable(foo foo.cu)
  cuda_add_executable(bar foo.cu)

Generate once for each value of CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE and compare the two versions of CMakeFiles/bar.dir/build.make.

With CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE set to true you will find an additional rule which invokes foo_generated_foo.cu.o.cmake which duplicates the rule in foo.dir/build.make.
(0038595)
James Bigler (developer)
2015-04-24 17:54

 OK, I can reproduce that the build rule is run three times when doing things in parallel but only once when run serially. This seems like a CMake bug.

$ make
[ 20%] Building NVCC (Device) object CMakeFiles/bar.dir/bar_generated_foo.cu.o
[ 40%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
Scanning dependencies of target bar
[ 60%] Linking CXX executable bar
[ 60%] Built target bar
Scanning dependencies of target foo
[ 80%] Linking CXX executable foo
[100%] Built target foo

$ make clean

$ make -j
[ 40%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
[ 60%] Building NVCC (Device) object CMakeFiles/bar.dir/bar_generated_foo.cu.o
[ 60%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
[ 80%] Linking CXX executable foo
[ 80%] Built target foo
[100%] Linking CXX executable bar
[100%] Built target bar


There might be something else going on here, since there is some kind of new policy relating to using the same file as a MAIN_DEPENDENCY in a custom_command. This used to work....
(0038596)
Nils Gladitz (developer)
2015-04-24 18:00

 MAIN_DEPENDENCY is documented to be broken when you reuse it for the same dependency.

This was broken before CMP0057 and the reason why I created the policy in the first place.

It just diagnoses this case it does not change behaviour otherwise.
(0038597)
James Bigler (developer)
2015-04-24 18:06

 It didn't used to be broken. It worked just fine for years.

Does this policy only extend to Makefiles or does this also include Visual Studio, because my users are going to be pretty ticked off that there will be no more attachment of compiles to files or a lot of work on my part to find all the duplicates and remove the MAIN_DEPENDENCY.

Is there a bug discussing the issue of why this policy was created and why the makefiles became broken in the first place?

How far does the check go? Directory level or globally?
(0038598)
Nils Gladitz (developer)
2015-04-24 18:31

 When did this work?
The oldest version I've got here right now is 2.8.7 and I can reproduce the issue there.

The diagnostic is not generator specific.

You can still use MAIN_DEPENDENCY when a source file is referenced only once.

Visual studio users which use source files more than once will already be ticked off if you remember 0014550 (This was also one of the reason for the policy).

Other reasons were 0015434 and that the documentation for MAIN_DEPENDENCY already states:

"At most one custom command may specify a given source file as its main dependency."

As far as I remember all file dependencies have directory scope.
(0042762)
Kitware Robot (administrator)
2016-06-10 14:29

 Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2015-04-21 02:48 Nils Gladitz New Issue
2015-04-23 02:38 Nils Gladitz Note Added: 0038586
2015-04-23 03:01 Nils Gladitz Note Added: 0038587
2015-04-23 03:57 Nils Gladitz Relationship added related to 0014550
2015-04-23 04:01 Nils Gladitz Note Added: 0038588
2015-04-23 12:14 James Bigler Note Added: 0038589
2015-04-23 12:30 Nils Gladitz Note Added: 0038590
2015-04-23 12:33 James Bigler Note Added: 0038591
2015-04-24 09:53 Nils Gladitz Note Added: 0038593
2015-04-24 17:54 James Bigler Note Added: 0038595
2015-04-24 18:00 Nils Gladitz Note Added: 0038596
2015-04-24 18:06 James Bigler Note Added: 0038597
2015-04-24 18:31 Nils Gladitz Note Added: 0038598
2016-06-10 14:29 Kitware Robot Note Added: 0042762
2016-06-10 14:29 Kitware Robot Status new => resolved
2016-06-10 14:29 Kitware Robot Resolution open => moved
2016-06-10 14:29 Kitware Robot Assigned To => Kitware Robot
2016-06-10 14:31 Kitware Robot Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team