MantisBT - CMake
View Issue Details
0015525CMakeModulespublic2015-04-21 02:482016-06-10 14:31
Nils Gladitz 
Kitware Robot 
normalminorhave not tried
closedmoved 
MacOS XYosemite 10.10.2
CMake 3.2.1 
 
0015525: FindCUDA: Sporadic build failures in parallel make on OS X
I sporadically get build errors like:

  CMake Error at foobar.cu.o.cmake:231 (message):
    Error generating
    /Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir//./foobar.cu.o

This only seems to happen in my OS X build; Linux and Windows builds seem to be unaffected.
No tags attached.
related to 0014550closed James Bigler FindCUDA: source files shared between multiple targets results in linker using objects from both targets 
Issue History
2015-04-21 02:48Nils GladitzNew Issue
2015-04-23 02:38Nils GladitzNote Added: 0038586
2015-04-23 03:01Nils GladitzNote Added: 0038587
2015-04-23 03:57Nils GladitzRelationship addedrelated to 0014550
2015-04-23 04:01Nils GladitzNote Added: 0038588
2015-04-23 12:14James BiglerNote Added: 0038589
2015-04-23 12:30Nils GladitzNote Added: 0038590
2015-04-23 12:33James BiglerNote Added: 0038591
2015-04-24 09:53Nils GladitzNote Added: 0038593
2015-04-24 17:54James BiglerNote Added: 0038595
2015-04-24 18:00Nils GladitzNote Added: 0038596
2015-04-24 18:06James BiglerNote Added: 0038597
2015-04-24 18:31Nils GladitzNote Added: 0038598
2016-06-10 14:29Kitware RobotNote Added: 0042762
2016-06-10 14:29Kitware RobotStatusnew => resolved
2016-06-10 14:29Kitware RobotResolutionopen => moved
2016-06-10 14:29Kitware RobotAssigned To => Kitware Robot
2016-06-10 14:31Kitware RobotStatusresolved => closed

Notes
(0038586)
Nils Gladitz   
2015-04-23 02:38   
I locally extended the cuda_execute_process macro to output stdout/stderr of the failing command.

This gives me something like e.g. the following on stderr:

CMake Error at /Applications/CMake.app/Contents/share/cmake-3.2/Modules/FindCUDA/make2cmake.cmake:38 (file):
  file failed to open for reading (No such file or directory):

    /Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir/foobar.cu.o.NVCC-depend
(0038587)
Nils Gladitz   
2015-04-23 03:01   
/Users/ngladitz/ctest/scheduled/malum/cdash/MyProject-bin/CMakeFiles/MyTarget.dir/foobar.cu.o.depend seems to exist.

Given that this is a full (non-incremental) build with a fresh build directory and that foobar.cu.o.depend is created based on foobar.cu.o.NVCC-depend I guess the same dependencies file is being generated twice and in parallel.

e.g. both processes generate foobar.cu.o.NVCC-depend at roughly the same time but the first process consumes and deletes it before the second process gets a chance to.
(0038588)
Nils Gladitz   
2015-04-23 04:01   
Sorry I think I stepped into 0014550 again.

The project has the following broken workaround:
set(CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE OFF)
find_package(CUDA 4.2 REQUIRED)

Due to CUDA_ATTACH_VS_BUILD_RULE being a cache variable initialized to ON by FindCUDA.cmake this leads to the variable being ON on the first run and OFF in subsequent runs; hence making it ineffective for the first run.
(0038589)
James Bigler   
2015-04-23 12:14   
How does the CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE affect a build on OSX?
(0038590)
Nils Gladitz   
2015-04-23 12:30   
Multiple custom commands with the same MAIN_DEPENDENCY can lead to the same command running more than once in parallel builds.

0015434 diagnoses this now but I was still running 3.2.1.

I am not sure why I did not hit this on the other platforms I build for but the issue is inherently sporadic.
(0038591)
James Bigler   
2015-04-23 12:33   
The MAIN_DEPENDENCY should have no affect beyond DEPENDENCY on makefile generators.

What generator are you using?
(0038593)
Nils Gladitz   
2015-04-24 09:53   
I used the "Unix Makefiles" generator.

Take e.g.:
  cmake_minimum_required(VERSION 3.2)

  project(DependencyTest CXX)

  find_package(CUDA REQUIRED)

  cuda_add_executable(foo foo.cu)
  cuda_add_executable(bar foo.cu)

Generate once for each value of CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE and compare the two versions of CMakeFiles/bar.dir/build.make.

With CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE set to true you will find an additional rule which invokes foo_generated_foo.cu.o.cmake which duplicates the rule in foo.dir/build.make.
(0038595)
James Bigler   
2015-04-24 17:54   
OK, I can reproduce that the build rule is run three times when doing things in parallel but only once when run serially. This seems like a CMake bug.

$ make
[ 20%] Building NVCC (Device) object CMakeFiles/bar.dir/bar_generated_foo.cu.o
[ 40%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
Scanning dependencies of target bar
[ 60%] Linking CXX executable bar
[ 60%] Built target bar
Scanning dependencies of target foo
[ 80%] Linking CXX executable foo
[100%] Built target foo

$ make clean

$ make -j
[ 40%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
[ 60%] Building NVCC (Device) object CMakeFiles/bar.dir/bar_generated_foo.cu.o
[ 60%] Building NVCC (Device) object CMakeFiles/foo.dir/foo_generated_foo.cu.o
[ 80%] Linking CXX executable foo
[ 80%] Built target foo
[100%] Linking CXX executable bar
[100%] Built target bar


There might be something else going on here, since there is some kind of new policy relating to using the same file as a MAIN_DEPENDENCY in a custom_command. This used to work....
(0038596)
Nils Gladitz   
2015-04-24 18:00   
MAIN_DEPENDENCY is documented to be broken when you reuse it for the same dependency.

This was broken before CMP0057 and the reason why I created the policy in the first place.

It just diagnoses this case it does not change behaviour otherwise.
(0038597)
James Bigler   
2015-04-24 18:06   
It didn't used to be broken. It worked just fine for years.

Does this policy only extend to Makefiles or does this also include Visual Studio, because my users are going to be pretty ticked off that there will be no more attachment of compiles to files or a lot of work on my part to find all the duplicates and remove the MAIN_DEPENDENCY.

Is there a bug discussing the issue of why this policy was created and why the makefiles became broken in the first place?

How far does the check go? Directory level or globally?
(0038598)
Nils Gladitz   
2015-04-24 18:31   
When did this work?
The oldest version I've got here right now is 2.8.7 and I can reproduce the issue there.

The diagnostic is not generator specific.

You can still use MAIN_DEPENDENCY when a source file is referenced only once.

Visual studio users which use source files more than once will already be ticked off if you remember 0014550 (This was also one of the reason for the policy).

Other reasons were 0015434 and that the documentation for MAIN_DEPENDENCY already states:

"At most one custom command may specify a given source file as its main dependency."

As far as I remember all file dependencies have directory scope.
(0042762)
Kitware Robot   
2016-06-10 14:29   
Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.