MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0015620CMakeCMakepublic2015-06-17 11:372016-01-04 11:51
ReporterJames Johnston 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformWindowsOSWindows 7 SP1 64-bitOS Version
Product VersionCMake 3.2.3 
Target VersionCMake 3.4Fixed in VersionCMake 3.4 
Summary0015620: Ninja generator triggers race condition in Borland bcc32 linker causing intermittent build failure
DescriptionUnfortunately, it appears that the Borland linker utilizes temporary files with a hard-coded name (e.g. turboc.$ln); these temporary files are written to the linker's current directory. (Determined via Process Monitor). If two or more concurrent linkers are run with the same current directory, some of them may fail as shown below because they will both be trying to use the same temporary filename.

The Ninja generator, unfortunately seems to keep the current directory to CMAKE_BINARY_DIR. If the CMake project contains multiple libraries (e.g. VTK 5.4.2, which I tested and has dozens), then it is possible that Ninja will invoke two concurrent linkers with the same current directory of CMAKE_BINARY_DIR. They may then conflict with each other - both attempting to write to a file named ${CMAKE_BINARY_DIR}/turboc.$ln, for example.

To compare, the NMake jom generator runs the linker for each target in a unique target-specific directory, such that this problem never arises. If the Ninja generator could also do this, it would work around this issue. (In fact, it would work around any compiler that happens to write temporary files to its current directory).
Steps To ReproduceMy configuration:
 * Quad-core hyper-threaded Core i7 CPU
 * C++ Builder 5 Update 1
 * VTK 5.4.2 (this version still builds under C++ Builder, unlike some newer versions)
 * CMake 3.3.0-rc1 (Kitware-provided binary)
 * Ninja 1.5.3, built from source using VC++ 2008 Express
 * VTK configured: CMAKE_BUILD_TYPE=RelWithDebInfo, VTK_REMOVE_LEGACY=ON, BUILD_TESTING=OFF, BUILD_SHARED_LIBS=ON

1. Configure VTK with the above settings. (e.g. to C:\VTK-build)
2. Run "ninja" to start the build.
3. The build *may* fail if two projects happen to link concurrently. Of course, it will always succeed if you pass "-j 1" to Ninja to only use one CPU core.
4. Even if you don't immediately experience failure, you can use Sysinternals Process Monitor from Microsoft to monitor the files being written on your system. Note that you'll see repeated writes to files like "C:\VTK-build\turboc.$ln".
Additional InformationWhile I tested this with C++ Builder 5, the error appears to exist even in the newest C++ Builder versions: http://docwiki.embarcadero.com/RADStudio/XE8/en/E2216_Unable_to_create_turboc.$ln_(C%2B%2B) [^]

Sample error when building VTK 5.4.2:
=====================
[214/2008] Linking CXX shared library bin\vtkDICOMParser.dll Borland C++ 5.5.1 for Win32 Copyright (c) 1993, 2000 Borland Turbo Incremental Link 5.66 Copyright (c) 1997-2002 Borland
Fatal: No object file or .EXE name was given

Borland Implib Version 3.0.22 Copyright (c) 1991, 2000 Inprise Corporation Error bin\: unable to open file [215/2008] Linking C shared library bin\vtkzlib.dll
FAILED: cmd.exe /C "cd . && C:\PROGRA~2\Borland\CBUILD~1\Bin\bcc32.exe -tWR
-tW- -tWD -ebin\vtkzlib.dll -tWM -lS:104857
6 -lSc:4098 -lH:1048576 -lHc:8192 -v import32.lib
Utilities\vtkzlib\CMakeFiles\vtkzlib.dir\adler32.obj <snip more OBJ files> && implib -c -w bin\vtkzlib.lib bin\vtkzlib.dll && cd ."
Borland C++ 5.5.1 for Win32 Copyright (c) 1993, 2000 Borland Turbo Incremental Link 5.66 Copyright (c) 1997-2002 Borland
Fatal: Unable to open file 'turboc.$ln'
=====================

Here is the offending snippet from one of the generated ninja files -
build.ninja:

=================================================
build bin\vtkzlib.dll bin\vtkzlib.lib: C_SHARED_LIBRARY_LINKER__vtkzlib Utilities\vtkzlib\CMakeFiles\vtkzlib.dir\adler32.obj <snip more OBJ files>
  LANGUAGE_COMPILE_FLAGS = -tWM -Od
  LINK_FLAGS = -tWM -lS:1048576 -lSc:4098 -lH:1048576 -lHc:8192 -v
  LINK_LIBRARIES = import32.lib
  POST_BUILD = cd .
  PRE_LINK = cd .
  TARGET_FILE = bin\vtkzlib.dll
  TARGET_IMPLIB = bin\vtkzlib.lib
  TARGET_PDB = vtkzlib.dll.dbg
=================================================

And the corresponding ninja rule:

=================================================
rule C_SHARED_LIBRARY_LINKER__vtkzlib
  command = cmd.exe /C "$PRE_LINK &&
C:\PROGRA~2\Borland\CBUILD~1\Bin\bcc32.exe -tWR -tW- -tWD -e$TARGET_FILE $LINK_FLAGS $LINK_PATH $LINK_LIBRARIES $in && implib -c -w $TARGET_IMPLIB $TARGET_FILE && $POST_BUILD"
  description = Linking C shared library $TARGET_FILE
  restat = $RESTAT
=================================================

Notice POST_BUILD and PRE_LINK variables don't change directory to something else. What I think will work around this bcc32 design flaw is if it can do something like:
  POST_BUILD = cd ..\..
  PRE_LINK = cd Utilities\vtkzlib
(Or alternatively, cd to absolute paths like the NMake JOM example below).
And of course, adjust the relative paths in other parameters to the build rule

To contrast, NMake JOM generator has it right in order to work around this
bcc32 issue - notice we change directory to a target-specific directory before linking, and then go back when done:

=================================================
Utilities\vtkzlib\CMakeFiles\vtkzlib.dir\build.make:
bin\vtkzlib.dll: Utilities\vtkzlib\CMakeFiles\vtkzlib.dir\adler32.obj
<snip more OBJ files>
bin\vtkzlib.dll: Utilities\vtkzlib\CMakeFiles\vtkzlib.dir\build.make
    <snip CMake progress command>
    cd
C:\Users\JamesJ\Documents\Repositories\SuperRepo\VTK-JOM\Utilities\vtkzlib
    C:\PROGRA~2\Borland\CBUILD~1\Bin\bcc32.exe -tWR -tW- -tWD @<< -e..\..\bin\vtkzlib.dll -tWM -lS:1048576 -lSc:4098 -lH:1048576 -lHc:8192 -v import32.lib $(vtkzlib_OBJECTS) $(vtkzlib_EXTERNAL_OBJECTS) <<
    implib -c -w ..\..\bin\vtkzlib.lib ..\..\bin\vtkzlib.dll
    cd C:\Users\JamesJ\Documents\Repositories\SuperRepo\VTK-JOM
=================================================

For what it's worth, at this time, I think this issue is limited to the linker and not to the single-unit compilation of a CPP file to OBJ file. I ran Process Monitor for a few seconds while VTK was compiling, and it seems that the only file writes to the build tree were either legitimate writes to named intermediate/output files, or else to these problematic files from the linker.
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2015-06-17 11:37James JohnstonNew Issue
2015-06-18 09:13Brad KingNote Added: 0038945
2015-06-18 13:45James JohnstonNote Added: 0038948
2015-06-18 14:13Brad KingNote Added: 0038949
2015-06-18 14:13Brad KingAssigned To => Brad King
2015-06-18 14:13Brad KingStatusnew => assigned
2015-06-18 14:13Brad KingTarget Version => CMake 3.4
2015-06-18 16:36Brad KingNote Added: 0038952
2015-06-18 16:36Brad KingStatusassigned => resolved
2015-06-18 16:36Brad KingResolutionopen => fixed
2015-06-18 16:36Brad KingFixed in Version => CMake 3.4
2016-01-04 11:51Robert MaynardNote Added: 0040072
2016-01-04 11:51Robert MaynardStatusresolved => closed

Notes
(0038945)
Brad King   
2015-06-18 09:13   
I don't think anyone has ever tried the Borland compiler with anything but the Borland Makefiles generator before, so one one has tried parallel builds with the Borland tools either. I'm happy that it gets this far.

You may be able to use job pools:

 http://www.cmake.org/cmake/help/v3.3/prop_gbl/JOB_POOLS.html [^]
 http://www.cmake.org/cmake/help/v3.3/prop_tgt/JOB_POOL_LINK.html [^]
 http://www.cmake.org/cmake/help/v3.3/variable/CMAKE_JOB_POOL_LINK.html [^]

to limit builds to one link command at a time.
(0038948)
James Johnston   
2015-06-18 13:45   
Thanks for the suggestion to use job pools. This worked successfully for me 5 out of 5 times, whereas I was failing to build 5 out of 5 times before (at various points in build, with various odd errors).

I think it's easier to do this than my original idea of changing the current directory.

I think this should be default behavior, so I've proposed a patch on the cmake-developers mailing list to make it so.
(0038949)
Brad King   
2015-06-18 14:13   
Re 0015620:0038948: Yes, thanks.

For reference, the mailing list thread is here:

 [PATCH] Fixed Borland linker issue when used with Ninja generator
 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/13477 [^]
(0038952)
Brad King   
2015-06-18 16:36   
Patch from mailing list applied:

 Embarcadero: Run at most one linker invocation at a time
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=078b60f0 [^]
(0040072)
Robert Maynard   
2016-01-04 11:51   
Closing resolved issues that have not been updated in more than 4 months.