MantisBT - CMake
View Issue Details
0013385CMakeCMakepublic2012-07-09 06:032013-01-09 10:57
Zaheer Chothia 
Peter Kuemmel 
normalmajoralways
closedfixed 
MicrosoftWindows 7, 64-bitSP1
CMake 2.8.8 
CMake 2.8.9 
0013385: Ninja: LINK_LIBRARIES not spilled to response file
* Command line length is limited on Windows.
* To alleviate this, CMake places object files into a response file, but the
  same is not done for libraries.
* This can pose a problem when the link line becomes too long.
See attached testcase generator.
I am primarily interested in a solution for the Ninja generator, although this
issue affects other generators too (e.g. NMake, MinGW Makefiles). For further discussion see the mailing list: http://www.cmake.org/pipermail/cmake/2012-June/051065.html [^]
No tags attached.
related to 0013461closed Peter Kuemmel Ninja: linker response file gets expanded, command to long 
? testcase.sh (997) 2012-07-09 06:03
https://public.kitware.com/Bug/file/4383/testcase.sh
bz2 cmake_testcase_many_libraries_rspfile.tar.bz2 (5,625) 2012-07-09 07:44
https://public.kitware.com/Bug/file/4384/cmake_testcase_many_libraries_rspfile.tar.bz2
? testcase-fixed.sh (972) 2012-07-09 07:47
https://public.kitware.com/Bug/file/4385/testcase-fixed.sh
patch NinjaRspFileIncludes_LINK_LIBRARIES.patch (2,039) 2012-07-09 12:34
https://public.kitware.com/Bug/file/4386/NinjaRspFileIncludes_LINK_LIBRARIES.patch
zip issue13385_build_ninja_mingw.zip (50,990) 2012-07-10 14:17
https://public.kitware.com/Bug/file/4388/issue13385_build_ninja_mingw.zip
Issue History
2012-07-09 06:03Zaheer ChothiaNew Issue
2012-07-09 06:03Zaheer ChothiaFile Added: testcase.sh
2012-07-09 07:41Peter KuemmelNote Added: 0029994
2012-07-09 07:44Zaheer ChothiaFile Added: cmake_testcase_many_libraries_rspfile.tar.bz2
2012-07-09 07:47Zaheer ChothiaNote Added: 0029995
2012-07-09 07:47Zaheer ChothiaFile Added: testcase-fixed.sh
2012-07-09 07:53Peter KuemmelNote Added: 0029996
2012-07-09 07:53Peter KuemmelStatusnew => confirmed
2012-07-09 09:13Peter KuemmelNote Edited: 0029996bug_revision_view_page.php?bugnote_id=29996#r737
2012-07-09 12:34ClausKleinFile Added: NinjaRspFileIncludes_LINK_LIBRARIES.patch
2012-07-09 12:37ClausKleinNote Added: 0030006
2012-07-10 05:05Peter KuemmelNote Added: 0030011
2012-07-10 05:06Peter KuemmelNote Edited: 0029996bug_revision_view_page.php?bugnote_id=29996#r738
2012-07-10 14:17Zaheer ChothiaNote Added: 0030026
2012-07-10 14:17Zaheer ChothiaFile Added: issue13385_build_ninja_mingw.zip
2012-07-11 03:08Peter KuemmelNote Added: 0030033
2012-07-11 04:44Peter KuemmelNote Added: 0030034
2012-07-11 04:45Peter KuemmelAssigned To => Peter Kuemmel
2012-07-11 05:48Zaheer ChothiaNote Added: 0030035
2012-07-11 05:58Peter KuemmelNote Added: 0030036
2012-07-11 06:21Zaheer ChothiaNote Added: 0030037
2012-08-12 06:16Peter KuemmelRelationship addedrelated to 0013461
2012-08-12 06:26Peter KuemmelStatusconfirmed => assigned
2012-08-13 08:34Peter KuemmelStatusassigned => resolved
2012-08-13 08:34Peter KuemmelResolutionopen => fixed
2012-08-13 08:34Peter KuemmelTarget Version => CMake 2.8.9
2013-01-09 10:57Robert MaynardNote Added: 0032048
2013-01-09 10:57Robert MaynardStatusresolved => closed

Notes
(0029994)
Peter Kuemmel   
2012-07-09 07:41   
I could have a look at it but your test script doesn't work,
neither in msys nor ubuntu: "invalid option" and after
removing -e it simply copies the loop code.
(0029995)
Zaheer Chothia   
2012-07-09 07:47   
Hello Peter, thanks for the quick response. I've uploaded a sample project. I think the script didn't work due to trailing whitespace when I copied from the mailing list post -- it should work fine now.
(0029996)
Peter Kuemmel   
2012-07-09 07:53   
(edited on: 2012-07-10 05:06)
Thanks, script works now, and I can reproduce the error.

Until I've pushed a patch you can fix rules.ninja:
move '$LINK_LIBRARIES' to 'rspfile_content = $in $LINK_LIBRARIES'

(0030006)
ClausKlein   
2012-07-09 12:37   
I have tested this patch only under Darwin!
Please check why the command line length is checked to use response files.
(0030011)
Peter Kuemmel   
2012-07-10 05:05   
I pushed a patch to next:
http://public.kitware.com/pipermail/cmake-commits/2012-July/013022.html [^]

It enables rspfile support now also on Linux.
For Darwin we need to figure out how to get the max command line length at runtime.
(0030026)
Zaheer Chothia   
2012-07-10 14:17   
The patch you pushed works fine on the testcase when using Microsoft's compilers,
although unfortunately it doesn't work correctly with MinGW.

The issue there is the following:
- The corresponding build rule (C_EXECUTABLE_LINKER_RSPFILE) first invokes 'ar'
  and passes LINK_LIBRARIES in the response file.
- LINK_LIBRARIES contains a mixture of library files (e.g. libxyz.a) as well as
  implicit dependencies (e.g. -lkernel32).
- As a result 'ar' fails with "invalid option" since it doesn't understand the
  implicit library options. What should be done instead, is to pass the first
  options to 'ar' and the latter only when calling 'gcc'.
- Attached are the relevant files from the build directory (issue13385_build_ninja_mingw.zip).
  The build log may be best to observe what I am referring to.

For reference these are the details of what I am using:
$ cmake --version
cmake version 2.8.8.20120710-ge5abdb
$ gcc --version
gcc (rubenvb-4.6.3) 4.6.3
(0030033)
Peter Kuemmel   
2012-07-11 03:08   
Explicit usage of ar.exe is a work around for older mingw versions.
This work around is not needed for gcc >4.4, so I've disabled it.
I don't know which mingw versions <4.4 will need this work around,
but I assume you can't use 3.4 with ninja now. Is this a problem?
(0030034)
Peter Kuemmel   
2012-07-11 04:44   
Also on Mac sysconf() can be used.
FYI, on Mac the maximum command line length is 262144.
(0030035)
Zaheer Chothia   
2012-07-11 05:48   
Thanks Peter - the testcase works as expected for me now (MSVC + GCC)! :-)

I would personally be fine with removing the 'ar' workaround. There may be
others, though who experience require this indirection, even when using a fairly
recent version of GCC. This is due to a configuration issue with MinGW-w64
builds which was only fixed in May this year:
http://sourceforge.net/mailarchive/forum.php?thread_name=2766613.iHjfUlaVMm%40speed&forum_name=mingw-w64-public [^]
(0030036)
Peter Kuemmel   
2012-07-11 05:58   
AH, I haven't tested -w64.
Thanks for the link, seems it is only a configure problem, --with-gnu-ld must be used.

Isn't MinGW-w64 more experimental than mingw32?
So most -w64 users are forced to update anyway?
(0030037)
Zaheer Chothia   
2012-07-11 06:21   
I agree that it is reasonable to assume an up-to-date toolset, especially since
it's quite easy to upgrade (unpack and add to PATH). In case someone runs into
this issue I would suggest trying either of these MinGW-w64 builds, which I have
found to be very stable:
http://sourceforge.net/projects/mingwbuilds/ [^]
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/rubenvb/release/ [^]
(0032048)
Robert Maynard   
2013-01-09 10:57   
Closing resolved issues that have not been updated in more than 4 months.