[CMake] LINK_LIBRARIES not spilled to response file

[Zaheer Chothia]

Hello,

I posted a mail here [1] but have yet to receive any replies.  I think my
last
message was too long and detailed so let me summarize:
* Command line length is limited on Windows.
* To alleviate this, CMake places object files into a response file, but the
  same is not done for libraries.
* This can pose a problem when the link line becomes too long.

I am primarily interested in a solution for the Ninja generator, although
this
issue affects other generators too.  My previous mail contains a testcase
and
proposed solution.  In the interim another issue [2] was posted, but that is
orthogonal and does not solve what is discussed here.

Kind regards,

*--Zaheer*
*
*
[1]: http://www.cmake.org/pipermail/cmake/2012-June/051065.html
[2]: http://public.kitware.com/Bug/view.php?id=13366

On Thu, Jun 28, 2012 at 3:50 PM, Zaheer Chothia <zaheer.chothia at gmail.com>wrote:

> Hello,
>
> I encountered an issue while building a CMake project where one target is
> linked
> against a large number of libraries.  Unlike object files, libraries are
> not
> placed into a response file, which can lead to build commands which exceed
> the
> length limits on Windows.  For reference, I am using the CMake 2.8.9-rc1
> and
> Ninja generator with Microsoft compilers.
>
> Following this mail is a testcase generator [1] to demonstrate this issue
> (sample project attached for convenience).  The build fails with this
> error (for
> readibility I replaced a long sequence of libraries with <...>):
>
>     FAILED: cmd.exe /c cd. && "C:\Program Files
> (x86)\CMake\bin\cmake.exe" -E vs_link_exe
> C:\PROGRA~2\MICROS~3.0\VC\bin\cl.exe  /nologo @hello.exe.rsp  /DWIN32
> /D_WINDOWS /W3 /Zm1000 /D_DEBUG /MDd /Zi  /Ob0 /Od /RTC1 /Fehello.exe
> /Fdhello.pdb -link /implib:hello.lib /version:0.0  /STACK:10000000
> /machine:X86  /debug /INCREMENTAL /subsystem:console
> src\abcdefghijklmnopqrstuvwxyz0123456789\library1.lib
> src\abcdefghijklmnopqrstuvwxyz0123456789\library2.lib <...>
> kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib
> oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd.
>     The command line is too long.
>
>     ninja: build stopped: subcommand failed.
>
> Although this example may seem artificial, with the use case I refer to (i)
> libraries are are specified by absolute paths so they are indeed
> reasonably long
> and (ii) since there are third-party libraries involved I would not be
> able to
> simply combine source files into one large library as is possible here.  I
> should also mention that this issue does not affect the Visual Studio
> generators, however it is present with the following: Ninja, MinGW
> Makefiles, NMake Makefiles, MSYS Makefiles.  For Ninja I suspect that the
> indirection via cmd.exe imposes a maximum command length of 8192 KB,
> whereas for
> the others this will likely be 32 KB (CreateProcess).
>
> I would be quite content if this is fixed for the Ninja generator.  A
> simple fix
> would be to adapt the build rules by moving $LINK_LIBRARIES from 'command'
> to
> 'rspfile_content':
>
>     --- rules.ninja.bak 2012-06-28 15:23:35 +0100
>     +++ rules.ninja     2012-06-28 15:38:09 +0100
>     @@ -40,10 +40,10 @@
>      # Rule for linking C executable.
>
>      rule C_EXECUTABLE_LINKER_RSPFILE
>     -  command = cmd.exe /c $PRE_LINK && "C:\Program Files
> (x86)\CMake\bin\cmake.exe" -E vs_link_exe
> C:\PROGRA~2\MICROS~3.0\VC\bin\cl.exe  /nologo @$out.rsp  $FLAGS
> /Fe$out /Fd$TARGET_PDB -link /implib:$TARGET_IMPLIB /version:0.0
> $LINK_FLAGS $LINK_LIBRARIES && $POST_BUILD
>     +  command = cmd.exe /c $PRE_LINK && "C:\Program Files
> (x86)\CMake\bin\cmake.exe" -E vs_link_exe
> C:\PROGRA~2\MICROS~3.0\VC\bin\cl.exe  /nologo @$out.rsp  $FLAGS
> /Fe$out /Fd$TARGET_PDB -link /implib:$TARGET_IMPLIB /version:0.0
> $LINK_FLAGS && $POST_BUILD
>        description = Linking C executable $out
>        rspfile = $out.rsp
>     -  rspfile_content = $in
>     +  rspfile_content = $in $LINK_LIBRARIES
>
> Best,
>
> --Zaheer
>
> ---- [1]: BEGIN: testcase.sh
> ---------------------------------------------------
> #!/bin/bash -e
>
> NUM_LIBRARIES=500
> # Use a long path to quickly exhaust the command-line length limit.
> SRC_DIR=src/abcdefghijklmnopqrstuvwxyz0123456789
>
> # Root directory: application and CMakeLists.txt
> echo "int main() { return 0; }" > hello.c
>
> cat << EOF > CMakeLists.txt
> cmake_minimum_required(VERSION 2.8)
>
> project(Hello)
>
> add_subdirectory($SRC_DIR)
>
> add_executable(hello hello.c)
> target_link_libraries(hello
> EOF
>
> for ((i = 1; i <= $NUM_LIBRARIES; i++)); do
>     echo "    library$i" >> "CMakeLists.txt"
> done
>
> echo ")" >> "CMakeLists.txt"
>
> # Libraries: sources and CMakeLists.txt
> mkdir -p "$SRC_DIR"
> [[ -f "$SRC_DIR/CMakeLists.txt" ]] && rm "$SRC_DIR/CMakeLists.txt"
> for ((i = 1; i <= $NUM_LIBRARIES; i++)); do
>     echo "int function$i() { return $i; }" > "$SRC_DIR/function$i.c"
>     echo "add_library(library$i function$i.c)" >> "$SRC_DIR/CMakeLists.txt"
> done
>
> echo "Testcase has been setup: now build with CMake and Ninja generator."
> ---- [1]: END: testcase.sh
> ---------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20120708/ca73122d/attachment.htm>