MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0014874CMakeCMakepublic2014-04-14 11:502015-01-05 08:38
Reporterzub 
Assigned ToBrad King 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
Platformx86_64OSLinuxOS VersionDebian sid
Product VersionCMake 2.8.12.1 
Target VersionCMake 3.1Fixed in VersionCMake 3.1 
Summary0014874: Static library can miss some object files
DescriptionWhen creating a static library using binutils (ar) and there exist a duplicate object name (e.g. a/Foo.cpp.o, b/Foo.cpp.o), the resulting static library can end up having only one of the duplicate objects.

My reading of ar man page implies ar in general is OK with duplicate object names. Also, creating a simple CMake project (just a/foo.cpp, b/foo.cpp, add_library(test STATIC a/foo.cpp b/foo.cpp) results in a correct libtest.a.

This bug only happens if there are many objects. This affects how CMake invokes ar. By default CMake invokes ar for the first time for a library like this:

ar cr libtest.a path/to/a/foo.cpp.o path/to/b/foo.cpp.o ...

and if this is the only invocation, it produces a correct libtest.a. But if there are many objects, CMake invokes ar several times, using the following for subsequent calls:

ar r libtest.a other objects...

Now if it happens that the first duplicate object is added in one invocation of ar, and then CMake invokes ar again to add another, ar just replaces the previous object.

When I set:

set(CMAKE_CXX_ARCHIVE_APPEND "<CMAKE_AR> <LINK_FLAGS> q <TARGET> <OBJECTS>")

to use "q" instead of "r", the issue is resolved for me, so the fix could be just using "q" when adding objects (CMAKE_CXX_ARCHIVE_APPEND, CMAKE_C_ARCHIVE_APPEND and perhaps also CMAKE_CXX_ARCHIVE_CREATE and CMAKE_C_ARCHIVE_CREATE, though this probably won't make a difference).

But I'm not sure how do other ar's handle this. I tested this using "GNU ar (GNU Binutils for Debian) 2.24".
Steps To ReproduceThe issue is deterministic, but it does not occur for every project with duplicate object names - it depends on the project.

To trigger this a project with many objects is required so that CMake in fact does invoke ar multiple times. Also, the duplicate objects must find their way into multiple invocations of ar, e.g.:

ar cr libtest.a objects... a/foo.cpp.o objects...
ar r libtest.a objects... b/foo.cpp.o objects...

Then ar removes the previous foo.cpp.o.

Creating a standalone testcase seems not so trivial; I run into this on a real project. If necessary, I can try to create one.
Additional Information
TagsNo tags attached.
Relationships
related to 0006284closed Brad King Linking static libraries with long list of objects files with Makefile generators fail 
has duplicate 0014994closed  Add an option to change the object file name 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2014-04-14 11:50zubNew Issue
2014-04-14 14:02Brad KingRelationship addedrelated to 0006284
2014-04-14 14:03Brad KingNote Added: 0035695
2014-04-15 03:26Marcel LooseNote Added: 0035702
2014-04-15 03:52zubNote Added: 0035703
2014-04-15 09:02Brad KingNote Added: 0035707
2014-04-15 09:03Brad KingAssigned To => Brad King
2014-04-15 09:03Brad KingStatusnew => resolved
2014-04-15 09:03Brad KingResolutionopen => fixed
2014-04-15 09:03Brad KingFixed in Version => CMake 3.1
2014-04-15 09:03Brad KingTarget Version => CMake 3.1
2014-04-15 17:05zubNote Added: 0035721
2014-04-15 17:05zubStatusresolved => feedback
2014-04-15 17:05zubResolutionfixed => reopened
2014-04-16 07:40Brad KingStatusfeedback => assigned
2014-04-16 07:40Brad KingResolutionreopened => open
2014-04-16 07:40Brad KingFixed in VersionCMake 3.1 =>
2014-04-16 08:12Brad KingNote Added: 0035722
2014-04-16 08:14Brad KingNote Added: 0035723
2014-04-16 08:27zubNote Added: 0035724
2014-04-16 08:34Brad KingNote Added: 0035725
2014-04-16 14:07Brad KingNote Added: 0035730
2014-04-17 10:08Brad KingNote Added: 0035735
2014-04-17 10:08Brad KingStatusassigned => resolved
2014-04-17 10:08Brad KingResolutionopen => fixed
2014-04-17 10:08Brad KingFixed in Version => CMake 3.1
2014-06-26 09:16Nils GladitzRelationship addedhas duplicate 0014994
2015-01-05 08:38Robert MaynardNote Added: 0037571
2015-01-05 08:38Robert MaynardStatusresolved => closed

Notes
(0035695)
Brad King   
2014-04-14 14:03   
Good catch, thanks. I think "q" will work because the only platform information file that uses the separate create/append steps is Windows-GNU.cmake where we know we have the GNU archiver.
(0035702)
Marcel Loose   
2014-04-15 03:26   
Well, there's one problem. `ar` doesn't record path names (see, e.g. https://stackoverflow.com/questions/4907121/static-library-having-object-files-with-same-name-ar [^]). How are you going to distinguish the two identically named object files?

$ touch a.o b.o
$ ar cr foo.a a.o b.o
$ ar tv foo.a
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 b.o
$ mkdir a b
$ touch a/a.o b/b.o
$ ar q foo.a a/a.o b/b.o
$ ar tv foo.a
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 b.o
rw-r--r-- 1000/1000 0 Apr 15 09:24 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:24 2014 b.o
(0035703)
zub   
2014-04-15 03:52   
Yes, the archive ends up with duplicate file names but when I look into binutils ar man page I think this is a supported usecase. binutils ar supports this modifier:

   N Uses the count parameter. This is used if there are multiple
       entries in the archive with the same name. Extract or delete
       instance count of the given name from the archive.

so it is possible to selectively extract individual objects:

$ mkdir a b
$ echo a > a/foo.o
$ echo b > b/foo.o
$ ar q foo.a a/foo.o b/foo.o
ar: creating foo.a
$ ar xvN 1 foo.a foo.o && cat foo.o
x - foo.o
a
$ ar xvN 2 foo.a foo.o && cat foo.o
x - foo.o
b

Indeed, one has to be careful when extracting the objects. But simply using the static library for linking seems to work OK. I think linker doesn't care about the object names.
(0035707)
Brad King   
2014-04-15 09:02   
Patch applied:

 Windows-GNU: Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=39d0ade0 [^]
(0035721)
zub   
2014-04-15 17:05   
Unfortunately the issue is not yet resolved. I can still reproduce it with 8472ef243ffc9988ea8fb83cbc7acdf3f0daa239.

I didn't fully understand Brad King's original comment. The bug happens not only on MinGW, but also on Linux. I experienced the issue in Debian.

There seem to be several places where the ar options are set. Grepping in Modules/ brings up:

* CMakeCInformation.cmake, CMakeCXXInformation.cmake and CMakeFortranInformation.cmake - these set CMAKE_lang_ARCHIVE_... to some default value if not set elsewhere

* Platform/Windows-GNU.cmake - this has been resolved by http://cmake.org/gitweb?p=cmake.git;a=commit;h=39d0ade0 [^]

* Compiler/TI-C.cmake and Compiler/TI-CXX.cmake - for a TI compiler

If I replace the "r" with "q" in the CMakeC/CXX/FotranInformation.cmake I can no longer reproduce the issue, so I believe that fixes it with GCC on Linux. This is the patch I used and tested on my box:

http://linux.fjfi.cvut.cz/~zub/0001-GNU-Support-duplicate-object-names-in-large-archives.patch [^]

(TBH I tested CXX only, but I think it's reasonable to assume the same behavior with C and Fortran.)

As for the TI compiler - it might be good to change it there too but I don't have the TI toolchain and I have no idea about their ar, so I can't test it.
(0035722)
Brad King   
2014-04-16 08:12   
Re 0014874:0035721: Ugh, I don't know how I missed those. We need only to grep for every place that sets the append rule:

$ git grep -l ARCHIVE_APPEND v3.0.0-rc3 -- Modules
v3.0.0-rc3:Modules/CMakeCInformation.cmake
v3.0.0-rc3:Modules/CMakeCXXInformation.cmake
v3.0.0-rc3:Modules/CMakeFortranInformation.cmake
v3.0.0-rc3:Modules/Platform/Windows-GNU.cmake
(0035723)
Brad King   
2014-04-16 08:14   
I think "q" is a POSIX ar option. It appears in "man ar" on several old UNIX machines too. Here is the more general fix:

 Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=f54be6c0 [^]
(0035724)
zub   
2014-04-16 08:27   
That fix will solve the issue for me.

What about Compiler/TI-C.cmake and Compiler/TI-CXX.cmake?
(0035725)
Brad King   
2014-04-16 08:34   
Re 0014874:0035724: The TI modules do not add any append rules, and I don't have the toolchain to test it. See 0014876 for dicussion on a rewrite of the TI modules.
(0035730)
Brad King   
2014-04-16 14:07   
Re 0014874:0035723: I rebased the fix back on the commit it generalizes:

 Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=1ec6485c [^]
(0035735)
Brad King   
2014-04-17 10:08   
The more general fix in 0014874:0035730 is now in 'master' and will be in 3.1.
(0037571)
Robert Maynard   
2015-01-05 08:38   
Closing resolved issues that have not been updated in more than 4 months.