View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005430CMakeCPackpublic2007-07-28 15:282011-01-18 16:33
Reporterirwin 
Assigned ToEric NOULARD 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in VersionCMake 2.8.3 
Summary0005430: cpack does not preserve symbolic links in the source distribution tarball
DescriptionI have some data directories in one of my source trees which are so large that they dominate the size of the source tree. For convenience they are symlinked from a number of different directories in the source tree. However, the source distribution tarball generated by

make package_source

is extremely large (five times the size which it ought to be for this particular case) because the symlinks are not preserved and instead are replaced with copies of the large data directories.

Symlinks are normally put into the source tree for good reasons. (For example, the cvs replacement subversion deals properly with symlinks for exactly this reason.) Thus, cpack should always preserve symlinks when creating a tarball.
TagsNo tags attached.
Attached Filestgz file icon cpack-duplicate-shared-libs.tgz [^] (21,421 bytes) 2010-07-29 06:52
patch file icon 0001-CPack-handle-symlinks-in-CPACK_INSTALLED_DIRECTORIES.patch [^] (4,080 bytes) 2010-07-29 08:37 [Show Content]

 Relationships
related to 0010162closedEric NOULARD CPack Cannot create binary zip archive containing shared libs 
related to 0010096closedBill Hoffman cpack does not preserve symlinks in CPACK_INSTALLED_DIRECTORIES 

  Notes
(0018659)
engels (reporter)
2009-12-02 15:43
edited on: 2009-12-02 15:59

Hi everyone,

I also think symbolic links should always be preserved (not only in source distribution tarballs). When packing shared libraries this can quickly become a major problem, since each library typically has two symbolic links for it's version.

If possible could someone please raise the priority of this issue?

Thanks, Jan.

(0018660)
Bill Hoffman (manager)
2009-12-02 15:59

Is this still a problem? Jan do have a case where shrared libraries are duplicated in cpack?
(0018661)
engels (reporter)
2009-12-02 16:04
edited on: 2009-12-02 16:17

Hi Bill,

sorry, I did not test properly. this problem does not occur when packaging a cmake project. only when using cpack without cmake.

(0018662)
Bill Hoffman (manager)
2009-12-02 16:18

With shared libraries?
(0018663)
engels (reporter)
2009-12-02 16:20
edited on: 2009-12-02 16:56

yes, I created a small example to reproduce the bug, please find it here:
http://www.desy.de/~engels/cpack-duplicate-shared-libs.tgz [^]

to run just type 'make'

(0019594)
engels (reporter)
2010-02-23 09:15

Hi Bill, are there any news on this issue?
(0021480)
Eric NOULARD (developer)
2010-07-26 07:34

I did add a link to a "related bug" concerning symlinks in ZIP archive.
Just to note that for CPackTGZ (binary) package the symlinks are preserved
if CMake+CPack is used.

However symlinks are not preserved if:

package_source
or
direct cpack usage as in the example from engels
(0021481)
Eric NOULARD (developer)
2010-07-26 07:36

Adding one more relationship.
(0021491)
Bill Hoffman (manager)
2010-07-27 09:57

If someone wants to create a patch for this that would be great. I think it is just figuring out what the arguments to libarchive calls need to be for this to work.
(0021493)
Eric NOULARD (developer)
2010-07-27 11:35

Hi Bill,

I've just has a deeper look into this.
It appears that libarchive is not the culprit.
The trouble comes from the usage of "CPACK_INSTALLED_DIRECTORIES"
which is copied by cmake from "INSTALLED DIRS" to
CPack own "CPACK_TEMPORARY_INSTALL_DIRECTORY"

have a look at: int cmCPackGenerator::InstallProjectViaInstalledDirectories.
 which ends up in a:

cmSystemTools::CopyFileIfDifferent

which clearly does not handle symbolic links.

so the loss of symbolic links has nothing to do with libarchive.

libarchive seems to handle symbolic links just fine for TAR archives,
for ZIP it does not (and there is no option for that) see bug 10162.

I don't really know is it would be a desirable behavior for
cmSystemTools::CopyFileIfDifferent to handle symbolic links?

And this is not that easy to do, because:

  - one should check if the target file is part of the files to be package
    or you'll end-up with a dangling link
 
  - does every usage of cmSystemTools::CopyFileIfDifferent implies
    we want to replicate the symlink or do we sometimes want to copy
    the target...

May be the good way to go would be to filter out symlinks
InstallProjectViaInstalledDirectories
and treat them after "regular" files have been copied.
(0021538)
Eric NOULARD (developer)
2010-07-29 08:43

Hi Bill,

You'll find a patch proposal attach.
The patch does modify the behavior of "InstallProjectViaInstalledDirectories":

- if a file is a symlink we do not "CopyFileIfDifferent"
  we remember the name of the link (and the target)

- at the end of the processing of the current INSTALLED dir
  we recreate the links.

The patch is OK on engels example.
It should work if the link are "local" to each directory of INSTALLED dir.
(We may do more to support "cross-directory" links but I'm not sure
 it's worth the extra complexity).

I cannot test how it behaves on platform not supporting symlinks.
However it should be harmless since in this case
FileIsSymlink should always answer "false".


I did re-check that the problem does NOT occurs for in-source links
and "make package_source".
(0021555)
engels (reporter)
2010-07-29 13:12

Hi Eric,

thank you very much for the patch. I did not yet try it but I will asap and hopefully I will then be able to use cpack to also package some external dependencies from our main project.

I tried to also look into fixing this problem, but as I started digging into the code I found out that it was not as trivial as I thought and that I do not yet have enough experience with the cmake source code.

So, once again.. thanks a lot!
Jan
(0021556)
Eric NOULARD (developer)
2010-07-29 13:21

Hi Jan,

No problem.
Tell us if it works for you, this way the patch may find its way to
next CMake release.

And don't be afraid by code digging :-)
(0021993)
Eric NOULARD (developer)
2010-08-28 10:53

Pushed to next:

commit 013e0039eeb7e8bf213c3851161ce9a18f7b940d
Author: Eric NOULARD <eric.noulard@gmail.com>
Date: Sat Aug 28 16:50:06 2010 +0200

    CPack handle symlinks in CPACK_INSTALLED_DIRECTORIES fix for bug5430
    
    The proposed solution is to avoid to CopyIfDifferent any links
    found in CPACK_INSTALLED_DIRECTORIES but memorize them instead
    then at the end of the processing for each dir do re-create the
    link in the installed tree.
    Current patch should work if the link are "local" to the directory.
(0022033)
Eric NOULARD (developer)
2010-08-31 16:32

> The proposed solution is to avoid to CopyIfDifferent any links
> found in CPACK_INSTALLED_DIRECTORIES but memorize them instead
> then at the end of the processing for each dir do re-create the
> link in the installed tree.
> Current patch should work if the link are "local" to the directory.

This brings up another place for refactoring. It looks like the code
this patch modifies does the same thing that the file(COPY) and
file(INSTALL) commands do. In cmFileCommand.cxx there is a class
cmFileCopier and subclass cmFileInstaller used to implement those
commands. These could be factored out of this file and used in
CPack to install the directories into the package more robustly.
(0024862)
David Cole (manager)
2011-01-18 11:18

Looking at "older" "feedback" status bugs today... is this still an issue that we want to address moving forward? If so, please remove the "feedback" status and just set it to "assigned"... If not, please resolve it.

Thanks.
(0024883)
engels (reporter)
2011-01-18 12:40
edited on: 2011-01-18 12:51

From my side this issue can be closed. The problem was fixed in version 2.8.3. Sorry, reading back previous posts I just realized that I forgot to give feedback to Eric regarding the patch. Sorry

(0024891)
Alan W. Irwin (reporter)
2011-01-18 14:11

I am the original reporter of this bug, but I am currently not in a good
position to test the fix. However, from the above notes it does appear to be
fixed in 2.8.3 so please go ahead and close it. I will open another bug
if it turns out there are still some issues left once I can test it
again.
(0024892)
David Cole (manager)
2011-01-18 14:17

Eric,

Was there a reason you wanted to leave this one open still, despite the original reporters saying the issue is fixed...?

Something about what's in the notes trail?

If not, please go ahead and resolve it as fixed in 2.8.3 or 2.8.4, whichever is appropriate.

Thanks,
David
(0024894)
Eric NOULARD (developer)
2011-01-18 15:35

I'll check later tonight.

As far as I remember my patch didn't reached master
(not merged from next) because the
patch was doing something similar to other CMake code which
was more up-to-date.

I'll check the status of current master (later tonight)
and close it if it was finally merged.
(0024898)
Eric NOULARD (developer)
2011-01-18 16:33

Now I remember the comment from Brad:
http://public.kitware.com/pipermail/cmake-developers/2010-August/000537.html [^]

I thought that the remark meant this hasn't been merged,
in fact the fix was merged in 2.8.3.

I did reopen the bug for auto-planning the proposed refactoring
and didn't have time to do it.
Closing the bug refactoring will have to wait.

 Issue History
Date Modified Username Field Change
2007-08-31 11:27 Alex Neundorf Assigned To System Admin => Bill Hoffman
2009-12-02 15:43 engels Note Added: 0018659
2009-12-02 15:59 engels Note Edited: 0018659
2009-12-02 15:59 Bill Hoffman Note Added: 0018660
2009-12-02 16:04 engels Note Added: 0018661
2009-12-02 16:17 engels Note Edited: 0018661
2009-12-02 16:18 Bill Hoffman Note Added: 0018662
2009-12-02 16:20 engels Note Added: 0018663
2009-12-02 16:56 engels Note Edited: 0018663
2010-02-23 09:15 engels Note Added: 0019594
2010-06-18 12:50 David Cole Summary cpack does not preserve symbolic links in the source distribution tarball => cpack does not preserve symbolic links in the source distribution tarball
2010-07-26 07:29 Eric NOULARD Relationship added related to 0010162
2010-07-26 07:34 Eric NOULARD Note Added: 0021480
2010-07-26 07:35 Eric NOULARD Relationship added related to 0010096
2010-07-26 07:36 Eric NOULARD Note Added: 0021481
2010-07-27 09:57 Bill Hoffman Note Added: 0021491
2010-07-27 11:35 Eric NOULARD Note Added: 0021493
2010-07-29 06:52 Eric NOULARD File Added: cpack-duplicate-shared-libs.tgz
2010-07-29 08:37 Eric NOULARD File Added: 0001-CPack-handle-symlinks-in-CPACK_INSTALLED_DIRECTORIES.patch
2010-07-29 08:43 Eric NOULARD Note Added: 0021538
2010-07-29 13:12 engels Note Added: 0021555
2010-07-29 13:21 Eric NOULARD Note Added: 0021556
2010-08-28 10:51 Eric NOULARD Assigned To Bill Hoffman => Eric NOULARD
2010-08-28 10:53 Eric NOULARD Note Added: 0021993
2010-08-28 10:53 Eric NOULARD Status assigned => closed
2010-08-28 10:53 Eric NOULARD Resolution open => fixed
2010-08-28 10:53 Eric NOULARD Fixed in Version => CMake-2-8
2010-08-31 16:32 Eric NOULARD Note Added: 0022033
2010-08-31 16:32 Eric NOULARD Status closed => feedback
2010-08-31 16:32 Eric NOULARD Resolution fixed => reopened
2010-10-06 14:24 David Cole Fixed in Version CMake-2-8 =>
2011-01-18 11:18 David Cole Note Added: 0024862
2011-01-18 12:40 engels Note Added: 0024883
2011-01-18 12:51 engels Note Edited: 0024883
2011-01-18 14:11 Alan W. Irwin Note Added: 0024891
2011-01-18 14:17 David Cole Note Added: 0024892
2011-01-18 15:35 Eric NOULARD Note Added: 0024894
2011-01-18 16:33 Eric NOULARD Note Added: 0024898
2011-01-18 16:33 Eric NOULARD Status feedback => closed
2011-01-18 16:33 Eric NOULARD Resolution reopened => fixed
2011-01-18 16:33 Eric NOULARD Fixed in Version => CMake 2.8.3


Copyright © 2000 - 2018 MantisBT Team