MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0011020CMakeCPackpublic2010-07-21 08:552010-09-10 00:09
ReporterThomas Themel 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product VersionCMake-2-8 
Target VersionCMake 2.8.3Fixed in VersionCMake 2.8.3 
Summary0011020: CPack produces uninstallable DEB files with 2.8.2, worked with 2.8.1
DescriptionHi,

I've recently upgraded from CMake 2.8.1 to 2.8.2, and my previously-working CPack package builds now produce uninstallable packages due to missing directories in the package file.

Steps To Reproduce
Additional InformationHere's what I get with 2.8.1:

themel@kallisti:~/work/qkd-network/q3p-device/build$ dpkg-deb -c ../../q3p-net/build/q3p-net-0.3-x86_64.deb | head
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/
-rw-r--r-- themel/themel 70036 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html
-rw-r--r-- themel/themel 3357 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/todo.html
-rw-r--r-- themel/themel 5749 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html
-rw-r--r-- themel/themel 2578 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/globals.html
-rw-r--r-- themel/themel 2585 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif
tar: write error
dpkg-deb: subprocess tar returned error exit status 2

2.8.2 gives me:

themel@kallisti:~/work/qkd-network/q3p-net/build$ dpkg-deb -c q3p-net-0.3-x86_64.deb | head
-rw-r--r-- themel/themel 70036 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html
-rw-r--r-- themel/themel 3357 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/todo.html
-rw-r--r-- themel/themel 5749 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html
-rw-r--r-- themel/themel 2578 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals.html
-rw-r--r-- themel/themel 2585 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif
-rw-r--r-- themel/themel 5902 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/control__cmds_8h_source.html
-rw-r--r-- themel/themel 23274 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_eval.html
-rw-r--r-- themel/themel 4074 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_defs.html
-rw-r--r-- themel/themel 8520 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/functions.html
-rw-r--r-- themel/themel 2971 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_enum.html

Trying to install the 2.8.2 package predictably fails when the directories do not exist:

themel@kallisti:~/work/qkd-network/q3p-net/build$ sudo dpkg -i q3p-net-0.3-x86_64.deb
(Reading database ... 372372 files and directories currently installed.)
Preparing to replace q3p-net 0.3.0 (using q3p-net-0.3-x86_64.deb) ...
Unpacking replacement q3p-net ...
dpkg: error processing q3p-net-0.3-x86_64.deb (--install):
 unable to create `/usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html.dpkg-new' (while processing `./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html'): No such file or directory
dpkg-deb: subprocess paste killed by signal (Broken pipe)
Errors were encountered while processing:
 q3p-net-0.3-x86_64.deb
themel@kallisti:~/work/qkd-network/q3p-net/build$

TagsNo tags attached.
Relationships
has duplicate 0011170closed Brad King 'cmake -E tar cfz' does not include preceding paths in archive but 'tar cfz' does on CentOS5.5 
related to 0011176closed Brad King "cmake -E tar" encodes long paths such that Fedora GNU tar 1.15.1 reports errors 
related to 0011958closed Brad King libarchive still produces PAX tar archives which are unreadable on older versions of tar 
Attached Filesgz cpack-bug-0.1.tar.gz (2,281) 2010-07-21 11:03
https://public.kitware.com/Bug/file/3258/cpack-bug-0.1.tar.gz
patch cmake-ustar.patch (626) 2010-08-02 12:12
https://public.kitware.com/Bug/file/3291/cmake-ustar.patch
diff cmake-tar-directories.diff (3,088) 2010-08-03 11:09
https://public.kitware.com/Bug/file/3294/cmake-tar-directories.diff
Issue History
Date ModifiedUsernameFieldChange
2010-07-21 08:55Thomas ThemelNew Issue
2010-07-21 10:30Bill HoffmanNote Added: 0021436
2010-07-21 10:30Bill HoffmanStatusnew => assigned
2010-07-21 10:30Bill HoffmanAssigned To => Bill Hoffman
2010-07-21 11:02Thomas ThemelNote Added: 0021439
2010-07-21 11:03Thomas ThemelFile Added: cpack-bug-0.1.tar.gz
2010-07-21 12:04Bill HoffmanNote Added: 0021440
2010-07-21 16:07Thomas ThemelNote Added: 0021443
2010-08-02 11:46simonhNote Added: 0021595
2010-08-02 12:12simonhFile Added: cmake-ustar.patch
2010-08-02 12:14simonhNote Added: 0021597
2010-08-03 09:19Brad KingNote Added: 0021602
2010-08-03 09:47Brad KingNote Added: 0021603
2010-08-03 09:47Brad KingNote Edited: 0021603
2010-08-03 09:50Brad KingNote Added: 0021604
2010-08-03 10:02simonhNote Added: 0021605
2010-08-03 10:04simonhNote Edited: 0021605
2010-08-03 10:36Brad KingNote Added: 0021606
2010-08-03 10:45Brad KingNote Added: 0021607
2010-08-03 11:09Thomas ThemelFile Added: cmake-tar-directories.diff
2010-08-03 11:11Thomas ThemelNote Added: 0021609
2010-08-03 11:14Brad KingNote Added: 0021610
2010-08-03 11:15Brad KingNote Edited: 0021610
2010-08-03 12:38simonhNote Added: 0021611
2010-08-03 14:45Thomas ThemelNote Edited: 0021609
2010-08-04 05:04simonhNote Added: 0021626
2010-08-04 07:54Brad KingNote Added: 0021627
2010-08-04 07:58simonhNote Added: 0021628
2010-08-04 08:14simonhNote Added: 0021629
2010-08-04 08:23simonhNote Added: 0021630
2010-08-05 17:53Brad KingAssigned ToBill Hoffman => Brad King
2010-08-06 15:43Brad KingNote Added: 0021689
2010-08-09 10:05Brad KingNote Added: 0021697
2010-08-25 09:33Brad KingRelationship addedhas duplicate 0011170
2010-08-26 13:38Brad KingRelationship addedrelated to 0011176
2010-08-26 13:40Brad KingNote Added: 0021977
2010-08-26 13:41Brad KingStatusassigned => closed
2010-08-26 13:41Brad KingResolutionopen => fixed
2010-08-31 18:02David ColeTarget Version => CMake 2.8.3
2010-09-10 00:09David ColeFixed in Version => CMake 2.8.3
2011-03-10 08:50Brad KingRelationship addedrelated to 0011958

Notes
(0021436)
Bill Hoffman   
2010-07-21 10:30   
I am going to need an example. My guess would be the install command is not working the same. What is in the _CPACK directory for each of these?
(0021439)
Thomas Themel   
2010-07-21 11:02   
Okay, here's a very simple example that illustrates the problem:

Again, 2.8.1:
themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb
drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/
drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/bin/
-rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug

With 2.8.2:
themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb
-rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug

Btw, all this on an am64 Debian Sid, with cmake 2.8.1/2.8.2 compiled from source.
(0021440)
Bill Hoffman   
2010-07-21 12:04   
Looks like an issue with the switch to libarchive.... I don't have a ton of time right now, if you could figure out the patch I could get it done quick. Looks like the cmake -E tar command is not keeping the directories anymore. We used to use libtar, and switched for 2.8.2 to libarchive.
(0021443)
Thomas Themel   
2010-07-21 16:07   
Mhm, the current cmSystemTools::CreateTar implementation looks exactly like it would produce the current behaviour. I'll look into it.
(0021595)
simonh   
2010-08-02 11:46   
I've just encountered a similar bug with 2.8.2. Specifically, there are older versions of tar which choke on the extensions which libarchive is now using.

See e.g. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-05/msg00700.html [^]

My tar outputs:
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.security.selinux'

... and finally exits with code 2 (instead of just issuing a warning and exiting with 0)
(0021597)
simonh   
2010-08-02 12:14   
I've attached a patch which works for me. After reading the libarchive docs (such as http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] ), I changed the CPack archive generator to use "ustar" output format, not the restricted pax format.
(0021602)
Brad King   
2010-08-03 09:19   
The current code came from commit 22fb266d (use different tar format to handle longer file names, 2009-11-14):

  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=22fb266d [^]

The commit switched from ustar to pax on purpose.

Here is a simple experiment:

$ mkdir foo
$ touch foo/a
$ tar cf foo.tar foo
$ tar tf foo.tar
foo/
foo/a
$ cmake -E tar tf foo.tar foo
foo/
foo/a

If I use CMake to create the tarball instead:

$ cmake -E tar cf foo.tar foo
$ tar tf foo.tar
foo/a
$ cmake -E tar tf foo.tar foo
foo/a

I tried applying your change to cmSystemTools::CreateTar (pax_restricted -> ustar) but the above experiment produces the same result.
(0021603)
Brad King   
2010-08-03 09:47   
Even the example in http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] produces just "foo/a" unless one explicitly lists "foo" on the command line.

Looking back at the old libtar-based implementation, it looks like libtar automatically handles directories so CMake's code could just send in those paths. Now the libarchive-based cmSystemTools::CreateTar does its own traversal. It uses a recursive glob search that only returns files and leaves out all their directories. This needs to be corrected to do a proper recursive directory traversal.
(0021604)
Brad King   
2010-08-03 09:50   
It looks like cmCPackGenerator::DoPackage also uses the recursive glob and does not return directories.
(0021605)
simonh   
2010-08-03 10:02   
(edited on: 2010-08-03 10:04)
How do we solve the problem that pax archives are not readable without error on older GNU tar implementations? Is there an actual bug report that indicates that 255 character filenames are actually needed, bearing in mind that these are always relative to the top-level archive directory (i.e. not whole filesystems)?

I'd always rather make the tar archives as portable as possible by default, and only add additional, more-specific variants (e.g. pax) as required.

BTW, I now realise this is an orthogonal issue? Should I report a separate bug?
(0021606)
Brad King   
2010-08-03 10:36   
Let's just keep the discussion together here.

The old libtar implementation did this:

void th_set_path(TAR *t, char *pathname)
{
  if (strlen(pathname)+strlen(suffix) >= T_NAMELEN && (t->options & TAR_GNU))
    { /* GNU-style long name */ }
  else if (strlen(pathname)+ strlen(suffix) >= T_NAMELEN)
    { /* POSIX-style prefix field */ }
  else
    { /* classic tar format */ }
}

In other words, it dynamically switched the entry type for each path.
(0021607)
Brad King   
2010-08-03 10:45   
On the other hand, the libtar-based implementation seems to produce bad archives when there is a very, very deep name. In a simple test with a 400-character long path the file did not extract correctly. It does extract correctly from an archive created with the libarchive-based implementation (with more recent GNU tar).
(0021609)
Thomas Themel   
2010-08-03 11:11   
(edited on: 2010-08-03 14:45)
I have attached a patch that fixes the (originally reported, not the path name length) issue in cmSystemTools by ensuring that every created file has all the parent directories it needs. Works for me, but please note that it seems that cmCPackArchiveGenerator::CompressFiles will exhibit the same issue for TGZ files and will need to be fixed in a similar way. Maybe it would be better to refactor so as to have a single place that turns a list of files into a tar file?
(0021610)
Brad King   
2010-08-03 11:14   
(edited on: 2010-08-03 11:15)
Yes, refactoring would be better. I think it should just be a function that takes the top-level directory by full path, a list of input paths, and a "struct archive*". Then it should load the directories and recurse itself.
(0021611)
simonh   
2010-08-03 12:38   
I'll take a look tomorrow at whether unrestricted pax archives are supported by older GNU tars. Versions 1.14 and above claim to support POSIX.1-2001.

I'll try "archive_write_set_format_pax"...
(0021626)
simonh   
2010-08-04 05:04   
Nope, that didn't help. I can't read the generated tar archives without a non-zero exit status from the system-installed "tar" program on the following Linux distributions:

Fedora Core 7
Red Hat Enterprise Linux 5

Some other distros (such as RHEL4) are fine, though...
(0021627)
Brad King   
2010-08-04 07:54   
What version of GNU tar (tar --version) are on the systems that do not work?
(0021628)
simonh   
2010-08-04 07:58   
tar (GNU tar) 1.15.1
(0021629)
simonh   
2010-08-04 08:14   
But note that Red Hat has heavily patched the tar sources; I think the Fedora/RHEL version has 30-odd different RPM-releases for the same 1.15.1 upstream source.

You can find the whole source + patches here: http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/7/Fedora/source/SRPMS/tar-1.15.1-26.fc7.src.rpm [^]
(0021630)
simonh   
2010-08-04 08:23   
Having dug into the Red Hat SRPM, their xattr patch raises an ERROR() macro whenever an unknown extend header keyword is encountered (not a WARN() macro).

They only support:
SCHILY.xattr.security.selinux, SCHILY.xattr.system.posix_acl_access, SCHILY.xattr.system.posix_acl_default, SCHILY.xattr.user,SCHILY.xattr.root for the Schilling extensions.
(0021689)
Brad King   
2010-08-06 15:43   
I pushed out a work-in-progress topic to address the directory entry problem with "cmake -E tar":

http://github.com/bradking/CMake/commits/libarchive-wrapper/ [^]

The .deb generator actually uses this to create the package (it does not derive from cmCPackArchiveGenerator).
(0021697)
Brad King   
2010-08-09 10:05   
The new "cmake -E tar" produces

  ./usr/
  ./usr/bin/
  ./usr/bin/cpack-bug

in the .deb file. I've merged the topic into next:

  http://cmake.org/gitweb?p=cmake.git;a=log;h=1b5b2ed3 [^]
(0021977)
Brad King   
2010-08-26 13:40   
simonh: Now that the original bug in this issue has been fixed I decided that your pax v. ustar issue deserves its own issue as you suggested earlier. I've opened 0011176 for discussion on that problem.

Now I'm closing this issue since it is fixed.