View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0011020CMakeCPackpublic2010-07-21 08:552010-09-10 00:09
ReporterThomas Themel 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product VersionCMake-2-8 
Target VersionCMake 2.8.3Fixed in VersionCMake 2.8.3 
Summary0011020: CPack produces uninstallable DEB files with 2.8.2, worked with 2.8.1
DescriptionHi,

I've recently upgraded from CMake 2.8.1 to 2.8.2, and my previously-working CPack package builds now produce uninstallable packages due to missing directories in the package file.

Additional InformationHere's what I get with 2.8.1:

themel@kallisti:~/work/qkd-network/q3p-device/build$ dpkg-deb -c ../../q3p-net/build/q3p-net-0.3-x86_64.deb | head
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/
drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/
-rw-r--r-- themel/themel 70036 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html
-rw-r--r-- themel/themel 3357 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/todo.html
-rw-r--r-- themel/themel 5749 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html
-rw-r--r-- themel/themel 2578 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/globals.html
-rw-r--r-- themel/themel 2585 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif
tar: write error
dpkg-deb: subprocess tar returned error exit status 2

2.8.2 gives me:

themel@kallisti:~/work/qkd-network/q3p-net/build$ dpkg-deb -c q3p-net-0.3-x86_64.deb | head
-rw-r--r-- themel/themel 70036 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html
-rw-r--r-- themel/themel 3357 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/todo.html
-rw-r--r-- themel/themel 5749 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html
-rw-r--r-- themel/themel 2578 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals.html
-rw-r--r-- themel/themel 2585 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif
-rw-r--r-- themel/themel 5902 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/control__cmds_8h_source.html
-rw-r--r-- themel/themel 23274 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_eval.html
-rw-r--r-- themel/themel 4074 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_defs.html
-rw-r--r-- themel/themel 8520 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/functions.html
-rw-r--r-- themel/themel 2971 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_enum.html

Trying to install the 2.8.2 package predictably fails when the directories do not exist:

themel@kallisti:~/work/qkd-network/q3p-net/build$ sudo dpkg -i q3p-net-0.3-x86_64.deb
(Reading database ... 372372 files and directories currently installed.)
Preparing to replace q3p-net 0.3.0 (using q3p-net-0.3-x86_64.deb) ...
Unpacking replacement q3p-net ...
dpkg: error processing q3p-net-0.3-x86_64.deb (--install):
 unable to create `/usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html.dpkg-new' (while processing `./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html'): No such file or directory
dpkg-deb: subprocess paste killed by signal (Broken pipe)
Errors were encountered while processing:
 q3p-net-0.3-x86_64.deb
themel@kallisti:~/work/qkd-network/q3p-net/build$

TagsNo tags attached.
Attached Filesgz file icon cpack-bug-0.1.tar.gz [^] (2,281 bytes) 2010-07-21 11:03
patch file icon cmake-ustar.patch [^] (626 bytes) 2010-08-02 12:12 [Show Content]
diff file icon cmake-tar-directories.diff [^] (3,088 bytes) 2010-08-03 11:09 [Show Content]

 Relationships
has duplicate 0011170closedBrad King 'cmake -E tar cfz' does not include preceding paths in archive but 'tar cfz' does on CentOS5.5 
related to 0011176closedBrad King "cmake -E tar" encodes long paths such that Fedora GNU tar 1.15.1 reports errors 
related to 0011958closedBrad King libarchive still produces PAX tar archives which are unreadable on older versions of tar 

  Notes
(0021436)
Bill Hoffman (manager)
2010-07-21 10:30

I am going to need an example. My guess would be the install command is not working the same. What is in the _CPACK directory for each of these?
(0021439)
Thomas Themel (reporter)
2010-07-21 11:02

Okay, here's a very simple example that illustrates the problem:

Again, 2.8.1:
themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb
drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/
drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/bin/
-rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug

With 2.8.2:
themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb
-rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug

Btw, all this on an am64 Debian Sid, with cmake 2.8.1/2.8.2 compiled from source.
(0021440)
Bill Hoffman (manager)
2010-07-21 12:04

Looks like an issue with the switch to libarchive.... I don't have a ton of time right now, if you could figure out the patch I could get it done quick. Looks like the cmake -E tar command is not keeping the directories anymore. We used to use libtar, and switched for 2.8.2 to libarchive.
(0021443)
Thomas Themel (reporter)
2010-07-21 16:07

Mhm, the current cmSystemTools::CreateTar implementation looks exactly like it would produce the current behaviour. I'll look into it.
(0021595)
simonh (reporter)
2010-08-02 11:46

I've just encountered a similar bug with 2.8.2. Specifically, there are older versions of tar which choke on the extensions which libarchive is now using.

See e.g. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-05/msg00700.html [^]

My tar outputs:
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.security.selinux'

... and finally exits with code 2 (instead of just issuing a warning and exiting with 0)
(0021597)
simonh (reporter)
2010-08-02 12:14

I've attached a patch which works for me. After reading the libarchive docs (such as http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] ), I changed the CPack archive generator to use "ustar" output format, not the restricted pax format.
(0021602)
Brad King (manager)
2010-08-03 09:19

The current code came from commit 22fb266d (use different tar format to handle longer file names, 2009-11-14):

  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=22fb266d [^]

The commit switched from ustar to pax on purpose.

Here is a simple experiment:

$ mkdir foo
$ touch foo/a
$ tar cf foo.tar foo
$ tar tf foo.tar
foo/
foo/a
$ cmake -E tar tf foo.tar foo
foo/
foo/a

If I use CMake to create the tarball instead:

$ cmake -E tar cf foo.tar foo
$ tar tf foo.tar
foo/a
$ cmake -E tar tf foo.tar foo
foo/a

I tried applying your change to cmSystemTools::CreateTar (pax_restricted -> ustar) but the above experiment produces the same result.
(0021603)
Brad King (manager)
2010-08-03 09:47
edited on: 2010-08-03 09:47

Even the example in http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] produces just "foo/a" unless one explicitly lists "foo" on the command line.

Looking back at the old libtar-based implementation, it looks like libtar automatically handles directories so CMake's code could just send in those paths. Now the libarchive-based cmSystemTools::CreateTar does its own traversal. It uses a recursive glob search that only returns files and leaves out all their directories. This needs to be corrected to do a proper recursive directory traversal.

(0021604)
Brad King (manager)
2010-08-03 09:50

It looks like cmCPackGenerator::DoPackage also uses the recursive glob and does not return directories.
(0021605)
simonh (reporter)
2010-08-03 10:02
edited on: 2010-08-03 10:04

How do we solve the problem that pax archives are not readable without error on older GNU tar implementations? Is there an actual bug report that indicates that 255 character filenames are actually needed, bearing in mind that these are always relative to the top-level archive directory (i.e. not whole filesystems)?

I'd always rather make the tar archives as portable as possible by default, and only add additional, more-specific variants (e.g. pax) as required.

BTW, I now realise this is an orthogonal issue? Should I report a separate bug?

(0021606)
Brad King (manager)
2010-08-03 10:36

Let's just keep the discussion together here.

The old libtar implementation did this:

void th_set_path(TAR *t, char *pathname)
{
  if (strlen(pathname)+strlen(suffix) >= T_NAMELEN && (t->options & TAR_GNU))
    { /* GNU-style long name */ }
  else if (strlen(pathname)+ strlen(suffix) >= T_NAMELEN)
    { /* POSIX-style prefix field */ }
  else
    { /* classic tar format */ }
}

In other words, it dynamically switched the entry type for each path.
(0021607)
Brad King (manager)
2010-08-03 10:45

On the other hand, the libtar-based implementation seems to produce bad archives when there is a very, very deep name. In a simple test with a 400-character long path the file did not extract correctly. It does extract correctly from an archive created with the libarchive-based implementation (with more recent GNU tar).
(0021609)
Thomas Themel (reporter)
2010-08-03 11:11
edited on: 2010-08-03 14:45

I have attached a patch that fixes the (originally reported, not the path name length) issue in cmSystemTools by ensuring that every created file has all the parent directories it needs. Works for me, but please note that it seems that cmCPackArchiveGenerator::CompressFiles will exhibit the same issue for TGZ files and will need to be fixed in a similar way. Maybe it would be better to refactor so as to have a single place that turns a list of files into a tar file?

(0021610)
Brad King (manager)
2010-08-03 11:14
edited on: 2010-08-03 11:15

Yes, refactoring would be better. I think it should just be a function that takes the top-level directory by full path, a list of input paths, and a "struct archive*". Then it should load the directories and recurse itself.

(0021611)
simonh (reporter)
2010-08-03 12:38

I'll take a look tomorrow at whether unrestricted pax archives are supported by older GNU tars. Versions 1.14 and above claim to support POSIX.1-2001.

I'll try "archive_write_set_format_pax"...
(0021626)
simonh (reporter)
2010-08-04 05:04

Nope, that didn't help. I can't read the generated tar archives without a non-zero exit status from the system-installed "tar" program on the following Linux distributions:

Fedora Core 7
Red Hat Enterprise Linux 5

Some other distros (such as RHEL4) are fine, though...
(0021627)
Brad King (manager)
2010-08-04 07:54

What version of GNU tar (tar --version) are on the systems that do not work?
(0021628)
simonh (reporter)
2010-08-04 07:58

tar (GNU tar) 1.15.1
(0021629)
simonh (reporter)
2010-08-04 08:14

But note that Red Hat has heavily patched the tar sources; I think the Fedora/RHEL version has 30-odd different RPM-releases for the same 1.15.1 upstream source.

You can find the whole source + patches here: http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/7/Fedora/source/SRPMS/tar-1.15.1-26.fc7.src.rpm [^]
(0021630)
simonh (reporter)
2010-08-04 08:23

Having dug into the Red Hat SRPM, their xattr patch raises an ERROR() macro whenever an unknown extend header keyword is encountered (not a WARN() macro).

They only support:
SCHILY.xattr.security.selinux, SCHILY.xattr.system.posix_acl_access, SCHILY.xattr.system.posix_acl_default, SCHILY.xattr.user,SCHILY.xattr.root for the Schilling extensions.
(0021689)
Brad King (manager)
2010-08-06 15:43

I pushed out a work-in-progress topic to address the directory entry problem with "cmake -E tar":

http://github.com/bradking/CMake/commits/libarchive-wrapper/ [^]

The .deb generator actually uses this to create the package (it does not derive from cmCPackArchiveGenerator).
(0021697)
Brad King (manager)
2010-08-09 10:05

The new "cmake -E tar" produces

  ./usr/
  ./usr/bin/
  ./usr/bin/cpack-bug

in the .deb file. I've merged the topic into next:

  http://cmake.org/gitweb?p=cmake.git;a=log;h=1b5b2ed3 [^]
(0021977)
Brad King (manager)
2010-08-26 13:40

simonh: Now that the original bug in this issue has been fixed I decided that your pax v. ustar issue deserves its own issue as you suggested earlier. I've opened 0011176 for discussion on that problem.

Now I'm closing this issue since it is fixed.

 Issue History
Date Modified Username Field Change
2010-07-21 08:55 Thomas Themel New Issue
2010-07-21 10:30 Bill Hoffman Note Added: 0021436
2010-07-21 10:30 Bill Hoffman Status new => assigned
2010-07-21 10:30 Bill Hoffman Assigned To => Bill Hoffman
2010-07-21 11:02 Thomas Themel Note Added: 0021439
2010-07-21 11:03 Thomas Themel File Added: cpack-bug-0.1.tar.gz
2010-07-21 12:04 Bill Hoffman Note Added: 0021440
2010-07-21 16:07 Thomas Themel Note Added: 0021443
2010-08-02 11:46 simonh Note Added: 0021595
2010-08-02 12:12 simonh File Added: cmake-ustar.patch
2010-08-02 12:14 simonh Note Added: 0021597
2010-08-03 09:19 Brad King Note Added: 0021602
2010-08-03 09:47 Brad King Note Added: 0021603
2010-08-03 09:47 Brad King Note Edited: 0021603
2010-08-03 09:50 Brad King Note Added: 0021604
2010-08-03 10:02 simonh Note Added: 0021605
2010-08-03 10:04 simonh Note Edited: 0021605
2010-08-03 10:36 Brad King Note Added: 0021606
2010-08-03 10:45 Brad King Note Added: 0021607
2010-08-03 11:09 Thomas Themel File Added: cmake-tar-directories.diff
2010-08-03 11:11 Thomas Themel Note Added: 0021609
2010-08-03 11:14 Brad King Note Added: 0021610
2010-08-03 11:15 Brad King Note Edited: 0021610
2010-08-03 12:38 simonh Note Added: 0021611
2010-08-03 14:45 Thomas Themel Note Edited: 0021609
2010-08-04 05:04 simonh Note Added: 0021626
2010-08-04 07:54 Brad King Note Added: 0021627
2010-08-04 07:58 simonh Note Added: 0021628
2010-08-04 08:14 simonh Note Added: 0021629
2010-08-04 08:23 simonh Note Added: 0021630
2010-08-05 17:53 Brad King Assigned To Bill Hoffman => Brad King
2010-08-06 15:43 Brad King Note Added: 0021689
2010-08-09 10:05 Brad King Note Added: 0021697
2010-08-25 09:33 Brad King Relationship added has duplicate 0011170
2010-08-26 13:38 Brad King Relationship added related to 0011176
2010-08-26 13:40 Brad King Note Added: 0021977
2010-08-26 13:41 Brad King Status assigned => closed
2010-08-26 13:41 Brad King Resolution open => fixed
2010-08-31 18:02 David Cole Target Version => CMake 2.8.3
2010-09-10 00:09 David Cole Fixed in Version => CMake 2.8.3
2011-03-10 08:50 Brad King Relationship added related to 0011958


Copyright © 2000 - 2018 MantisBT Team