View Issue Details [ Jump to Notes ] | [ Print ] | ||||||||
ID | Project | Category | View Status | Date Submitted | Last Update | ||||
0011020 | CMake | CPack | public | 2010-07-21 08:55 | 2010-09-10 00:09 | ||||
Reporter | Thomas Themel | ||||||||
Assigned To | Brad King | ||||||||
Priority | normal | Severity | major | Reproducibility | always | ||||
Status | closed | Resolution | fixed | ||||||
Platform | OS | OS Version | |||||||
Product Version | CMake-2-8 | ||||||||
Target Version | CMake 2.8.3 | Fixed in Version | CMake 2.8.3 | ||||||
Summary | 0011020: CPack produces uninstallable DEB files with 2.8.2, worked with 2.8.1 | ||||||||
Description | Hi, I've recently upgraded from CMake 2.8.1 to 2.8.2, and my previously-working CPack package builds now produce uninstallable packages due to missing directories in the package file. | ||||||||
Additional Information | Here's what I get with 2.8.1: themel@kallisti:~/work/qkd-network/q3p-device/build$ dpkg-deb -c ../../q3p-net/build/q3p-net-0.3-x86_64.deb | head drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/ drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/ drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/ drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/ drwxr-xr-x themel/themel 0 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/ -rw-r--r-- themel/themel 70036 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html -rw-r--r-- themel/themel 3357 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/todo.html -rw-r--r-- themel/themel 5749 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html -rw-r--r-- themel/themel 2578 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/globals.html -rw-r--r-- themel/themel 2585 2010-07-20 10:52 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif tar: write error dpkg-deb: subprocess tar returned error exit status 2 2.8.2 gives me: themel@kallisti:~/work/qkd-network/q3p-net/build$ dpkg-deb -c q3p-net-0.3-x86_64.deb | head -rw-r--r-- themel/themel 70036 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html -rw-r--r-- themel/themel 3357 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/todo.html -rw-r--r-- themel/themel 5749 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/q3p__dispatcher__command_8h.html -rw-r--r-- themel/themel 2578 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals.html -rw-r--r-- themel/themel 2585 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/tab_r.gif -rw-r--r-- themel/themel 5902 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/control__cmds_8h_source.html -rw-r--r-- themel/themel 23274 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_eval.html -rw-r--r-- themel/themel 4074 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_defs.html -rw-r--r-- themel/themel 8520 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/functions.html -rw-r--r-- themel/themel 2971 2010-07-21 14:51 ./usr/share/doc/q3p-net-0.3/html/globals_enum.html Trying to install the 2.8.2 package predictably fails when the directories do not exist: themel@kallisti:~/work/qkd-network/q3p-net/build$ sudo dpkg -i q3p-net-0.3-x86_64.deb (Reading database ... 372372 files and directories currently installed.) Preparing to replace q3p-net 0.3.0 (using q3p-net-0.3-x86_64.deb) ... Unpacking replacement q3p-net ... dpkg: error processing q3p-net-0.3-x86_64.deb (--install): unable to create `/usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html.dpkg-new' (while processing `./usr/share/doc/q3p-net-0.3/html/q3p__connection_8h.html'): No such file or directory dpkg-deb: subprocess paste killed by signal (Broken pipe) Errors were encountered while processing: q3p-net-0.3-x86_64.deb themel@kallisti:~/work/qkd-network/q3p-net/build$ | ||||||||
Tags | No tags attached. | ||||||||
Attached Files | cpack-bug-0.1.tar.gz [^] (2,281 bytes) 2010-07-21 11:03 cmake-ustar.patch [^] (626 bytes) 2010-08-02 12:12 [Show Content] cmake-tar-directories.diff [^] (3,088 bytes) 2010-08-03 11:09 [Show Content] | ||||||||
Relationships | ||||||||||||||||
|
Relationships |
Notes | |
(0021436) Bill Hoffman (manager) 2010-07-21 10:30 |
I am going to need an example. My guess would be the install command is not working the same. What is in the _CPACK directory for each of these? |
(0021439) Thomas Themel (reporter) 2010-07-21 11:02 |
Okay, here's a very simple example that illustrates the problem: Again, 2.8.1: themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/ drwxr-xr-x themel/themel 0 2010-07-21 16:59 ./usr/bin/ -rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug With 2.8.2: themel@kallisti:~/src/cpack-bug/build$ dpkg-deb -c cpack-bug-0.1-x86_64.deb -rwxr-xr-x themel/themel 7144 2010-07-21 16:59 ./usr/bin/cpack-bug Btw, all this on an am64 Debian Sid, with cmake 2.8.1/2.8.2 compiled from source. |
(0021440) Bill Hoffman (manager) 2010-07-21 12:04 |
Looks like an issue with the switch to libarchive.... I don't have a ton of time right now, if you could figure out the patch I could get it done quick. Looks like the cmake -E tar command is not keeping the directories anymore. We used to use libtar, and switched for 2.8.2 to libarchive. |
(0021443) Thomas Themel (reporter) 2010-07-21 16:07 |
Mhm, the current cmSystemTools::CreateTar implementation looks exactly like it would produce the current behaviour. I'll look into it. |
(0021595) simonh (reporter) 2010-08-02 11:46 |
I've just encountered a similar bug with 2.8.2. Specifically, there are older versions of tar which choke on the extensions which libarchive is now using. See e.g. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-05/msg00700.html [^] My tar outputs: tar: Ignoring unknown extended header keyword `SCHILY.dev' tar: Ignoring unknown extended header keyword `SCHILY.ino' tar: Ignoring unknown extended header keyword `SCHILY.nlink' tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.security.selinux' ... and finally exits with code 2 (instead of just issuing a warning and exiting with 0) |
(0021597) simonh (reporter) 2010-08-02 12:14 |
I've attached a patch which works for me. After reading the libarchive docs (such as http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] ), I changed the CPack archive generator to use "ustar" output format, not the restricted pax format. |
(0021602) Brad King (manager) 2010-08-03 09:19 |
The current code came from commit 22fb266d (use different tar format to handle longer file names, 2009-11-14): http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=22fb266d [^] The commit switched from ustar to pax on purpose. Here is a simple experiment: $ mkdir foo $ touch foo/a $ tar cf foo.tar foo $ tar tf foo.tar foo/ foo/a $ cmake -E tar tf foo.tar foo foo/ foo/a If I use CMake to create the tarball instead: $ cmake -E tar cf foo.tar foo $ tar tf foo.tar foo/a $ cmake -E tar tf foo.tar foo foo/a I tried applying your change to cmSystemTools::CreateTar (pax_restricted -> ustar) but the above experiment produces the same result. |
(0021603) Brad King (manager) 2010-08-03 09:47 edited on: 2010-08-03 09:47 |
Even the example in http://code.google.com/p/libarchive/wiki/ManPageArchiveWrite3 [^] produces just "foo/a" unless one explicitly lists "foo" on the command line. Looking back at the old libtar-based implementation, it looks like libtar automatically handles directories so CMake's code could just send in those paths. Now the libarchive-based cmSystemTools::CreateTar does its own traversal. It uses a recursive glob search that only returns files and leaves out all their directories. This needs to be corrected to do a proper recursive directory traversal. |
(0021604) Brad King (manager) 2010-08-03 09:50 |
It looks like cmCPackGenerator::DoPackage also uses the recursive glob and does not return directories. |
(0021605) simonh (reporter) 2010-08-03 10:02 edited on: 2010-08-03 10:04 |
How do we solve the problem that pax archives are not readable without error on older GNU tar implementations? Is there an actual bug report that indicates that 255 character filenames are actually needed, bearing in mind that these are always relative to the top-level archive directory (i.e. not whole filesystems)? I'd always rather make the tar archives as portable as possible by default, and only add additional, more-specific variants (e.g. pax) as required. BTW, I now realise this is an orthogonal issue? Should I report a separate bug? |
(0021606) Brad King (manager) 2010-08-03 10:36 |
Let's just keep the discussion together here. The old libtar implementation did this: void th_set_path(TAR *t, char *pathname) { if (strlen(pathname)+strlen(suffix) >= T_NAMELEN && (t->options & TAR_GNU)) { /* GNU-style long name */ } else if (strlen(pathname)+ strlen(suffix) >= T_NAMELEN) { /* POSIX-style prefix field */ } else { /* classic tar format */ } } In other words, it dynamically switched the entry type for each path. |
(0021607) Brad King (manager) 2010-08-03 10:45 |
On the other hand, the libtar-based implementation seems to produce bad archives when there is a very, very deep name. In a simple test with a 400-character long path the file did not extract correctly. It does extract correctly from an archive created with the libarchive-based implementation (with more recent GNU tar). |
(0021609) Thomas Themel (reporter) 2010-08-03 11:11 edited on: 2010-08-03 14:45 |
I have attached a patch that fixes the (originally reported, not the path name length) issue in cmSystemTools by ensuring that every created file has all the parent directories it needs. Works for me, but please note that it seems that cmCPackArchiveGenerator::CompressFiles will exhibit the same issue for TGZ files and will need to be fixed in a similar way. Maybe it would be better to refactor so as to have a single place that turns a list of files into a tar file? |
(0021610) Brad King (manager) 2010-08-03 11:14 edited on: 2010-08-03 11:15 |
Yes, refactoring would be better. I think it should just be a function that takes the top-level directory by full path, a list of input paths, and a "struct archive*". Then it should load the directories and recurse itself. |
(0021611) simonh (reporter) 2010-08-03 12:38 |
I'll take a look tomorrow at whether unrestricted pax archives are supported by older GNU tars. Versions 1.14 and above claim to support POSIX.1-2001. I'll try "archive_write_set_format_pax"... |
(0021626) simonh (reporter) 2010-08-04 05:04 |
Nope, that didn't help. I can't read the generated tar archives without a non-zero exit status from the system-installed "tar" program on the following Linux distributions: Fedora Core 7 Red Hat Enterprise Linux 5 Some other distros (such as RHEL4) are fine, though... |
(0021627) Brad King (manager) 2010-08-04 07:54 |
What version of GNU tar (tar --version) are on the systems that do not work? |
(0021628) simonh (reporter) 2010-08-04 07:58 |
tar (GNU tar) 1.15.1 |
(0021629) simonh (reporter) 2010-08-04 08:14 |
But note that Red Hat has heavily patched the tar sources; I think the Fedora/RHEL version has 30-odd different RPM-releases for the same 1.15.1 upstream source. You can find the whole source + patches here: http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/7/Fedora/source/SRPMS/tar-1.15.1-26.fc7.src.rpm [^] |
(0021630) simonh (reporter) 2010-08-04 08:23 |
Having dug into the Red Hat SRPM, their xattr patch raises an ERROR() macro whenever an unknown extend header keyword is encountered (not a WARN() macro). They only support: SCHILY.xattr.security.selinux, SCHILY.xattr.system.posix_acl_access, SCHILY.xattr.system.posix_acl_default, SCHILY.xattr.user,SCHILY.xattr.root for the Schilling extensions. |
(0021689) Brad King (manager) 2010-08-06 15:43 |
I pushed out a work-in-progress topic to address the directory entry problem with "cmake -E tar": http://github.com/bradking/CMake/commits/libarchive-wrapper/ [^] The .deb generator actually uses this to create the package (it does not derive from cmCPackArchiveGenerator). |
(0021697) Brad King (manager) 2010-08-09 10:05 |
The new "cmake -E tar" produces ./usr/ ./usr/bin/ ./usr/bin/cpack-bug in the .deb file. I've merged the topic into next: http://cmake.org/gitweb?p=cmake.git;a=log;h=1b5b2ed3 [^] |
(0021977) Brad King (manager) 2010-08-26 13:40 |
simonh: Now that the original bug in this issue has been fixed I decided that your pax v. ustar issue deserves its own issue as you suggested earlier. I've opened 0011176 for discussion on that problem. Now I'm closing this issue since it is fixed. |
Notes |
Issue History | |||
Date Modified | Username | Field | Change |
2010-07-21 08:55 | Thomas Themel | New Issue | |
2010-07-21 10:30 | Bill Hoffman | Note Added: 0021436 | |
2010-07-21 10:30 | Bill Hoffman | Status | new => assigned |
2010-07-21 10:30 | Bill Hoffman | Assigned To | => Bill Hoffman |
2010-07-21 11:02 | Thomas Themel | Note Added: 0021439 | |
2010-07-21 11:03 | Thomas Themel | File Added: cpack-bug-0.1.tar.gz | |
2010-07-21 12:04 | Bill Hoffman | Note Added: 0021440 | |
2010-07-21 16:07 | Thomas Themel | Note Added: 0021443 | |
2010-08-02 11:46 | simonh | Note Added: 0021595 | |
2010-08-02 12:12 | simonh | File Added: cmake-ustar.patch | |
2010-08-02 12:14 | simonh | Note Added: 0021597 | |
2010-08-03 09:19 | Brad King | Note Added: 0021602 | |
2010-08-03 09:47 | Brad King | Note Added: 0021603 | |
2010-08-03 09:47 | Brad King | Note Edited: 0021603 | |
2010-08-03 09:50 | Brad King | Note Added: 0021604 | |
2010-08-03 10:02 | simonh | Note Added: 0021605 | |
2010-08-03 10:04 | simonh | Note Edited: 0021605 | |
2010-08-03 10:36 | Brad King | Note Added: 0021606 | |
2010-08-03 10:45 | Brad King | Note Added: 0021607 | |
2010-08-03 11:09 | Thomas Themel | File Added: cmake-tar-directories.diff | |
2010-08-03 11:11 | Thomas Themel | Note Added: 0021609 | |
2010-08-03 11:14 | Brad King | Note Added: 0021610 | |
2010-08-03 11:15 | Brad King | Note Edited: 0021610 | |
2010-08-03 12:38 | simonh | Note Added: 0021611 | |
2010-08-03 14:45 | Thomas Themel | Note Edited: 0021609 | |
2010-08-04 05:04 | simonh | Note Added: 0021626 | |
2010-08-04 07:54 | Brad King | Note Added: 0021627 | |
2010-08-04 07:58 | simonh | Note Added: 0021628 | |
2010-08-04 08:14 | simonh | Note Added: 0021629 | |
2010-08-04 08:23 | simonh | Note Added: 0021630 | |
2010-08-05 17:53 | Brad King | Assigned To | Bill Hoffman => Brad King |
2010-08-06 15:43 | Brad King | Note Added: 0021689 | |
2010-08-09 10:05 | Brad King | Note Added: 0021697 | |
2010-08-25 09:33 | Brad King | Relationship added | has duplicate 0011170 |
2010-08-26 13:38 | Brad King | Relationship added | related to 0011176 |
2010-08-26 13:40 | Brad King | Note Added: 0021977 | |
2010-08-26 13:41 | Brad King | Status | assigned => closed |
2010-08-26 13:41 | Brad King | Resolution | open => fixed |
2010-08-31 18:02 | David Cole | Target Version | => CMake 2.8.3 |
2010-09-10 00:09 | David Cole | Fixed in Version | => CMake 2.8.3 |
2011-03-10 08:50 | Brad King | Relationship added | related to 0011958 |
Issue History |
Copyright © 2000 - 2018 MantisBT Team |