February 5, 2008

CLI Magic: Manage all your archives with atool

Author: Ben Martin

The atool package is a collection of Perl scripts that allows you to handle many different archive formats and compression schemes using a single command-line interface. Atool uses other tools behind the scenes to perform the heavy lifting. With atool, you can handle any archive without having to remember what command-line tool to use to expand it and which options that particular tool expects.

Atool comes with a collection of commands, including aunpack, apack, als, acat, adiff, and arepack. The tools are all links to the main atool Perl script, but they perform the actions that their name suggests -- for example, unpacking or creating a new archive.

By default aunpack places all extracted files into a subdirectory, except that when an archive contains only a single file, that file will by default be extracted directly into the current directory. You can override this behavior with the -subdir (or -D) option, which will force any extracted files to be placed into a subdirectory even when the archive contains only a single file. This is great for extracting an archive to /tmp and not accidentally placing all the files directly in /tmp because the archive does not have the files in a subdirectory.

Atool can handle tar files compressed with gzip, bzip2, compress, lzop, and 7z, as well ass rar, lha, ace, arj, rpm, deb, jar, 7zip, and cpio and a few other formats. You must have the native tools installed for each format in order to use atool against that format. Some archive formats can only be listed or extracted, because the program to create such an archive is not available. For example, ace archives cannot be created or updated.

Let's see how it works. In the session below I first create a trivial little filesystem and then create a tar.bz2 file. Most folks know the command to make a tar compressed with either bzip2 or gzip by heart, but as the end of the example shows, you can use the same syntax to create an archive in any supported format.

$ cd /tmp
$ mkdir atest
$ cd atest
$ mkdir testdir
$ cd testdir
$ date >df1.txt
$ date >df2.txt
$ mkdir subdir1
$ date >subdir1/subfileb.txt
$ cd /tmp/atest
$ date >rawfile.txt
$ apack testdir.tar.bz2 testdir
testdir/
testdir/df1.txt
testdir/subdir1/
testdir/subdir1/subfileb.txt
testdir/df2.txt
$ file testdir.tar.bz2
testdir.tar.bz2: bzip2 compressed data, block size = 900k
$ tar tjvf testdir.tar.bz2
drwxrwxr-x ben/ben 0 2008-01-17 18:11 testdir/
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/df1.txt
drwxrwxr-x ben/ben 0 2008-01-17 18:11 testdir/subdir1/
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/subdir1/subfileb.txt
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/df2.txt
$ als testdir.tar.bz2
drwxrwxr-x ben/ben 0 2008-01-17 18:11 testdir/
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/df1.txt
drwxrwxr-x ben/ben 0 2008-01-17 18:11 testdir/subdir1/
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/subdir1/subfileb.txt
-rw-rw-r-- ben/ben 29 2008-01-17 18:11 testdir/df2.txt

$ apack testdir.cpio.7z testdir
7-Zip (A) 4.51 beta Copyright (c) 1999-2007 Igor Pavlov 2007-07-25
p7zip Version 4.51 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)
Scanning

Creating archive testdir.cpio.7z

Compressing testdir/df1.txt
Compressing testdir/df2.txt
Compressing testdir/subdir1/subfileb.txt

Everything is Ok

$ als testdir.cpio.7z

7-Zip (A) 4.51 beta Copyright (c) 1999-2007 Igor Pavlov 2007-07-25
p7zip Version 4.51 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

Listing archive: testdir.cpio.7z

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2008-01-17 18:11:04 ....A 29 42 testdir/df1.txt
2008-01-17 18:11:06 ....A 29 testdir/df2.txt
2008-01-17 18:11:16 ....A 29 testdir/subdir1/subfileb.txt
2008-01-17 18:11:16 D.... 0 0 testdir/subdir1
2008-01-17 18:11:11 D.... 0 0 testdir
------------------- ----- ------------ ------------ ------------------------
87 42 3 files, 2 folders

By default atool tries to handle the 7zip format using the 7z tool. You will likely have a p7zip package available for your distribution, and may have to tell atool to use its 7za executable instead of just 7z. Atool provides a simple key/value configuration file for all users in /etc/atool.conf and on a per-user basis using ~/.atoolrc. The below configuration makes atool use the 7za executable by default.

# cat /etc/atool.conf
path_7z 7za

You can get at single files within archives by using acat. Once again the command line is simple, and you do not have to worry about switches to the decompression and expansion tools in order to pick off a single file in an archive. By default the selected file is written to stdout. As shown below, you can use the quiet command-line option to suppress output from the decompression process when viewing a file in a terminal window. Unfortunately, the tools used to handle 7zip cannot write directly to the console and must be sent to a pager. By default the command pager is used to display files, though this may be overridden if the PAGER environment variable is set.

$ acat testdir.tar.bz2 testdir/df1.txt
testdir/df1.txt
Thu Jan 17 18:11:04 EST 2008
$ acat -q testdir.tar.bz2 testdir/df1.txt
Thu Jan 17 18:11:04 EST 2008

$ export PAGER=less
$ acat -q -p testdir.cpio.7z testdir/df1.txt
7-Zip (A) 4.51 beta Copyright (c) 1999-2007 Igor Pavlov 2007-07-25
p7zip Version 4.51 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

Processing archive: testdir.cpio.7z

Extracting testdir/df1.txt

Everything is Ok
Thu Jan 17 18:11:04 EST 2008

$ acat -p testdir.cpio.7z testdir/df1.txt 2>/dev/null
Thu Jan 17 18:11:04 EST 2008

Extraction of archives is where atool really shines. As mentioned before atool lets you avoid having to always check if an archive is friendly enough to extract its multiple files into a single subdirectory. In the below example I first extract the cpio file that I created before. In this case the files are already in a subdirectory, so atool did not have to perform any cleanup. I then create a jar archive which contains two files directly in the root directory of the archive. Using aunpack to expand this jar file will result in the two files being placed into a subdirectory automatically.

$ cd /tmp/atest
$ mkdir extraction
$ cd extraction
$ aunpack ../testdir.cpio.7z
...
Processing archive: ../testdir.cpio.7z

Extracting testdir/df1.txt
Extracting testdir/df2.txt
Extracting testdir/subdir1/subfileb.txt
Extracting testdir/subdir1
Extracting testdir

Everything is Ok
testdir.cpio.7z: extracted to `testdir'
[ben@alkid extraction]$ ls -lh
total 8.0K
drwxrwxr-x 3 ben ben 4.0K Thu.17.Jan 18:01 testdir/
$ cat ./testdir/df2.txt
Thu Jan 17 18:11:06 EST 2008

$ cd /tmp/atest/testdir
[ben@alkid testdir]$ ls -l
total 24
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df1.txt
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df2.txt
drwxrwxr-x 2 ben ben 4096 2008-01-17 18:11 subdir1
$ apack ../nasty-archive.jar df1.txt df2.txt
adding: df1.txt (stored 0%)
adding: df2.txt (stored 0%)
$ cd ..
$ rm -rf extraction
$ cd extraction
$ aunpack ../nasty-archive.jar
Archive: ../nasty-archive.jar
extracting: Unpack-3078/df1.txt
extracting: Unpack-3078/df2.txt
nasty-archive.jar: extracted to `nasty-archive' (multiple files in root)
$ ls -lR
.:
total 8
drwx------ 2 ben ben 4096 2008-01-17 18:40 nasty-archive

./nasty-archive:
total 16
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df1.txt
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df2.txt

To automatically change into the new subdirectory that atool creates when extracting an archive -- which is the default behaviour many people will want from the command -- you have to wrap aunpack in a shell function. The shell function below is based on one from the atool manual page, with some slight modifications to subprocess execution and the ability to work around an rm command that is aliased to include -i by default.

aunpack () {

TMP=$(mktemp /tmp/aunpack.XXXXXXXXXX);
atool -x --save-outdir=$TMP "$@"
DIR="$(cat $TMP)"
[ "$DIR" != "" -a -d "$DIR" ] && cd "$DIR"
/bin/rm -f $TMP
}

The commands arepack and adiff make archives a convenient unit of data to work with. With arepack you can convert an archive from one format into another. The adiff command shows the output of the diff command between two archives. The args_diff option defaults to -ru for diff generation. The example shown below starts with testdir as it was in a previous example. Toward the end of the example I modify the default diff command-line options and regenerate the diff showing only the additions.

$ ls -l testdir
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df1.txt
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 df2.txt
drwxrwxr-x 2 ben ben 4096 2008-01-17 18:11 subdir1
$ ls -l testdir/subdir1/
-rw-rw-r-- 1 ben ben 29 2008-01-17 18:11 subfileb.txt
$ apack testdir-first.tar.gz testdir
testdir/
testdir/df1.txt
testdir/subdir1/
testdir/subdir1/subfileb.txt
testdir/df2.txt
$ mv testdir/df2.txt testdir/subdir1/
$ date >>testdir/subdir1/subfileb.txt
$ date >testdir/newfileA.txt
$ apack testdir-second.tar.gz testdir
testdir/
testdir/newfileA.txt
testdir/df1.txt
testdir/subdir1/
testdir/subdir1/subfileb.txt
testdir/subdir1/df2.txt
$ adiff testdir-first.tar.gz testdir-second.tar.gz
Only in Unpack-1710/testdir: df2.txt
Only in Unpack-7823/testdir: newfileA.txt
Only in Unpack-7823/testdir/subdir1: df2.txt
diff -ru Unpack-1710/testdir/subdir1/subfileb.txt Unpack-7823/testdir/subdir1/subfileb.txt
--- Unpack-1710/testdir/subdir1/subfileb.txt 2008-01-17 18:11:16.000000000 +1000
+++ Unpack-7823/testdir/subdir1/subfileb.txt 2008-01-17 22:39:40.000000000 +1000
@@ -1 +1,2 @@
Thu Jan 17 18:11:16 EST 2008
+Thu Jan 17 22:39:40 EST 2008

$ vi ~/.atoolrc
args_diff -Nuar
$ adiff testdir-first.tar.gz testdir-second.tar.gz |grep ^+
+++ Unpack-9186/testdir/df2.txt 1970-01-01 10:00:00.000000000 +1000
+++ Unpack-9186/testdir/newfileA.txt 2008-01-17 22:39:48.000000000 +1000
+Thu Jan 17 22:39:48 EST 2008
+++ Unpack-9186/testdir/subdir1/df2.txt 2008-01-17 18:11:06.000000000 +1000
+Thu Jan 17 18:11:06 EST 2008
+++ Unpack-9186/testdir/subdir1/subfileb.txt 2008-01-17 22:39:40.000000000 +1000
+Thu Jan 17 22:39:40 EST 2008

Finally, with atool you can add support for new compression schemes and archive formats without having to change the commands you use to manage and compress archives.

Categories:

  • Desktop Software
  • Tools & Utilities
Click Here!