2025-02-17 21:04:20 +01:00
|
|
|
\input texinfo @c -*-texinfo-*-
|
|
|
|
@c %**start of header
|
|
|
|
@setfilename tarlz.info
|
|
|
|
@documentencoding ISO-8859-15
|
|
|
|
@settitle Tarlz Manual
|
|
|
|
@finalout
|
|
|
|
@c %**end of header
|
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
@set UPDATED 31 January 2019
|
|
|
|
@set VERSION 0.10
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@dircategory Data Compression
|
|
|
|
@direntry
|
|
|
|
* Tarlz: (tarlz). Archiver with multimember lzip compression
|
|
|
|
@end direntry
|
|
|
|
|
|
|
|
|
|
|
|
@ifnothtml
|
|
|
|
@titlepage
|
|
|
|
@title Tarlz
|
|
|
|
@subtitle Archiver with multimember lzip compression
|
|
|
|
@subtitle for Tarlz version @value{VERSION}, @value{UPDATED}
|
|
|
|
@author by Antonio Diaz Diaz
|
|
|
|
|
|
|
|
@page
|
|
|
|
@vskip 0pt plus 1filll
|
|
|
|
@end titlepage
|
|
|
|
|
|
|
|
@contents
|
|
|
|
@end ifnothtml
|
|
|
|
|
|
|
|
@node Top
|
|
|
|
@top
|
|
|
|
|
|
|
|
This manual is for Tarlz (version @value{VERSION}, @value{UPDATED}).
|
|
|
|
|
|
|
|
@menu
|
2025-02-17 21:10:01 +01:00
|
|
|
* Introduction:: Purpose and features of tarlz
|
|
|
|
* Invoking tarlz:: Command line interface
|
|
|
|
* File format:: Detailed format of the compressed archive
|
|
|
|
* Amendments to pax format:: The reasons for the differences with pax
|
2025-02-17 21:10:53 +01:00
|
|
|
* Multi-threaded tar:: Limitations of parallel tar decoding
|
2025-02-17 21:10:01 +01:00
|
|
|
* Examples:: A small tutorial with examples
|
|
|
|
* Problems:: Reporting bugs
|
|
|
|
* Concept index:: Index of concepts
|
2025-02-17 21:04:20 +01:00
|
|
|
@end menu
|
|
|
|
|
|
|
|
@sp 1
|
2025-02-17 21:10:53 +01:00
|
|
|
Copyright @copyright{} 2013-2019 Antonio Diaz Diaz.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
This manual is free documentation: you have unlimited permission
|
|
|
|
to copy, distribute and modify it.
|
|
|
|
|
|
|
|
|
|
|
|
@node Introduction
|
|
|
|
@chapter Introduction
|
|
|
|
@cindex introduction
|
|
|
|
|
2025-02-17 21:10:53 +01:00
|
|
|
@uref{http://www.nongnu.org/lzip/tarlz.html,,Tarlz} is a combined
|
|
|
|
implementation of the tar archiver and the
|
|
|
|
@uref{http://www.nongnu.org/lzip/lzip.html,,lzip} compressor. By default
|
|
|
|
tarlz creates, lists and extracts archives in a simplified posix pax format
|
|
|
|
compressed with lzip on a per file basis. Each tar member is compressed in
|
|
|
|
its own lzip member, as well as the end-of-file blocks. This method adds an
|
|
|
|
indexed lzip layer on top of the tar archive, making it possible to decode
|
|
|
|
the archive safely in parallel. The resulting multimember tar.lz archive is
|
|
|
|
fully backward compatible with standard tar tools like GNU tar, which treat
|
|
|
|
it like any other tar.lz archive. Tarlz can append files to the end of such
|
|
|
|
compressed archives.
|
|
|
|
|
|
|
|
Tarlz can create tar archives with four levels of compression granularity;
|
|
|
|
per file, per directory, appendable solid, and solid.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@noindent
|
|
|
|
Of course, compressing each file (or each directory) individually is
|
|
|
|
less efficient than compressing the whole tar archive, but it has the
|
|
|
|
following advantages:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
|
|
|
The resulting multimember tar.lz archive can be decompressed in
|
2025-02-17 21:10:53 +01:00
|
|
|
parallel, multiplying the decompression speed.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item
|
2025-02-17 21:10:01 +01:00
|
|
|
New members can be appended to the archive (by removing the EOF
|
2025-02-17 21:04:20 +01:00
|
|
|
member) just like to an uncompressed tar archive.
|
|
|
|
|
|
|
|
@item
|
|
|
|
It is a safe posix-style backup format. In case of corruption,
|
|
|
|
tarlz can extract all the undamaged members from the tar.lz
|
|
|
|
archive, skipping over the damaged members, just like the standard
|
2025-02-17 21:11:12 +01:00
|
|
|
(uncompressed) tar. Moreover, the option @samp{--keep-damaged} can be
|
2025-02-17 21:10:01 +01:00
|
|
|
used to recover as much data as possible from each damaged member,
|
|
|
|
and lziprecover can be used to recover some of the damaged members.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item
|
|
|
|
A multimember tar.lz archive is usually smaller than the
|
|
|
|
corresponding solidly compressed tar.gz archive, except when
|
|
|
|
individually compressing files smaller than about 32 KiB.
|
|
|
|
@end itemize
|
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
Tarlz protects the extended records with a CRC in a way compatible with
|
|
|
|
standard tar tools. @xref{crc32}.
|
|
|
|
|
|
|
|
Tarlz does not understand other tar formats like @samp{gnu}, @samp{oldgnu},
|
|
|
|
@samp{star} or @samp{v7}.
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@node Invoking tarlz
|
|
|
|
@chapter Invoking tarlz
|
|
|
|
@cindex invoking
|
|
|
|
@cindex options
|
|
|
|
@cindex usage
|
|
|
|
@cindex version
|
|
|
|
|
|
|
|
The format for running tarlz is:
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz [@var{options}] [@var{files}]
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
On archive creation or appending, tarlz removes leading and trailing
|
2025-02-17 21:10:01 +01:00
|
|
|
slashes from filenames, as well as filename prefixes containing a
|
2025-02-17 21:04:20 +01:00
|
|
|
@samp{..} component. On extraction, archive members containing a
|
2025-02-17 21:10:01 +01:00
|
|
|
@samp{..} component are skipped. Tarlz detects when the archive being
|
|
|
|
created or enlarged is among the files to be dumped, appended or
|
|
|
|
concatenated, and skips it.
|
|
|
|
|
|
|
|
On extraction and listing, tarlz removes leading @samp{./} strings from
|
|
|
|
member names in the archive or given in the command line, so that
|
|
|
|
@w{@code{tarlz -xf foo ./bar baz}} extracts members @samp{bar} and
|
|
|
|
@samp{./baz} from archive @samp{foo}.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
tarlz supports the following options:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item -h
|
|
|
|
@itemx --help
|
|
|
|
Print an informative help message describing the options and exit.
|
|
|
|
|
|
|
|
@item -V
|
|
|
|
@itemx --version
|
|
|
|
Print the version number of tarlz on the standard output and exit.
|
2025-02-17 21:10:01 +01:00
|
|
|
This version number should be included in all bug reports.
|
|
|
|
|
|
|
|
@item -A
|
|
|
|
@itemx --concatenate
|
|
|
|
Append tar.lz archives to the end of a tar.lz archive. All the archives
|
|
|
|
involved must be regular (seekable) files compressed as multimember lzip
|
|
|
|
files, and the two end-of-file blocks plus any zero padding must be
|
|
|
|
contained in the last lzip member of each archive. The intermediate
|
|
|
|
end-of-file blocks are removed as each new archive is concatenated. Exit
|
|
|
|
with status 0 without modifying the archive if no @var{files} have been
|
|
|
|
specified. Tarlz can't concatenate uncompressed tar archives.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
@anchor{--data-size}
|
|
|
|
@item -B @var{bytes}
|
|
|
|
@itemx --data-size=@var{bytes}
|
|
|
|
Set target size of input data blocks for the @samp{--bsolid} option. Valid
|
|
|
|
values range from @w{8 KiB} to @w{1 GiB}. Default value is two times the
|
|
|
|
dictionary size, except for option @samp{-0} where it defaults to @w{1 MiB}.
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item -c
|
|
|
|
@itemx --create
|
2025-02-17 21:10:01 +01:00
|
|
|
Create a new archive from @var{files}.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item -C @var{dir}
|
|
|
|
@itemx --directory=@var{dir}
|
|
|
|
Change to directory @var{dir}. When creating or appending, the position
|
2025-02-17 21:11:12 +01:00
|
|
|
of each @samp{-C} option in the command line is significant; it will
|
2025-02-17 21:04:20 +01:00
|
|
|
change the current working directory for the following @var{files} until
|
2025-02-17 21:11:12 +01:00
|
|
|
a new @samp{-C} option appears in the command line. When extracting, all
|
|
|
|
the @samp{-C} options are executed in sequence before starting the
|
|
|
|
extraction. Listing ignores any @samp{-C} options specified. @var{dir}
|
2025-02-17 21:04:20 +01:00
|
|
|
is relative to the then current working directory, perhaps changed by a
|
2025-02-17 21:11:12 +01:00
|
|
|
previous @samp{-C} option.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item -f @var{archive}
|
|
|
|
@itemx --file=@var{archive}
|
|
|
|
Use archive file @var{archive}. @samp{-} used as an @var{archive}
|
|
|
|
argument reads from standard input or writes to standard output.
|
|
|
|
|
2025-02-17 21:10:53 +01:00
|
|
|
@item -n @var{n}
|
|
|
|
@itemx --threads=@var{n}
|
|
|
|
Set the number of decompression threads, overriding the system's default.
|
|
|
|
Valid values range from 0 to "as many as your system can support". A value
|
|
|
|
of 0 disables threads entirely. If this option is not used, tarlz tries to
|
|
|
|
detect the number of processors in the system and use it as default value.
|
|
|
|
@w{@samp{tarlz --help}} shows the system's default value. This option
|
|
|
|
currently only has effect when listing the contents of a multimember
|
|
|
|
compressed archive. @xref{Multi-threaded tar}.
|
|
|
|
|
|
|
|
Note that the number of usable threads is limited during decompression to
|
|
|
|
the number of lzip members in the tar.lz archive, which you can find by
|
|
|
|
running @w{@code{lzip -lv archive.tar.lz}}.
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item -q
|
|
|
|
@itemx --quiet
|
|
|
|
Quiet operation. Suppress all messages.
|
|
|
|
|
|
|
|
@item -r
|
|
|
|
@itemx --append
|
2025-02-17 21:10:01 +01:00
|
|
|
Append files to the end of a tar.lz archive. The archive must be a
|
|
|
|
regular (seekable) file compressed as a multimember lzip file, and the
|
|
|
|
two end-of-file blocks plus any zero padding must be contained in the
|
|
|
|
last lzip member of the archive. First this last member is removed, then
|
|
|
|
the new members are appended, and then a new end-of-file member is
|
|
|
|
appended to the archive. Exit with status 0 without modifying the
|
|
|
|
archive if no @var{files} have been specified. Tarlz can't append files
|
|
|
|
to an uncompressed tar archive.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item -t
|
|
|
|
@itemx --list
|
2025-02-17 21:10:01 +01:00
|
|
|
List the contents of an archive. If @var{files} are given, list only the
|
|
|
|
given @var{files}.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item -v
|
|
|
|
@itemx --verbose
|
|
|
|
Verbosely list files processed.
|
|
|
|
|
|
|
|
@item -x
|
|
|
|
@itemx --extract
|
2025-02-17 21:10:01 +01:00
|
|
|
Extract files from an archive. If @var{files} are given, extract only
|
|
|
|
the given @var{files}. Else extract all the files in the archive.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item -0 .. -9
|
|
|
|
Set the compression level. The default compression level is @samp{-6}.
|
2025-02-17 21:10:01 +01:00
|
|
|
Like lzip, tarlz also minimizes the dictionary size of the lzip members
|
|
|
|
it creates, reducing the amount of memory required for decompression.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
@multitable {Level} {Dictionary size} {Match length limit}
|
|
|
|
@item Level @tab Dictionary size @tab Match length limit
|
|
|
|
@item -0 @tab 64 KiB @tab 16 bytes
|
|
|
|
@item -1 @tab 1 MiB @tab 5 bytes
|
|
|
|
@item -2 @tab 1.5 MiB @tab 6 bytes
|
|
|
|
@item -3 @tab 2 MiB @tab 8 bytes
|
|
|
|
@item -4 @tab 3 MiB @tab 12 bytes
|
|
|
|
@item -5 @tab 4 MiB @tab 20 bytes
|
|
|
|
@item -6 @tab 8 MiB @tab 36 bytes
|
|
|
|
@item -7 @tab 16 MiB @tab 68 bytes
|
|
|
|
@item -8 @tab 24 MiB @tab 132 bytes
|
|
|
|
@item -9 @tab 32 MiB @tab 273 bytes
|
|
|
|
@end multitable
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item --asolid
|
|
|
|
When creating or appending to a compressed archive, use appendable solid
|
|
|
|
compression. All the files being added to the archive are compressed
|
|
|
|
into a single lzip member, but the end-of-file blocks are compressed
|
|
|
|
into a separate lzip member. This creates a solidly compressed
|
|
|
|
appendable archive.
|
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
@item --bsolid
|
|
|
|
When creating or appending to a compressed archive, compress tar members
|
|
|
|
together in a lzip member until they approximate a target uncompressed size.
|
|
|
|
The size can't be exact because each solidly compressed data block must
|
|
|
|
contain an integer number of tar members. This option improves compression
|
|
|
|
efficiency for archives with lots of small files. @xref{--data-size}, to set
|
|
|
|
the target block size.
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item --dsolid
|
|
|
|
When creating or appending to a compressed archive, use solid
|
|
|
|
compression for each directory especified in the command line. The
|
|
|
|
end-of-file blocks are compressed into a separate lzip member. This
|
|
|
|
creates a compressed appendable archive with a separate lzip member for
|
|
|
|
each top-level directory.
|
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
@item --no-solid
|
|
|
|
When creating or appending to a compressed archive, compress each file
|
|
|
|
separately. The end-of-file blocks are compressed into a separate lzip
|
|
|
|
member. This creates a compressed appendable archive with a separate
|
|
|
|
lzip member for each file. This option allows tarlz revert to default
|
|
|
|
behavior if, for example, tarlz is invoked through an alias like
|
|
|
|
@code{tar='tarlz --solid'}.
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item --solid
|
|
|
|
When creating or appending to a compressed archive, use solid
|
|
|
|
compression. The files being added to the archive, along with the
|
|
|
|
end-of-file blocks, are compressed into a single lzip member. The
|
|
|
|
resulting archive is not appendable. No more files can be later appended
|
2025-02-17 21:10:01 +01:00
|
|
|
to the archive.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
@item --anonymous
|
2025-02-17 21:11:12 +01:00
|
|
|
Equivalent to @samp{--owner=root --group=root}.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@item --owner=@var{owner}
|
|
|
|
When creating or appending, use @var{owner} for files added to the
|
|
|
|
archive. If @var{owner} is not a valid user name, it is decoded as a
|
|
|
|
decimal numeric user ID.
|
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
@item --group=@var{group}
|
|
|
|
When creating or appending, use @var{group} for files added to the
|
|
|
|
archive. If @var{group} is not a valid group name, it is decoded as a
|
|
|
|
decimal numeric group ID.
|
|
|
|
|
|
|
|
@item --keep-damaged
|
|
|
|
Don't delete partially extracted files. If a decompression error happens
|
|
|
|
while extracting a file, keep the partial data extracted. Use this
|
|
|
|
option to recover as much data as possible from each damaged member.
|
|
|
|
|
|
|
|
@item --missing-crc
|
|
|
|
Exit with error status 2 if the CRC of the extended records is missing.
|
|
|
|
When this option is used, tarlz detects any corruption in the extended
|
|
|
|
records (only limited by CRC collisions). But note that a corrupt
|
|
|
|
@samp{GNU.crc32} keyword, for example @samp{GNU.crc33}, is reported as a
|
|
|
|
missing CRC instead of as a corrupt record. This misleading
|
|
|
|
@samp{Missing CRC} message is the consequence of a flaw in the posix pax
|
|
|
|
format; i.e., the lack of a mandatory check sequence in the extended
|
|
|
|
records. @xref{crc32}.
|
|
|
|
|
|
|
|
@ignore
|
|
|
|
@item --permissive
|
|
|
|
Allow some violations of the archive format, like consecutive extended
|
|
|
|
headers preceding a ustar header, or several records with the same
|
|
|
|
keyword appearing in the same block of extended records.
|
|
|
|
@end ignore
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@item --uncompressed
|
2025-02-17 21:11:12 +01:00
|
|
|
With @samp{--create}, don't compress the created tar archive. Create an
|
2025-02-17 21:04:20 +01:00
|
|
|
uncompressed tar archive instead.
|
|
|
|
|
|
|
|
@end table
|
|
|
|
|
|
|
|
Exit status: 0 for a normal exit, 1 for environmental problems (file not
|
|
|
|
found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or
|
|
|
|
invalid input file, 3 for an internal consistency error (eg, bug) which
|
|
|
|
caused tarlz to panic.
|
|
|
|
|
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
@node File format
|
|
|
|
@chapter File format
|
|
|
|
@cindex file format
|
|
|
|
|
|
|
|
In the diagram below, a box like this:
|
|
|
|
@verbatim
|
|
|
|
+---+
|
|
|
|
| | <-- the vertical bars might be missing
|
|
|
|
+---+
|
|
|
|
@end verbatim
|
|
|
|
|
|
|
|
represents one byte; a box like this:
|
|
|
|
@verbatim
|
|
|
|
+==============+
|
|
|
|
| |
|
|
|
|
+==============+
|
|
|
|
@end verbatim
|
|
|
|
|
|
|
|
represents a variable number of bytes or a fixed but large number of
|
|
|
|
bytes (for example 512).
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
A tar.lz file consists of a series of lzip members (compressed data sets).
|
|
|
|
The members simply appear one after another in the file, with no
|
|
|
|
additional information before, between, or after them.
|
|
|
|
|
|
|
|
Each lzip member contains one or more tar members in a simplified posix
|
|
|
|
pax interchange format; the only pax typeflag value supported by tarlz
|
|
|
|
(in addition to the typeflag values defined by the ustar format) is
|
|
|
|
@samp{x}. The pax format is an extension on top of the ustar format that
|
|
|
|
removes the size limitations of the ustar format.
|
|
|
|
|
|
|
|
Each tar member contains one file archived, and is represented by the
|
|
|
|
following sequence:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
|
|
|
An optional extended header block with extended header records. This
|
|
|
|
header block is of the form described in pax header block, with a
|
|
|
|
typeflag value of @samp{x}. The extended header records are included as
|
|
|
|
the data for this header block.
|
|
|
|
|
|
|
|
@item
|
|
|
|
A header block in ustar format that describes the file. Any fields
|
|
|
|
defined in the preceding optional extended header records override the
|
|
|
|
associated fields in this header block for this file.
|
|
|
|
|
|
|
|
@item
|
|
|
|
Zero or more blocks that contain the contents of the file.
|
|
|
|
@end itemize
|
|
|
|
|
2025-02-17 21:10:53 +01:00
|
|
|
Each tar member must be contiguously stored in a lzip member for the
|
2025-02-17 21:11:12 +01:00
|
|
|
parallel decoding operations like @samp{--list} to work. If any tar member
|
2025-02-17 21:10:53 +01:00
|
|
|
is split over two or more lzip members, the archive must be decoded
|
|
|
|
sequentially. @xref{Multi-threaded tar}.
|
|
|
|
|
2025-02-17 21:10:01 +01:00
|
|
|
At the end of the archive file there are two 512-byte blocks filled with
|
|
|
|
binary zeros, interpreted as an end-of-archive indicator. These EOF
|
|
|
|
blocks are either compressed in a separate lzip member or compressed
|
|
|
|
along with the tar members contained in the last lzip member.
|
|
|
|
|
|
|
|
The diagram below shows the correspondence between each tar member
|
|
|
|
(formed by one or two headers plus optional data) in the tar archive and
|
|
|
|
each
|
|
|
|
@uref{http://www.nongnu.org/lzip/manual/lzip_manual.html#File-format,,lzip member}
|
|
|
|
in the resulting multimember tar.lz archive:
|
|
|
|
@ifnothtml
|
|
|
|
@xref{File format,,,lzip}.
|
|
|
|
@end ifnothtml
|
|
|
|
|
|
|
|
@verbatim
|
|
|
|
tar
|
|
|
|
+========+======+=================+===============+========+======+========+
|
|
|
|
| header | data | extended header | extended data | header | data | EOF |
|
|
|
|
+========+======+=================+===============+========+======+========+
|
|
|
|
|
|
|
|
tar.lz
|
|
|
|
+===============+=================================================+========+
|
|
|
|
| member | member | member |
|
|
|
|
+===============+=================================================+========+
|
|
|
|
@end verbatim
|
|
|
|
|
|
|
|
@ignore
|
2025-02-17 21:11:12 +01:00
|
|
|
When @samp{--permissive} is used, the following violations of the
|
2025-02-17 21:10:01 +01:00
|
|
|
archive format are allowed:@*
|
|
|
|
If several extended headers precede an ustar header, only the last
|
|
|
|
extended header takes effect. The other extended headers are ignored.
|
|
|
|
Similarly, if several records with the same keyword appear in the same
|
|
|
|
block of extended records, only the last record for the repeated keyword
|
|
|
|
takes effect. The other records for the repeated keyword are ignored.
|
|
|
|
@end ignore
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@section Pax header block
|
|
|
|
|
|
|
|
The pax header block is identical to the ustar header block described below
|
|
|
|
except that the typeflag has the value @samp{x} (extended). The size field
|
|
|
|
is the size of the extended header data in bytes. Most other fields in the
|
|
|
|
pax header block are zeroed on archive creation to prevent trouble if the
|
|
|
|
archive is read by an ustar tool, and are ignored by tarlz on archive
|
|
|
|
extraction. @xref{flawed-compat}.
|
|
|
|
|
|
|
|
The pax extended header data consists of one or more records, each of
|
|
|
|
them constructed as follows:@*
|
|
|
|
@code{"%d %s=%s\n", <length>, <keyword>, <value>}
|
|
|
|
|
|
|
|
The <length>, <blank>, <keyword>, <equals-sign>, and <newline> in the
|
|
|
|
record must be limited to the portable character set. The <length> field
|
|
|
|
contains the decimal length of the record in bytes, including the
|
|
|
|
trailing <newline>. The <value> field is stored as-is, without
|
|
|
|
conversion to UTF-8 nor any other transformation.
|
|
|
|
|
|
|
|
These are the <keyword> fields currently supported by tarlz:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item linkpath
|
|
|
|
The pathname of a link being created to another file, of any type,
|
|
|
|
previously archived. This record overrides the linkname field in the
|
|
|
|
following ustar header block. The following ustar header block
|
|
|
|
determines the type of link created. If typeflag of the following header
|
|
|
|
block is 1, it will be a hard link. If typeflag is 2, it will be a
|
|
|
|
symbolic link and the linkpath value will be used as the contents of the
|
|
|
|
symbolic link.
|
|
|
|
|
|
|
|
@item path
|
|
|
|
The pathname of the following file. This record overrides the name and
|
|
|
|
prefix fields in the following ustar header block.
|
|
|
|
|
|
|
|
@item size
|
|
|
|
The size of the file in bytes, expressed as a decimal number using
|
|
|
|
digits from the ISO/IEC 646:1991 (ASCII) standard. This record overrides
|
|
|
|
the size field in the following ustar header block. The size record is
|
|
|
|
used only for files with a size value greater than 8_589_934_591
|
|
|
|
@w{(octal 77777777777)}. This is 2^33 bytes or larger.
|
|
|
|
|
|
|
|
@anchor{key_crc32}
|
|
|
|
@item GNU.crc32
|
|
|
|
CRC32-C (Castagnoli) of the extended header data excluding the 8 bytes
|
|
|
|
representing the CRC <value> itself. The <value> is represented as 8
|
|
|
|
hexadecimal digits in big endian order,
|
|
|
|
@w{@samp{22 GNU.crc32=00000000\n}}. The keyword of the CRC record is
|
|
|
|
protected by the CRC to guarante that corruption is always detected
|
|
|
|
(except in case of CRC collision). A CRC was chosen because a checksum
|
|
|
|
is too weak for a potentially large list of variable sized records. A
|
|
|
|
checksum can't detect simple errors like the swapping of two bytes.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@section Ustar header block
|
|
|
|
|
|
|
|
The ustar header block has a length of 512 bytes and is structured as
|
|
|
|
shown in the following table. All lengths and offsets are in decimal.
|
|
|
|
|
|
|
|
@multitable {Field Name} {Offset} {Length (in bytes)}
|
|
|
|
@item Field Name @tab Offset @tab Length (in bytes)
|
|
|
|
@item name @tab 0 @tab 100
|
|
|
|
@item mode @tab 100 @tab 8
|
|
|
|
@item uid @tab 108 @tab 8
|
|
|
|
@item gid @tab 116 @tab 8
|
|
|
|
@item size @tab 124 @tab 12
|
|
|
|
@item mtime @tab 136 @tab 12
|
|
|
|
@item chksum @tab 148 @tab 8
|
|
|
|
@item typeflag @tab 156 @tab 1
|
|
|
|
@item linkname @tab 157 @tab 100
|
|
|
|
@item magic @tab 257 @tab 6
|
|
|
|
@item version @tab 263 @tab 2
|
|
|
|
@item uname @tab 265 @tab 32
|
|
|
|
@item gname @tab 297 @tab 32
|
|
|
|
@item devmajor @tab 329 @tab 8
|
|
|
|
@item devminor @tab 337 @tab 8
|
|
|
|
@item prefix @tab 345 @tab 155
|
|
|
|
@end multitable
|
|
|
|
|
|
|
|
All characters in the header block are coded using the ISO/IEC 646:1991
|
|
|
|
(ASCII) standard, except in fields storing names for files, users, and
|
|
|
|
groups. For maximum portability between implementations, names should
|
|
|
|
only contain characters from the portable filename character set. But if
|
|
|
|
an implementation supports the use of characters outside of @samp{/} and
|
|
|
|
the portable filename character set in names for files, users, and
|
|
|
|
groups, tarlz will use the byte values in these names unmodified.
|
|
|
|
|
|
|
|
The fields name, linkname, and prefix are null-terminated character
|
|
|
|
strings except when all characters in the array contain non-null
|
|
|
|
characters including the last character.
|
|
|
|
|
|
|
|
The name and the prefix fields produce the pathname of the file. A new
|
|
|
|
pathname is formed, if prefix is not an empty string (its first
|
|
|
|
character is not null), by concatenating prefix (up to the first null
|
|
|
|
character), a <slash> character, and name; otherwise, name is used
|
|
|
|
alone. In either case, name is terminated at the first null character.
|
|
|
|
If prefix begins with a null character, it is ignored. In this manner,
|
|
|
|
pathnames of at most 256 characters can be supported. If a pathname does
|
|
|
|
not fit in the space provided, an extended record is used to store the
|
|
|
|
pathname.
|
|
|
|
|
|
|
|
The linkname field does not use the prefix to produce a pathname. If the
|
|
|
|
linkname does not fit in the 100 characters provided, an extended record
|
|
|
|
is used to store the linkname.
|
|
|
|
|
|
|
|
The mode field provides 12 access permission bits. The following table
|
|
|
|
shows the symbolic name of each bit and its octal value:
|
|
|
|
|
2025-02-17 21:10:53 +01:00
|
|
|
@multitable {Bit Name} {Value} {Bit Name} {Value} {Bit Name} {Value}
|
|
|
|
@headitem Bit Name @tab Value @tab Bit Name @tab Value @tab Bit Name @tab Value
|
|
|
|
@item S_ISUID @tab 04000 @tab S_ISGID @tab 02000 @tab S_ISVTX @tab 01000
|
|
|
|
@item S_IRUSR @tab 00400 @tab S_IWUSR @tab 00200 @tab S_IXUSR @tab 00100
|
|
|
|
@item S_IRGRP @tab 00040 @tab S_IWGRP @tab 00020 @tab S_IXGRP @tab 00010
|
|
|
|
@item S_IROTH @tab 00004 @tab S_IWOTH @tab 00002 @tab S_IXOTH @tab 00001
|
2025-02-17 21:10:01 +01:00
|
|
|
@end multitable
|
|
|
|
|
|
|
|
The uid and gid fields are the user and group ID of the owner and group
|
|
|
|
of the file, respectively.
|
|
|
|
|
|
|
|
The size field contains the octal representation of the size of the file
|
|
|
|
in bytes. If the typeflag field specifies a file of type '0' (regular
|
|
|
|
file) or '7' (high performance regular file), the number of logical
|
|
|
|
records following the header is @w{(size / 512)} rounded to the next
|
|
|
|
integer. For all other values of typeflag, tarlz either sets the size
|
|
|
|
field to 0 or ignores it, and does not store or expect any logical
|
|
|
|
records following the header. If the file size is larger than
|
|
|
|
8_589_934_591 bytes @w{(octal 77777777777)}, an extended record is used
|
|
|
|
to store the file size.
|
|
|
|
|
|
|
|
The mtime field contains the octal representation of the modification
|
|
|
|
time of the file at the time it was archived, obtained from the stat()
|
|
|
|
function.
|
|
|
|
|
|
|
|
The chksum field contains the octal representation of the value of the
|
|
|
|
simple sum of all bytes in the header logical record. Each byte in the
|
|
|
|
header is treated as an unsigned value. When calculating the checksum,
|
|
|
|
the chksum field is treated as if it were all <space> characters.
|
|
|
|
|
|
|
|
The typeflag field contains a single character specifying the type of
|
|
|
|
file archived:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item '0'
|
|
|
|
Regular file.
|
|
|
|
|
|
|
|
@item '1'
|
|
|
|
Hard link to another file, of any type, previously archived.
|
|
|
|
|
|
|
|
@item '2'
|
|
|
|
Symbolic link.
|
|
|
|
|
|
|
|
@item '3', '4'
|
|
|
|
Character special file and block special file respectively. In this case
|
|
|
|
the devmajor and devminor fields contain information defining the
|
|
|
|
device in unspecified format.
|
|
|
|
|
|
|
|
@item '5'
|
|
|
|
Directory.
|
|
|
|
|
|
|
|
@item '6'
|
|
|
|
FIFO special file.
|
|
|
|
|
|
|
|
@item '7'
|
|
|
|
Reserved to represent a file to which an implementation has associated
|
|
|
|
some high-performance attribute. Tarlz treats this type of file as a
|
|
|
|
regular file (type 0).
|
|
|
|
|
|
|
|
@end table
|
|
|
|
|
|
|
|
The magic field contains the ASCII null-terminated string "ustar". The
|
2025-02-17 21:10:53 +01:00
|
|
|
version field contains the characters "00" (0x30,0x30). The fields uname,
|
|
|
|
and gname are null-terminated character strings except when all characters
|
|
|
|
in the array contain non-null characters including the last character. Each
|
|
|
|
numeric field contains a leading space- or zero-filled, optionally
|
|
|
|
null-terminated octal number using digits from the ISO/IEC 646:1991 (ASCII)
|
|
|
|
standard. Tarlz is able to decode numeric fields 1 byte larger than standard
|
|
|
|
ustar by not requiring a terminating null character.
|
2025-02-17 21:10:01 +01:00
|
|
|
|
|
|
|
|
|
|
|
@node Amendments to pax format
|
|
|
|
@chapter The reasons for the differences with pax
|
|
|
|
@cindex Amendments to pax format
|
|
|
|
|
|
|
|
Tarlz is meant to reliably detect invalid or corrupt metadata during
|
|
|
|
extraction and to not create safety risks in the archives it creates. In
|
|
|
|
order to achieve these goals, tarlz makes some changes to the variant of the
|
|
|
|
pax format that it uses. This chapter describes these changes and the
|
|
|
|
concrete reasons to implement them.
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@anchor{crc32}
|
|
|
|
@section Add a CRC of the extended records
|
|
|
|
|
|
|
|
The posix pax format has a serious flaw. The metadata stored in pax extended
|
|
|
|
records are not protected by any kind of check sequence. Corruption in a
|
|
|
|
long filename may cause the extraction of the file in the wrong place
|
2025-02-17 21:10:53 +01:00
|
|
|
without warning. Corruption in a large file size may cause the truncation of
|
2025-02-17 21:10:01 +01:00
|
|
|
the file or the appending of garbage to the file, both followed by a
|
|
|
|
spurious warning about a corrupt header far from the place of the undetected
|
|
|
|
corruption.
|
|
|
|
|
|
|
|
Metadata like filename and file size must be always protected in an archive
|
|
|
|
format because of the adverse effects of undetected corruption in them,
|
|
|
|
potentially much worse that undetected corruption in the data. Even more so
|
|
|
|
in the case of pax because the amount of metadata it stores is potentially
|
|
|
|
large, making undetected corruption more probable.
|
|
|
|
|
|
|
|
Because of the above, tarlz protects the extended records with a CRC in
|
|
|
|
a way compatible with standard tar tools. @xref{key_crc32}.
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@anchor{flawed-compat}
|
|
|
|
@section Remove flawed backward compatibility
|
|
|
|
|
|
|
|
In order to allow the extraction of pax archives by a tar utility conforming
|
|
|
|
to the POSIX-2:1993 standard, POSIX.1-2008 recommends selecting extended
|
|
|
|
header field values that allow such tar to create a regular file containing
|
|
|
|
the extended header records as data. This approach is broken because if the
|
|
|
|
extended header is needed because of a long filename, the name and prefix
|
|
|
|
fields will be unable to contain the full pathname of the file. Therefore
|
|
|
|
the files corresponding to both the extended header and the overridden ustar
|
|
|
|
header will be extracted using truncated filenames, perhaps overwriting
|
|
|
|
existing files or directories. It may be a security risk to extract a file
|
|
|
|
with a truncated filename.
|
|
|
|
|
|
|
|
To avoid this problem, tarlz writes extended headers with all fields zeroed
|
|
|
|
except size, chksum, typeflag, magic and version. This prevents old tar
|
|
|
|
programs from extracting the extended records as a file in the wrong place.
|
|
|
|
Tarlz also sets to zero those fields of the ustar header overridden by
|
|
|
|
extended records.
|
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
If an extended header is required for any reason (for example a file size
|
|
|
|
larger than @w{8 GiB} or a link name longer than 100 bytes), tarlz moves the
|
|
|
|
filename also to the extended header to prevent an ustar tool from trying to
|
|
|
|
extract the file or link. This also makes easier during parallel extraction
|
|
|
|
or listing the detection of a tar member split between two lzip members at
|
|
|
|
the boundary between the extended header and the ustar header.
|
2025-02-17 21:10:01 +01:00
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@section As simple as possible (but not simpler)
|
|
|
|
|
|
|
|
The tarlz format is mainly ustar. Extended pax headers are used only when
|
|
|
|
needed because the length of a filename or link name, or the size of a file
|
|
|
|
exceed the limits of the ustar format. Adding extended headers to each
|
|
|
|
member just to record subsecond timestamps seems wasteful for a backup
|
|
|
|
format.
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@section Avoid misconversions to/from UTF-8
|
|
|
|
|
|
|
|
There is no portable way to tell what charset a text string is coded into.
|
|
|
|
Therefore, tarlz stores all fields representing text strings as-is, without
|
|
|
|
conversion to UTF-8 nor any other transformation. This prevents accidental
|
|
|
|
double UTF-8 conversions. If the need arises this behavior will be adjusted
|
|
|
|
with a command line option in the future.
|
|
|
|
|
|
|
|
|
2025-02-17 21:10:53 +01:00
|
|
|
@node Multi-threaded tar
|
|
|
|
@chapter Limitations of parallel tar decoding
|
|
|
|
|
|
|
|
Safely decoding an arbitrary tar archive in parallel is impossible. For
|
|
|
|
example, if a tar archive containing another tar archive is decoded starting
|
|
|
|
from some position other than the beginning, there is no way to know if the
|
|
|
|
first header found there belongs to the outer tar archive or to the inner
|
|
|
|
tar archive. Tar is a format inherently serial; it was designed for tapes.
|
|
|
|
|
|
|
|
In the case of compressed tar archives, the start of each compressed block
|
|
|
|
determines one point through which the tar archive can be decoded in
|
|
|
|
parallel. Therefore, in tar.lz archives the decoding operations can't be
|
|
|
|
parallelized if the tar members are not aligned with the lzip members. Tar
|
|
|
|
archives compressed with plzip can't be decoded in parallel because tar and
|
|
|
|
plzip do not have a way to align both sets of members. Certainly one can
|
|
|
|
decompress one such archive with a multi-threaded tool like plzip, but the
|
|
|
|
increase in speed is not as large as it could be because plzip must
|
|
|
|
serialize the decompressed data and pass them to tar, which decodes them
|
|
|
|
sequentially, one tar member at a time.
|
|
|
|
|
|
|
|
On the other hand, if the tar.lz archive is created with a tool like tarlz,
|
|
|
|
which can guarantee the alignment between tar members and lzip members
|
|
|
|
because it controls both archiving and compression, then the lzip format
|
|
|
|
becomes an indexed layer on top of the tar archive which makes possible
|
|
|
|
decoding it safely in parallel.
|
|
|
|
|
|
|
|
Tarlz is able to automatically decode aligned and unaligned multimember
|
|
|
|
tar.lz archives, keeping backwards compatibility. If tarlz finds a member
|
|
|
|
misalignment during multi-threaded decoding, it switches to single-threaded
|
2025-02-17 21:11:12 +01:00
|
|
|
mode and continues decoding the archive. Currently only the @samp{--list}
|
2025-02-17 21:10:53 +01:00
|
|
|
option is able to do multi-threaded decoding.
|
|
|
|
|
2025-02-17 21:11:12 +01:00
|
|
|
If the files in the archive are large, multi-threaded @samp{--list} on a
|
|
|
|
regular (seekable) tar.lz archive can be hundreds of times faster than
|
|
|
|
sequential @samp{--list} because, in addition to using several processors,
|
|
|
|
it only needs to decompress part of each lzip member. See the following
|
|
|
|
example listing the Silesia corpus on a dual core machine:
|
2025-02-17 21:10:53 +01:00
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz -9 -cf silesia.tar.lz silesia
|
|
|
|
time lzip -cd silesia.tar.lz | tar -tf - (5.032s)
|
|
|
|
time plzip -cd silesia.tar.lz | tar -tf - (3.256s)
|
|
|
|
time tarlz -tf silesia.tar.lz (0.020s)
|
|
|
|
@end example
|
|
|
|
|
|
|
|
|
2025-02-17 21:04:20 +01:00
|
|
|
@node Examples
|
|
|
|
@chapter A small tutorial with examples
|
|
|
|
@cindex examples
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
Example 1: Create a multimember compressed archive @samp{archive.tar.lz}
|
|
|
|
containing files @samp{a}, @samp{b} and @samp{c}.
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz -cf archive.tar.lz a b c
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 2: Append files @samp{d} and @samp{e} to the multimember
|
|
|
|
compressed archive @samp{archive.tar.lz}.
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz -rf archive.tar.lz d e
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 3: Create a solidly compressed appendable archive
|
|
|
|
@samp{archive.tar.lz} containing files @samp{a}, @samp{b} and @samp{c}.
|
|
|
|
Then append files @samp{d} and @samp{e} to the archive.
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz --asolid -cf archive.tar.lz a b c
|
|
|
|
tarlz --asolid -rf archive.tar.lz d e
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 4: Create a compressed appendable archive containing directories
|
|
|
|
@samp{dir1}, @samp{dir2} and @samp{dir3} with a separate lzip member per
|
|
|
|
directory. Then append files @samp{a}, @samp{b}, @samp{c}, @samp{d} and
|
|
|
|
@samp{e} to the archive, all of them contained in a single lzip member.
|
|
|
|
The resulting archive @samp{archive.tar.lz} contains 5 lzip members
|
2025-02-17 21:10:01 +01:00
|
|
|
(including the EOF member).
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz --dsolid -cf archive.tar.lz dir1 dir2 dir3
|
|
|
|
tarlz --asolid -rf archive.tar.lz a b c d e
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 5: Create a solidly compressed archive @samp{archive.tar.lz}
|
|
|
|
containing files @samp{a}, @samp{b} and @samp{c}. Note that no more
|
2025-02-17 21:10:01 +01:00
|
|
|
files can be later appended to the archive.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz --solid -cf archive.tar.lz a b c
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 6: Extract all files from archive @samp{archive.tar.lz}.
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz -xf archive.tar.lz
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 7: Extract files @samp{a} and @samp{c} from archive
|
|
|
|
@samp{archive.tar.lz}.
|
|
|
|
|
|
|
|
@example
|
|
|
|
tarlz -xf archive.tar.lz a c
|
|
|
|
@end example
|
|
|
|
|
|
|
|
@sp 1
|
|
|
|
@noindent
|
|
|
|
Example 8: Copy the contents of directory @samp{sourcedir} to the
|
2025-02-17 21:11:12 +01:00
|
|
|
directory @samp{destdir}.
|
2025-02-17 21:04:20 +01:00
|
|
|
|
|
|
|
@example
|
2025-02-17 21:11:12 +01:00
|
|
|
tarlz -C sourcedir -c . | tarlz -C destdir -x
|
2025-02-17 21:04:20 +01:00
|
|
|
@end example
|
|
|
|
|
|
|
|
|
|
|
|
@node Problems
|
|
|
|
@chapter Reporting bugs
|
|
|
|
@cindex bugs
|
|
|
|
@cindex getting help
|
|
|
|
|
|
|
|
There are probably bugs in tarlz. There are certainly errors and
|
|
|
|
omissions in this manual. If you report them, they will get fixed. If
|
|
|
|
you don't, no one will ever know about them and they will remain unfixed
|
|
|
|
for all eternity, if not longer.
|
|
|
|
|
|
|
|
If you find a bug in tarlz, please send electronic mail to
|
|
|
|
@email{lzip-bug@@nongnu.org}. Include the version number, which you can
|
|
|
|
find by running @w{@code{tarlz --version}}.
|
|
|
|
|
|
|
|
|
|
|
|
@node Concept index
|
|
|
|
@unnumbered Concept index
|
|
|
|
|
|
|
|
@printindex cp
|
|
|
|
|
|
|
|
@bye
|