Merging upstream version 1.12~rc1.
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
parent
4ddb634c25
commit
cd6a248630
GPG key ID:
FBB4F0E80A80222F
24 changed files with 874 additions and 719 deletions
35
doc/plzip.1
35
doc/plzip.1
|
|
@ -1,32 +1,33 @@
|
|||
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.2.
|
||||
.TH PLZIP "1" "January 2024" "plzip 1.11" "User Commands"
|
||||
.TH PLZIP "1" "November 2024" "plzip 1.12-rc1" "User Commands"
|
||||
.SH NAME
|
||||
plzip \- reduces the size of files
|
||||
.SH SYNOPSIS
|
||||
.B plzip
|
||||
[\fI\,options\/\fR] [\fI\,files\/\fR]
|
||||
.SH DESCRIPTION
|
||||
Plzip is a massively parallel (multi\-threaded) implementation of lzip,
|
||||
compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib.
|
||||
Plzip is a massively parallel (multi\-threaded) implementation of lzip. Plzip
|
||||
uses the compression library lzlib.
|
||||
.PP
|
||||
Lzip is a lossless data compressor with a user interface similar to the one
|
||||
of gzip or bzip2. Lzip uses a simplified form of the 'Lempel\-Ziv\-Markov
|
||||
chain\-Algorithm' (LZMA) stream format to maximize interoperability. The
|
||||
maximum dictionary size is 512 MiB so that any lzip file can be decompressed
|
||||
on 32\-bit machines. Lzip provides accurate and robust 3\-factor integrity
|
||||
checking. Lzip can compress about as fast as gzip (lzip \fB\-0\fR) or compress most
|
||||
files more than bzip2 (lzip \fB\-9\fR). Decompression speed is intermediate between
|
||||
gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery
|
||||
perspective. Lzip has been designed, written, and tested with great care to
|
||||
replace gzip and bzip2 as the standard general\-purpose compressed format for
|
||||
Unix\-like systems.
|
||||
of gzip or bzip2. Lzip uses a simplified form of LZMA (Lempel\-Ziv\-Markov
|
||||
chain\-Algorithm) designed to achieve complete interoperability between
|
||||
implementations. The maximum dictionary size is 512 MiB so that any lzip
|
||||
file can be decompressed on 32\-bit machines. Lzip provides accurate and
|
||||
robust 3\-factor integrity checking. 'lzip \fB\-0\fR' compresses about as fast as
|
||||
gzip, while 'lzip \fB\-9\fR' compresses most files more than bzip2. Decompression
|
||||
speed is intermediate between gzip and bzip2. Lzip provides better data
|
||||
recovery capabilities than gzip and bzip2. Lzip has been designed, written,
|
||||
and tested with great care to replace gzip and bzip2 as general\-purpose
|
||||
compressed format for Unix\-like systems.
|
||||
.PP
|
||||
Plzip can compress/decompress large files on multiprocessor machines much
|
||||
faster than lzip, at the cost of a slightly reduced compression ratio (0.4
|
||||
to 2 percent larger compressed files). Note that the number of usable
|
||||
threads is limited by file size; on files larger than a few GB plzip can use
|
||||
hundreds of processors, but on files of only a few MB plzip is no faster
|
||||
than lzip.
|
||||
hundreds of processors, but on files smaller than 1 MiB plzip is no faster
|
||||
than lzip (even at compression level \fB\-0\fR).
|
||||
The number of threads defaults to the number of processors.
|
||||
.SH OPTIONS
|
||||
.TP
|
||||
\fB\-h\fR, \fB\-\-help\fR
|
||||
|
|
@ -132,8 +133,8 @@ License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>
|
|||
.br
|
||||
This is free software: you are free to change and redistribute it.
|
||||
There is NO WARRANTY, to the extent permitted by law.
|
||||
Using lzlib 1.14
|
||||
Using LZ_API_VERSION = 1014
|
||||
Using lzlib 1.15\-rc1
|
||||
Using LZ_API_VERSION = 1015
|
||||
.SH "SEE ALSO"
|
||||
The full documentation for
|
||||
.B plzip
|
||||
|
|
|
|||
347
doc/plzip.info
347
doc/plzip.info
|
|
@ -11,21 +11,22 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir)
|
|||
Plzip Manual
|
||||
************
|
||||
|
||||
This manual is for Plzip (version 1.11, 21 January 2024).
|
||||
This manual is for Plzip (version 1.12-rc1, 19 November 2024).
|
||||
|
||||
* Menu:
|
||||
|
||||
* Introduction:: Purpose and features of plzip
|
||||
* Output:: Meaning of plzip's output
|
||||
* Invoking plzip:: Command-line interface
|
||||
* Program design:: Internal structure of plzip
|
||||
* Memory requirements:: Memory required to compress and decompress
|
||||
* Minimum file sizes:: Minimum file sizes required for full speed
|
||||
* File format:: Detailed format of the compressed file
|
||||
* Trailing data:: Extra data appended to the file
|
||||
* Examples:: A small tutorial with examples
|
||||
* Problems:: Reporting bugs
|
||||
* Concept index:: Index of concepts
|
||||
* Introduction:: Purpose and features of plzip
|
||||
* Output:: Meaning of plzip's output
|
||||
* Invoking plzip:: Command-line interface
|
||||
* Argument syntax:: By convention, options start with a hyphen
|
||||
* File format:: Detailed format of the compressed file
|
||||
* Program design:: Internal structure of plzip
|
||||
* Memory requirements:: Memory required to compress and decompress
|
||||
* Minimum file sizes:: Minimum file sizes required for full speed
|
||||
* Trailing data:: Extra data appended to the file
|
||||
* Examples:: A small tutorial with examples
|
||||
* Problems:: Reporting bugs
|
||||
* Concept index:: Index of concepts
|
||||
|
||||
|
||||
Copyright (C) 2009-2024 Antonio Diaz Diaz.
|
||||
|
|
@ -39,27 +40,27 @@ File: plzip.info, Node: Introduction, Next: Output, Prev: Top, Up: Top
|
|||
1 Introduction
|
||||
**************
|
||||
|
||||
Plzip is a massively parallel (multi-threaded) implementation of lzip,
|
||||
compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib.
|
||||
Plzip is a massively parallel (multi-threaded) implementation of lzip.
|
||||
Plzip uses the compression library lzlib.
|
||||
|
||||
Lzip is a lossless data compressor with a user interface similar to the
|
||||
one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov
|
||||
chain-Algorithm' (LZMA) stream format to maximize interoperability. The
|
||||
maximum dictionary size is 512 MiB so that any lzip file can be decompressed
|
||||
on 32-bit machines. Lzip provides accurate and robust 3-factor integrity
|
||||
checking. Lzip can compress about as fast as gzip (lzip -0) or compress most
|
||||
files more than bzip2 (lzip -9). Decompression speed is intermediate between
|
||||
gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery
|
||||
perspective. Lzip has been designed, written, and tested with great care to
|
||||
replace gzip and bzip2 as the standard general-purpose compressed format for
|
||||
Unix-like systems.
|
||||
one of gzip or bzip2. Lzip uses a simplified form of LZMA (Lempel-Ziv-Markov
|
||||
chain-Algorithm) designed to achieve complete interoperability between
|
||||
implementations. The maximum dictionary size is 512 MiB so that any lzip
|
||||
file can be decompressed on 32-bit machines. Lzip provides accurate and
|
||||
robust 3-factor integrity checking. 'lzip -0' compresses about as fast as
|
||||
gzip, while 'lzip -9' compresses most files more than bzip2. Decompression
|
||||
speed is intermediate between gzip and bzip2. Lzip provides better data
|
||||
recovery capabilities than gzip and bzip2. Lzip has been designed, written,
|
||||
and tested with great care to replace gzip and bzip2 as general-purpose
|
||||
compressed format for Unix-like systems.
|
||||
|
||||
Plzip can compress/decompress large files on multiprocessor machines much
|
||||
faster than lzip, at the cost of a slightly reduced compression ratio (0.4
|
||||
to 2 percent larger compressed files). Note that the number of usable
|
||||
threads is limited by file size; on files larger than a few GB plzip can use
|
||||
hundreds of processors, but on files of only a few MB plzip is no faster
|
||||
than lzip. *Note Minimum file sizes::.
|
||||
hundreds of processors, but on files smaller than 1 MiB plzip is no faster
|
||||
than lzip (even at compression level -0). *Note Minimum file sizes::.
|
||||
|
||||
For creation and manipulation of compressed tar archives tarlz can be
|
||||
more efficient than using tar and plzip because tarlz is able to keep the
|
||||
|
|
@ -96,9 +97,9 @@ makes it safer than compressors returning ambiguous warning values (like
|
|||
gzip) when it is used as a back end for other programs like tar or zutils.
|
||||
|
||||
Plzip automatically uses for each file the largest dictionary size that
|
||||
does not exceed neither the file size nor the limit given. Keep in mind
|
||||
that the decompression memory requirement is affected at compression time
|
||||
by the choice of dictionary size limit. *Note Memory requirements::.
|
||||
does not exceed neither the file size nor the limit given. The dictionary
|
||||
size used for decompression is the same dictionary size used for
|
||||
compression. *Note Memory requirements::.
|
||||
|
||||
When compressing, plzip replaces every file given in the command line
|
||||
with a compressed version of itself, with the name "original_name.lz". When
|
||||
|
|
@ -174,7 +175,7 @@ have been compressed. Decompressed is used to refer to data which have
|
|||
undergone the process of decompression.
|
||||
|
||||
|
||||
File: plzip.info, Node: Invoking plzip, Next: Program design, Prev: Output, Up: Top
|
||||
File: plzip.info, Node: Invoking plzip, Next: Argument syntax, Prev: Output, Up: Top
|
||||
|
||||
3 Invoking plzip
|
||||
****************
|
||||
|
|
@ -189,8 +190,7 @@ means standard input. It can be mixed with other FILES and is read just
|
|||
once, the first time it appears in the command line. Remember to prepend
|
||||
'./' to any file name beginning with a hyphen, or use '--'.
|
||||
|
||||
plzip supports the following options: *Note Argument syntax:
|
||||
(arg_parser)Argument syntax.
|
||||
plzip supports the following options: *Note Argument syntax::.
|
||||
|
||||
'-h'
|
||||
'--help'
|
||||
|
|
@ -235,7 +235,8 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
status 1. If a file fails to decompress, or is a terminal, plzip exits
|
||||
immediately with error status 2 without decompressing the rest of the
|
||||
files. A terminal is considered an uncompressed file, and therefore
|
||||
invalid.
|
||||
invalid. A multimember file with one or more empty members is accepted
|
||||
if redirected to standard input.
|
||||
|
||||
'-f'
|
||||
'--force'
|
||||
|
|
@ -259,7 +260,8 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
'-v', the dictionary size, the number of members in the file, and the
|
||||
amount of trailing data (if any) are also printed. With '-vv', the
|
||||
positions and sizes of each member in multimember files are also
|
||||
printed.
|
||||
printed. A multimember file with one or more empty members is accepted
|
||||
if redirected to standard input.
|
||||
|
||||
If any file is damaged, does not exist, can't be opened, or is not
|
||||
regular, the final exit status is > 0. '-lq' can be used to check
|
||||
|
|
@ -278,8 +280,8 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
'-n N'
|
||||
'--threads=N'
|
||||
Set the maximum number of worker threads, overriding the system's
|
||||
default. Valid values range from 1 to "as many as your system can
|
||||
support". If this option is not used, plzip tries to detect the number
|
||||
default. Valid values range from 1 to as many as your system can
|
||||
support. If this option is not used, plzip tries to detect the number
|
||||
of processors in the system and use it as default value. When
|
||||
compressing on a 32 bit system, plzip tries to limit the memory use to
|
||||
under 2.22 GiB (4 worker threads at level -9) by reducing the number
|
||||
|
|
@ -338,7 +340,8 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
fails the test, does not exist, can't be opened, or is a terminal,
|
||||
plzip continues testing the rest of the files. A final diagnostic is
|
||||
shown at verbosity level 1 or higher if any file fails the test when
|
||||
testing multiple files.
|
||||
testing multiple files. A multimember file with one or more empty
|
||||
members is accepted if redirected to standard input.
|
||||
|
||||
'-v'
|
||||
'--verbose'
|
||||
|
|
@ -368,6 +371,7 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
'-s64MiB -m273'
|
||||
|
||||
Level Dictionary size (-s) Match length limit (-m)
|
||||
------------------------------------------------------
|
||||
-0 64 KiB 16 bytes
|
||||
-1 1 MiB 5 bytes
|
||||
-2 1.5 MiB 6 bytes
|
||||
|
|
@ -387,7 +391,7 @@ once, the first time it appears in the command line. Remember to prepend
|
|||
When decompressing, testing, or listing, allow trailing data whose
|
||||
first bytes are so similar to the magic bytes of a lzip header that
|
||||
they can be confused with a corrupt header. Use this option if a file
|
||||
triggers a "corrupt header" error and the cause is not indeed a
|
||||
triggers a 'corrupt header' error and the cause is not indeed a
|
||||
corrupt header.
|
||||
|
||||
'--in-slots=N'
|
||||
|
|
@ -421,6 +425,7 @@ and may be followed by a multiplier and an optional 'B' for "byte".
|
|||
Table of SI and binary prefixes (unit multipliers):
|
||||
|
||||
Prefix Value | Prefix Value
|
||||
----------------------------------------------------------------------
|
||||
k kilobyte (10^3 = 1000) | Ki kibibyte (2^10 = 1024)
|
||||
M megabyte (10^6) | Mi mebibyte (2^20)
|
||||
G gigabyte (10^9) | Gi gibibyte (2^30)
|
||||
|
|
@ -439,9 +444,131 @@ corrupt or invalid input file, 3 for an internal consistency error (e.g.,
|
|||
bug) which caused plzip to panic.
|
||||
|
||||
|
||||
File: plzip.info, Node: Program design, Next: Memory requirements, Prev: Invoking plzip, Up: Top
|
||||
File: plzip.info, Node: Argument syntax, Next: File format, Prev: Invoking plzip, Up: Top
|
||||
|
||||
4 Internal structure of plzip
|
||||
4 Syntax of command-line arguments
|
||||
**********************************
|
||||
|
||||
POSIX recommends these conventions for command-line arguments.
|
||||
|
||||
* A command-line argument is an option if it begins with a hyphen ('-').
|
||||
|
||||
* Option names are single alphanumeric characters.
|
||||
|
||||
* Certain options require an argument.
|
||||
|
||||
* An option and its argument may or may not appear as separate tokens.
|
||||
(In other words, the whitespace separating them is optional, unless the
|
||||
argument is the empty string). Thus, '-o foo' and '-ofoo' are
|
||||
equivalent.
|
||||
|
||||
* One or more options without arguments, followed by at most one option
|
||||
that takes an argument, may follow a hyphen in a single token. Thus,
|
||||
'-abc' is equivalent to '-a -b -c'.
|
||||
|
||||
* Options typically precede other non-option arguments.
|
||||
|
||||
* The argument '--' terminates all options; any following arguments are
|
||||
treated as non-option arguments, even if they begin with a hyphen.
|
||||
|
||||
* A token consisting of a single hyphen character is interpreted as an
|
||||
ordinary non-option argument. By convention, it is used to specify
|
||||
standard input, standard output, or a file named '-'.
|
||||
|
||||
GNU adds "long options" to these conventions:
|
||||
|
||||
* A long option consists of two hyphens ('--') followed by a name made
|
||||
of alphanumeric characters and hyphens. Option names are typically one
|
||||
to three words long, with hyphens to separate words. Abbreviations can
|
||||
be used for the long option names as long as the abbreviations are
|
||||
unique.
|
||||
|
||||
* A long option and its argument may or may not appear as separate
|
||||
tokens. In the latter case they must be separated by an equal sign '='.
|
||||
Thus, '--foo bar' and '--foo=bar' are equivalent.
|
||||
|
||||
The syntax of options with an optional argument is
|
||||
'-<short_option><argument>' (without whitespace), or
|
||||
'--<long_option>=<argument>'.
|
||||
|
||||
|
||||
File: plzip.info, Node: File format, Next: Program design, Prev: Argument syntax, Up: Top
|
||||
|
||||
5 File format
|
||||
*************
|
||||
|
||||
Perfection is reached, not when there is no longer anything to add, but
|
||||
when there is no longer anything to take away.
|
||||
-- Antoine de Saint-Exupery
|
||||
|
||||
In the diagram below, a box like this:
|
||||
|
||||
+---+
|
||||
| | <-- the vertical bars might be missing
|
||||
+---+
|
||||
|
||||
represents one byte; a box like this:
|
||||
|
||||
+==============+
|
||||
| |
|
||||
+==============+
|
||||
|
||||
represents a variable number of bytes.
|
||||
|
||||
A lzip file consists of one or more independent "members" (compressed data
|
||||
sets). The members simply appear one after another in the file, with no
|
||||
additional information before, between, or after them. Each member can
|
||||
encode in compressed form up to 16 EiB - 1 byte of uncompressed data. The
|
||||
size of a multimember file is unlimited. Empty members (data size = 0) are
|
||||
not allowed in multimember files.
|
||||
|
||||
Each member has the following structure:
|
||||
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
|
||||
All multibyte values are stored in little endian order.
|
||||
|
||||
'ID string (the "magic" bytes)'
|
||||
A four byte string, identifying the lzip format, with the value "LZIP"
|
||||
(0x4C, 0x5A, 0x49, 0x50).
|
||||
|
||||
'VN (version number, 1 byte)'
|
||||
Just in case something needs to be modified in the future. 1 for now.
|
||||
|
||||
'DS (coded dictionary size, 1 byte)'
|
||||
The dictionary size is calculated by taking a power of 2 (the base
|
||||
size) and subtracting from it a fraction between 0/16 and 7/16 of the
|
||||
base size.
|
||||
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).
|
||||
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
|
||||
from the base size to obtain the dictionary size.
|
||||
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB
|
||||
Valid values for dictionary size range from 4 KiB to 512 MiB.
|
||||
|
||||
'LZMA stream'
|
||||
The LZMA stream, terminated by an 'End Of Stream' marker. Uses default
|
||||
values for encoder properties. *Note Stream format: (lzip)Stream
|
||||
format, for a complete description.
|
||||
|
||||
'CRC32 (4 bytes)'
|
||||
Cyclic Redundancy Check (CRC) of the original uncompressed data.
|
||||
|
||||
'Data size (8 bytes)'
|
||||
Size of the original uncompressed data.
|
||||
|
||||
'Member size (8 bytes)'
|
||||
Total size of the member, including header and trailer. This field acts
|
||||
as a distributed index, improves the checking of stream integrity, and
|
||||
facilitates the safe recovery of undamaged members from multimember
|
||||
files. Lzip limits the member size to 2 PiB to prevent the data size
|
||||
field from overflowing.
|
||||
|
||||
|
||||
File: plzip.info, Node: Program design, Next: Memory requirements, Prev: File format, Up: Top
|
||||
|
||||
6 Internal structure of plzip
|
||||
*****************************
|
||||
|
||||
When compressing, plzip divides the input file into chunks and compresses as
|
||||
|
|
@ -456,8 +583,8 @@ because lzip usually produces single-member files, which can't be
|
|||
decompressed in parallel.
|
||||
|
||||
For each input file, a splitter thread and several worker threads are
|
||||
created, acting the main thread as muxer (multiplexer) thread. A "packet
|
||||
courier" takes care of data transfers among threads and limits the maximum
|
||||
created, acting the main thread as muxer (multiplexer) thread. A 'packet
|
||||
courier' takes care of data transfers among threads and limits the maximum
|
||||
number of data blocks (packets) being processed simultaneously.
|
||||
|
||||
The splitter reads data blocks from the input file, and distributes them
|
||||
|
|
@ -486,7 +613,7 @@ only limited by the number of processors available and by I/O speed.
|
|||
|
||||
File: plzip.info, Node: Memory requirements, Next: Minimum file sizes, Prev: Program design, Up: Top
|
||||
|
||||
5 Memory required to compress and decompress
|
||||
7 Memory required to compress and decompress
|
||||
********************************************
|
||||
|
||||
The amount of memory required *per worker thread* for decompression or
|
||||
|
|
@ -520,6 +647,7 @@ The following table shows the memory required *per thread* for compression
|
|||
at a given level, using the default data size for each level:
|
||||
|
||||
Level Memory required
|
||||
------------------------
|
||||
-0 4.875 MiB
|
||||
-1 17.75 MiB
|
||||
-2 26.625 MiB
|
||||
|
|
@ -532,9 +660,9 @@ Level Memory required
|
|||
-9 568 MiB
|
||||
|
||||
|
||||
File: plzip.info, Node: Minimum file sizes, Next: File format, Prev: Memory requirements, Up: Top
|
||||
File: plzip.info, Node: Minimum file sizes, Next: Trailing data, Prev: Memory requirements, Up: Top
|
||||
|
||||
6 Minimum file sizes required for full compression speed
|
||||
8 Minimum file sizes required for full compression speed
|
||||
********************************************************
|
||||
|
||||
When compressing, plzip divides the input file into chunks and compresses
|
||||
|
|
@ -569,85 +697,9 @@ Level
|
|||
-9 128 MiB 256 MiB 512 MiB 1 GiB 4 GiB 16 GiB
|
||||
|
||||
|
||||
File: plzip.info, Node: File format, Next: Trailing data, Prev: Minimum file sizes, Up: Top
|
||||
File: plzip.info, Node: Trailing data, Next: Examples, Prev: Minimum file sizes, Up: Top
|
||||
|
||||
7 File format
|
||||
*************
|
||||
|
||||
Perfection is reached, not when there is no longer anything to add, but
|
||||
when there is no longer anything to take away.
|
||||
-- Antoine de Saint-Exupery
|
||||
|
||||
|
||||
In the diagram below, a box like this:
|
||||
|
||||
+---+
|
||||
| | <-- the vertical bars might be missing
|
||||
+---+
|
||||
|
||||
represents one byte; a box like this:
|
||||
|
||||
+==============+
|
||||
| |
|
||||
+==============+
|
||||
|
||||
represents a variable number of bytes.
|
||||
|
||||
|
||||
A lzip file consists of one or more independent "members" (compressed
|
||||
data sets). The members simply appear one after another in the file, with no
|
||||
additional information before, between, or after them. Each member can
|
||||
encode in compressed form up to 16 EiB - 1 byte of uncompressed data. The
|
||||
size of a multimember file is unlimited.
|
||||
|
||||
Each member has the following structure:
|
||||
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
|
||||
All multibyte values are stored in little endian order.
|
||||
|
||||
'ID string (the "magic" bytes)'
|
||||
A four byte string, identifying the lzip format, with the value "LZIP"
|
||||
(0x4C, 0x5A, 0x49, 0x50).
|
||||
|
||||
'VN (version number, 1 byte)'
|
||||
Just in case something needs to be modified in the future. 1 for now.
|
||||
|
||||
'DS (coded dictionary size, 1 byte)'
|
||||
The dictionary size is calculated by taking a power of 2 (the base
|
||||
size) and subtracting from it a fraction between 0/16 and 7/16 of the
|
||||
base size.
|
||||
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).
|
||||
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
|
||||
from the base size to obtain the dictionary size.
|
||||
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB
|
||||
Valid values for dictionary size range from 4 KiB to 512 MiB.
|
||||
|
||||
'LZMA stream'
|
||||
The LZMA stream, finished by an "End Of Stream" marker. Uses default
|
||||
values for encoder properties. *Note Stream format: (lzip)Stream
|
||||
format, for a complete description.
|
||||
|
||||
'CRC32 (4 bytes)'
|
||||
Cyclic Redundancy Check (CRC) of the original uncompressed data.
|
||||
|
||||
'Data size (8 bytes)'
|
||||
Size of the original uncompressed data.
|
||||
|
||||
'Member size (8 bytes)'
|
||||
Total size of the member, including header and trailer. This field acts
|
||||
as a distributed index, improves the checking of stream integrity, and
|
||||
facilitates the safe recovery of undamaged members from multimember
|
||||
files. Lzip limits the member size to 2 PiB to prevent the data size
|
||||
field from overflowing.
|
||||
|
||||
|
||||
|
||||
File: plzip.info, Node: Trailing data, Next: Examples, Prev: File format, Up: Top
|
||||
|
||||
8 Extra data appended to the file
|
||||
9 Extra data appended to the file
|
||||
*********************************
|
||||
|
||||
Sometimes extra data are found appended to a lzip file after the last
|
||||
|
|
@ -657,7 +709,7 @@ member. Such trailing data may be:
|
|||
example when writing to a tape. It is safe to append any amount of
|
||||
padding zero bytes to a lzip file.
|
||||
|
||||
* Useful data added by the user; an "End Of File" string (to check that
|
||||
* Useful data added by the user; an 'End Of File' string (to check that
|
||||
the file has not been truncated), a cryptographically secure hash, a
|
||||
description of file contents, etc. It is safe to append any amount of
|
||||
text to a lzip file as long as none of the first four bytes of the
|
||||
|
|
@ -693,8 +745,8 @@ where a file containing trailing data must be rejected, the option
|
|||
|
||||
File: plzip.info, Node: Examples, Next: Problems, Prev: Trailing data, Up: Top
|
||||
|
||||
9 A small tutorial with examples
|
||||
********************************
|
||||
10 A small tutorial with examples
|
||||
*********************************
|
||||
|
||||
WARNING! Even if plzip is bug-free, other causes may result in a corrupt
|
||||
compressed file (bugs in the system libraries, memory errors, etc).
|
||||
|
|
@ -706,38 +758,32 @@ comparing the compressed file with the original because the corruption
|
|||
happens before plzip compresses the RAM contents, resulting in a valid
|
||||
compressed file containing wrong data.
|
||||
|
||||
|
||||
Example 1: Extract all the files from archive 'foo.tar.lz'.
|
||||
|
||||
tar -xf foo.tar.lz
|
||||
or
|
||||
plzip -cd foo.tar.lz | tar -xf -
|
||||
|
||||
|
||||
Example 2: Replace a regular file with its compressed version 'file.lz' and
|
||||
show the compression ratio.
|
||||
|
||||
plzip -v file
|
||||
|
||||
|
||||
Example 3: Like example 2 but the created 'file.lz' has a block size of
|
||||
1 MiB. The compression ratio is not shown.
|
||||
|
||||
plzip -B 1MiB file
|
||||
|
||||
|
||||
Example 4: Restore a regular file from its compressed version 'file.lz'. If
|
||||
the operation is successful, 'file.lz' is removed.
|
||||
|
||||
plzip -d file.lz
|
||||
|
||||
|
||||
Example 5: Check the integrity of the compressed file 'file.lz' and show
|
||||
status.
|
||||
|
||||
plzip -tv file.lz
|
||||
|
||||
|
||||
Example 6: The right way of concatenating the decompressed output of two or
|
||||
more compressed files. *Note Trailing data::.
|
||||
|
||||
|
|
@ -746,19 +792,16 @@ more compressed files. *Note Trailing data::.
|
|||
Do this instead
|
||||
plzip -cd file1.lz file2.lz file3.lz
|
||||
|
||||
|
||||
Example 7: Decompress 'file.lz' partially until 10 KiB of decompressed data
|
||||
are produced.
|
||||
|
||||
plzip -cd file.lz | dd bs=1024 count=10
|
||||
|
||||
|
||||
Example 8: Decompress 'file.lz' partially from decompressed byte at offset
|
||||
10000 to decompressed byte at offset 14999 (5000 bytes are produced).
|
||||
|
||||
plzip -cd file.lz | dd bs=1000 skip=10 count=5
|
||||
|
||||
|
||||
Example 9: Compress a whole device in /dev/sdc and send the output to
|
||||
'file.lz'.
|
||||
|
||||
|
|
@ -769,7 +812,7 @@ Example 9: Compress a whole device in /dev/sdc and send the output to
|
|||
|
||||
File: plzip.info, Node: Problems, Next: Concept index, Prev: Examples, Up: Top
|
||||
|
||||
10 Reporting bugs
|
||||
11 Reporting bugs
|
||||
*****************
|
||||
|
||||
There are probably bugs in plzip. There are certainly errors and omissions
|
||||
|
|
@ -790,6 +833,7 @@ Concept index
|
|||
[index ]
|
||||
* Menu:
|
||||
|
||||
* argument syntax: Argument syntax. (line 6)
|
||||
* bugs: Problems. (line 6)
|
||||
* examples: Examples. (line 6)
|
||||
* file format: File format. (line 6)
|
||||
|
|
@ -809,21 +853,22 @@ Concept index
|
|||
|
||||
Tag Table:
|
||||
Node: Top217
|
||||
Node: Introduction1156
|
||||
Node: Output5934
|
||||
Node: Invoking plzip7497
|
||||
Ref: --trailing-error8372
|
||||
Ref: --data-size8610
|
||||
Node: Program design19519
|
||||
Node: Memory requirements21818
|
||||
Node: Minimum file sizes23503
|
||||
Node: File format25506
|
||||
Ref: coded-dict-size26945
|
||||
Node: Trailing data28195
|
||||
Node: Examples30531
|
||||
Ref: concat-example31964
|
||||
Node: Problems32721
|
||||
Node: Concept index33276
|
||||
Node: Introduction1207
|
||||
Node: Output5956
|
||||
Node: Invoking plzip7519
|
||||
Ref: --trailing-error8365
|
||||
Ref: --data-size8603
|
||||
Node: Argument syntax19941
|
||||
Node: File format21886
|
||||
Ref: coded-dict-size23386
|
||||
Node: Program design24637
|
||||
Node: Memory requirements26933
|
||||
Node: Minimum file sizes28643
|
||||
Node: Trailing data30648
|
||||
Node: Examples32991
|
||||
Ref: concat-example34420
|
||||
Node: Problems35174
|
||||
Node: Concept index35729
|
||||
|
||||
End Tag Table
|
||||
|
||||
|
|
|
|||
359
doc/plzip.texi
359
doc/plzip.texi
|
|
@ -6,8 +6,8 @@
|
|||
@finalout
|
||||
@c %**end of header
|
||||
|
||||
@set UPDATED 21 January 2024
|
||||
@set VERSION 1.11
|
||||
@set UPDATED 19 November 2024
|
||||
@set VERSION 1.12-rc1
|
||||
|
||||
@dircategory Compression
|
||||
@direntry
|
||||
|
|
@ -36,17 +36,18 @@
|
|||
This manual is for Plzip (version @value{VERSION}, @value{UPDATED}).
|
||||
|
||||
@menu
|
||||
* Introduction:: Purpose and features of plzip
|
||||
* Output:: Meaning of plzip's output
|
||||
* Invoking plzip:: Command-line interface
|
||||
* Program design:: Internal structure of plzip
|
||||
* Memory requirements:: Memory required to compress and decompress
|
||||
* Minimum file sizes:: Minimum file sizes required for full speed
|
||||
* File format:: Detailed format of the compressed file
|
||||
* Trailing data:: Extra data appended to the file
|
||||
* Examples:: A small tutorial with examples
|
||||
* Problems:: Reporting bugs
|
||||
* Concept index:: Index of concepts
|
||||
* Introduction:: Purpose and features of plzip
|
||||
* Output:: Meaning of plzip's output
|
||||
* Invoking plzip:: Command-line interface
|
||||
* Argument syntax:: By convention, options start with a hyphen
|
||||
* File format:: Detailed format of the compressed file
|
||||
* Program design:: Internal structure of plzip
|
||||
* Memory requirements:: Memory required to compress and decompress
|
||||
* Minimum file sizes:: Minimum file sizes required for full speed
|
||||
* Trailing data:: Extra data appended to the file
|
||||
* Examples:: A small tutorial with examples
|
||||
* Problems:: Reporting bugs
|
||||
* Concept index:: Index of concepts
|
||||
@end menu
|
||||
|
||||
@sp 1
|
||||
|
|
@ -61,30 +62,29 @@ distribute, and modify it.
|
|||
@chapter Introduction
|
||||
@cindex introduction
|
||||
|
||||
@uref{http://www.nongnu.org/lzip/plzip.html,,Plzip}
|
||||
is a massively parallel (multi-threaded) implementation of lzip,
|
||||
compatible with lzip 1.4 or newer. Plzip uses the compression library
|
||||
@uref{http://www.nongnu.org/lzip/plzip.html,,Plzip} is a massively parallel
|
||||
(multi-threaded) implementation of lzip. Plzip uses the compression library
|
||||
@uref{http://www.nongnu.org/lzip/lzlib.html,,lzlib}.
|
||||
|
||||
@uref{http://www.nongnu.org/lzip/lzip.html,,Lzip}
|
||||
is a lossless data compressor with a user interface similar to the one
|
||||
of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov
|
||||
chain-Algorithm' (LZMA) stream format to maximize interoperability. The
|
||||
maximum dictionary size is 512 MiB so that any lzip file can be decompressed
|
||||
on 32-bit machines. Lzip provides accurate and robust 3-factor integrity
|
||||
checking. Lzip can compress about as fast as gzip @w{(lzip -0)} or compress most
|
||||
files more than bzip2 @w{(lzip -9)}. Decompression speed is intermediate between
|
||||
gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery
|
||||
perspective. Lzip has been designed, written, and tested with great care to
|
||||
replace gzip and bzip2 as the standard general-purpose compressed format for
|
||||
Unix-like systems.
|
||||
of gzip or bzip2. Lzip uses a simplified form of LZMA (Lempel-Ziv-Markov
|
||||
chain-Algorithm) designed to achieve complete interoperability between
|
||||
implementations. The maximum dictionary size is 512 MiB so that any lzip
|
||||
file can be decompressed on 32-bit machines. Lzip provides accurate and
|
||||
robust 3-factor integrity checking. @w{@samp{lzip -0}} compresses about as fast as
|
||||
gzip, while @w{@samp{lzip -9}} compresses most files more than bzip2. Decompression
|
||||
speed is intermediate between gzip and bzip2. Lzip provides better data
|
||||
recovery capabilities than gzip and bzip2. Lzip has been designed, written,
|
||||
and tested with great care to replace gzip and bzip2 as general-purpose
|
||||
compressed format for Unix-like systems.
|
||||
|
||||
Plzip can compress/decompress large files on multiprocessor machines much
|
||||
faster than lzip, at the cost of a slightly reduced compression ratio (0.4
|
||||
to 2 percent larger compressed files). Note that the number of usable
|
||||
threads is limited by file size; on files larger than a few GB plzip can use
|
||||
hundreds of processors, but on files of only a few MB plzip is no faster
|
||||
than lzip. @xref{Minimum file sizes}.
|
||||
hundreds of processors, but on files smaller than @w{1 MiB} plzip is no faster
|
||||
than lzip (even at compression level -0). @xref{Minimum file sizes}.
|
||||
|
||||
For creation and manipulation of compressed tar archives
|
||||
@uref{http://www.nongnu.org/lzip/manual/tarlz_manual.html,,tarlz} can be more
|
||||
|
|
@ -132,9 +132,9 @@ makes it safer than compressors returning ambiguous warning values (like
|
|||
gzip) when it is used as a back end for other programs like tar or zutils.
|
||||
|
||||
Plzip automatically uses for each file the largest dictionary size that does
|
||||
not exceed neither the file size nor the limit given. Keep in mind that the
|
||||
decompression memory requirement is affected at compression time by the
|
||||
choice of dictionary size limit. @xref{Memory requirements}.
|
||||
not exceed neither the file size nor the limit given. The dictionary size
|
||||
used for decompression is the same dictionary size used for compression.
|
||||
@xref{Memory requirements}.
|
||||
|
||||
When compressing, plzip replaces every file given in the command line
|
||||
with a compressed version of itself, with the name "original_name.lz".
|
||||
|
|
@ -235,11 +235,8 @@ argument means standard input. It can be mixed with other @var{files} and is
|
|||
read just once, the first time it appears in the command line. Remember to
|
||||
prepend @file{./} to any file name beginning with a hyphen, or use @samp{--}.
|
||||
|
||||
plzip supports the following
|
||||
@uref{http://www.nongnu.org/arg-parser/manual/arg_parser_manual.html#Argument-syntax,,options}:
|
||||
@ifnothtml
|
||||
@xref{Argument syntax,,,arg_parser}.
|
||||
@end ifnothtml
|
||||
@noindent
|
||||
plzip supports the following options: @xref{Argument syntax}.
|
||||
|
||||
@table @code
|
||||
@item -h
|
||||
|
|
@ -286,7 +283,8 @@ already exists and @option{--force} has not been specified, plzip continues
|
|||
decompressing the rest of the files and exits with error status 1. If a file
|
||||
fails to decompress, or is a terminal, plzip exits immediately with error
|
||||
status 2 without decompressing the rest of the files. A terminal is
|
||||
considered an uncompressed file, and therefore invalid.
|
||||
considered an uncompressed file, and therefore invalid. A multimember file
|
||||
with one or more empty members is accepted if redirected to standard input.
|
||||
|
||||
@item -f
|
||||
@itemx --force
|
||||
|
|
@ -295,7 +293,7 @@ Force overwrite of output files.
|
|||
@item -F
|
||||
@itemx --recompress
|
||||
When compressing, force re-compression of files whose name already has
|
||||
the @samp{.lz} or @samp{.tlz} suffix.
|
||||
the @file{.lz} or @file{.tlz} suffix.
|
||||
|
||||
@item -k
|
||||
@itemx --keep
|
||||
|
|
@ -309,7 +307,8 @@ even for multimember files. If more than one file is given, a final line
|
|||
containing the cumulative sizes is printed. With @option{-v}, the dictionary
|
||||
size, the number of members in the file, and the amount of trailing data (if
|
||||
any) are also printed. With @option{-vv}, the positions and sizes of each
|
||||
member in multimember files are also printed.
|
||||
member in multimember files are also printed. A multimember file with one or
|
||||
more empty members is accepted if redirected to standard input.
|
||||
|
||||
If any file is damaged, does not exist, can't be opened, or is not regular,
|
||||
the final exit status is @w{> 0}. @option{-lq} can be used to check quickly
|
||||
|
|
@ -327,7 +326,7 @@ times.
|
|||
@item -n @var{n}
|
||||
@itemx --threads=@var{n}
|
||||
Set the maximum number of worker threads, overriding the system's default.
|
||||
Valid values range from 1 to "as many as your system can support". If this
|
||||
Valid values range from 1 to as many as your system can support. If this
|
||||
option is not used, plzip tries to detect the number of processors in the
|
||||
system and use it as default value. When compressing on a @w{32 bit} system,
|
||||
plzip tries to limit the memory use to under @w{2.22 GiB} (4 worker threads
|
||||
|
|
@ -353,10 +352,10 @@ to @option{-c}. @option{-o} has no effect when testing or listing.
|
|||
|
||||
In order to keep backward compatibility with plzip versions prior to 1.9,
|
||||
when compressing from standard input and no other file names are given, the
|
||||
extension @samp{.lz} is appended to @var{file} unless it already ends in
|
||||
@samp{.lz} or @samp{.tlz}. This feature will be removed in a future version
|
||||
extension @file{.lz} is appended to @var{file} unless it already ends in
|
||||
@file{.lz} or @file{.tlz}. This feature will be removed in a future version
|
||||
of plzip. Meanwhile, redirection may be used instead of @option{-o} to write
|
||||
the compressed output to a file without the extension @samp{.lz} in its
|
||||
the compressed output to a file without the extension @file{.lz} in its
|
||||
name: @w{@samp{plzip < file > foo}}.
|
||||
|
||||
@item -q
|
||||
|
|
@ -386,7 +385,8 @@ together with @option{-v} to see information about the files. If a file
|
|||
fails the test, does not exist, can't be opened, or is a terminal, plzip
|
||||
continues testing the rest of the files. A final diagnostic is shown at
|
||||
verbosity level 1 or higher if any file fails the test when testing multiple
|
||||
files.
|
||||
files. A multimember file with one or more empty members is accepted if
|
||||
redirected to standard input.
|
||||
|
||||
@item -v
|
||||
@itemx --verbose
|
||||
|
|
@ -416,7 +416,7 @@ given, the last setting is used. For example @w{@option{-9 -s64MiB}} is
|
|||
equivalent to @w{@option{-s64MiB -m273}}
|
||||
|
||||
@multitable {Level} {Dictionary size (-s)} {Match length limit (-m)}
|
||||
@item Level @tab Dictionary size (-s) @tab Match length limit (-m)
|
||||
@headitem Level @tab Dictionary size (-s) @tab Match length limit (-m)
|
||||
@item -0 @tab 64 KiB @tab 16 bytes
|
||||
@item -1 @tab 1 MiB @tab 5 bytes
|
||||
@item -2 @tab 1.5 MiB @tab 6 bytes
|
||||
|
|
@ -437,7 +437,7 @@ Aliases for GNU gzip compatibility.
|
|||
When decompressing, testing, or listing, allow trailing data whose first
|
||||
bytes are so similar to the magic bytes of a lzip header that they can
|
||||
be confused with a corrupt header. Use this option if a file triggers a
|
||||
"corrupt header" error and the cause is not indeed a corrupt header.
|
||||
'corrupt header' error and the cause is not indeed a corrupt header.
|
||||
|
||||
@item --in-slots=@var{n}
|
||||
Number of @w{1 MiB} input packets buffered per worker thread when
|
||||
|
|
@ -474,7 +474,7 @@ and may be followed by a multiplier and an optional @samp{B} for "byte".
|
|||
Table of SI and binary prefixes (unit multipliers):
|
||||
|
||||
@multitable {Prefix} {kilobyte (10^3 = 1000)} {|} {Prefix} {kibibyte (2^10 = 1024)}
|
||||
@item Prefix @tab Value @tab | @tab Prefix @tab Value
|
||||
@headitem Prefix @tab Value @tab | @tab Prefix @tab Value
|
||||
@item k @tab kilobyte (10^3 = 1000) @tab | @tab Ki @tab kibibyte (2^10 = 1024)
|
||||
@item M @tab megabyte (10^6) @tab | @tab Mi @tab mebibyte (2^20)
|
||||
@item G @tab gigabyte (10^9) @tab | @tab Gi @tab gibibyte (2^30)
|
||||
|
|
@ -494,6 +494,148 @@ indicate a corrupt or invalid input file, 3 for an internal consistency
|
|||
error (e.g., bug) which caused plzip to panic.
|
||||
|
||||
|
||||
@node Argument syntax
|
||||
@chapter Syntax of command-line arguments
|
||||
@cindex argument syntax
|
||||
|
||||
POSIX recommends these conventions for command-line arguments.
|
||||
|
||||
@itemize @bullet
|
||||
@item A command-line argument is an option if it begins with a hyphen
|
||||
(@samp{-}).
|
||||
|
||||
@item Option names are single alphanumeric characters.
|
||||
|
||||
@item Certain options require an argument.
|
||||
|
||||
@item An option and its argument may or may not appear as separate tokens.
|
||||
(In other words, the whitespace separating them is optional, unless the
|
||||
argument is the empty string).
|
||||
Thus, @w{@option{-o foo}} and @option{-ofoo} are equivalent.
|
||||
|
||||
@item One or more options without arguments, followed by at most one option
|
||||
that takes an argument, may follow a hyphen in a single token.
|
||||
Thus, @option{-abc} is equivalent to @w{@option{-a -b -c}}.
|
||||
|
||||
@item Options typically precede other non-option arguments.
|
||||
|
||||
@item The argument @samp{--} terminates all options; any following arguments
|
||||
are treated as non-option arguments, even if they begin with a hyphen.
|
||||
|
||||
@item A token consisting of a single hyphen character is interpreted as an
|
||||
ordinary non-option argument. By convention, it is used to specify standard
|
||||
input, standard output, or a file named @samp{-}.
|
||||
@end itemize
|
||||
|
||||
@noindent
|
||||
GNU adds @dfn{long options} to these conventions:
|
||||
|
||||
@itemize @bullet
|
||||
@item A long option consists of two hyphens (@samp{--}) followed by a name
|
||||
made of alphanumeric characters and hyphens. Option names are typically one
|
||||
to three words long, with hyphens to separate words. Abbreviations can be
|
||||
used for the long option names as long as the abbreviations are unique.
|
||||
|
||||
@item A long option and its argument may or may not appear as separate
|
||||
tokens. In the latter case they must be separated by an equal sign @samp{=}.
|
||||
Thus, @w{@option{--foo bar}} and @option{--foo=bar} are equivalent.
|
||||
@end itemize
|
||||
|
||||
@noindent
|
||||
The syntax of options with an optional argument is
|
||||
@option{-<short_option><argument>} (without whitespace), or
|
||||
@option{--<long_option>=<argument>}.
|
||||
|
||||
|
||||
@node File format
|
||||
@chapter File format
|
||||
@cindex file format
|
||||
|
||||
Perfection is reached, not when there is no longer anything to add, but
|
||||
when there is no longer anything to take away.@*
|
||||
--- Antoine de Saint-Exupery
|
||||
|
||||
In the diagram below, a box like this:
|
||||
|
||||
@verbatim
|
||||
+---+
|
||||
| | <-- the vertical bars might be missing
|
||||
+---+
|
||||
@end verbatim
|
||||
|
||||
represents one byte; a box like this:
|
||||
|
||||
@verbatim
|
||||
+==============+
|
||||
| |
|
||||
+==============+
|
||||
@end verbatim
|
||||
|
||||
represents a variable number of bytes.
|
||||
|
||||
@noindent
|
||||
A lzip file consists of one or more independent "members" (compressed data
|
||||
sets). The members simply appear one after another in the file, with no
|
||||
additional information before, between, or after them. Each member can
|
||||
encode in compressed form up to @w{16 EiB - 1 byte} of uncompressed data.
|
||||
The size of a multimember file is unlimited. Empty members (data size = 0)
|
||||
are not allowed in multimember files.
|
||||
|
||||
Each member has the following structure:
|
||||
|
||||
@verbatim
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
@end verbatim
|
||||
|
||||
All multibyte values are stored in little endian order.
|
||||
|
||||
@table @samp
|
||||
@item ID string (the "magic" bytes)
|
||||
A four byte string, identifying the lzip format, with the value "LZIP"
|
||||
(0x4C, 0x5A, 0x49, 0x50).
|
||||
|
||||
@item VN (version number, 1 byte)
|
||||
Just in case something needs to be modified in the future. 1 for now.
|
||||
|
||||
@anchor{coded-dict-size}
|
||||
@item DS (coded dictionary size, 1 byte)
|
||||
The dictionary size is calculated by taking a power of 2 (the base size)
|
||||
and subtracting from it a fraction between 0/16 and 7/16 of the base size.@*
|
||||
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@*
|
||||
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
|
||||
from the base size to obtain the dictionary size.@*
|
||||
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
|
||||
Valid values for dictionary size range from 4 KiB to 512 MiB.
|
||||
|
||||
@item LZMA stream
|
||||
The LZMA stream, terminated by an 'End Of Stream' marker. Uses default values
|
||||
for encoder properties.
|
||||
@ifnothtml
|
||||
@xref{Stream format,,,lzip},
|
||||
@end ifnothtml
|
||||
@ifhtml
|
||||
See
|
||||
@uref{http://www.nongnu.org/lzip/manual/lzip_manual.html#Stream-format,,Stream format}
|
||||
@end ifhtml
|
||||
for a complete description.
|
||||
|
||||
@item CRC32 (4 bytes)
|
||||
Cyclic Redundancy Check (CRC) of the original uncompressed data.
|
||||
|
||||
@item Data size (8 bytes)
|
||||
Size of the original uncompressed data.
|
||||
|
||||
@item Member size (8 bytes)
|
||||
Total size of the member, including header and trailer. This field acts
|
||||
as a distributed index, improves the checking of stream integrity, and
|
||||
facilitates the safe recovery of undamaged members from multimember files.
|
||||
Lzip limits the member size to @w{2 PiB} to prevent the data size field from
|
||||
overflowing.
|
||||
@end table
|
||||
|
||||
|
||||
@node Program design
|
||||
@chapter Internal structure of plzip
|
||||
@cindex program design
|
||||
|
|
@ -510,8 +652,8 @@ because lzip usually produces single-member files, which can't be
|
|||
decompressed in parallel.
|
||||
|
||||
For each input file, a splitter thread and several worker threads are
|
||||
created, acting the main thread as muxer (multiplexer) thread. A "packet
|
||||
courier" takes care of data transfers among threads and limits the
|
||||
created, acting the main thread as muxer (multiplexer) thread. A 'packet
|
||||
courier' takes care of data transfers among threads and limits the
|
||||
maximum number of data blocks (packets) being processed simultaneously.
|
||||
|
||||
The splitter reads data blocks from the input file, and distributes them
|
||||
|
|
@ -587,7 +729,7 @@ The following table shows the memory required @strong{per thread} for
|
|||
compression at a given level, using the default data size for each level:
|
||||
|
||||
@multitable {Level} {Memory required}
|
||||
@item Level @tab Memory required
|
||||
@headitem Level @tab Memory required
|
||||
@item -0 @tab 4.875 MiB
|
||||
@item -1 @tab 17.75 MiB
|
||||
@item -2 @tab 26.625 MiB
|
||||
|
|
@ -638,96 +780,6 @@ data size for each level:
|
|||
@end multitable
|
||||
|
||||
|
||||
@node File format
|
||||
@chapter File format
|
||||
@cindex file format
|
||||
|
||||
Perfection is reached, not when there is no longer anything to add, but
|
||||
when there is no longer anything to take away.@*
|
||||
--- Antoine de Saint-Exupery
|
||||
|
||||
@sp 1
|
||||
In the diagram below, a box like this:
|
||||
|
||||
@verbatim
|
||||
+---+
|
||||
| | <-- the vertical bars might be missing
|
||||
+---+
|
||||
@end verbatim
|
||||
|
||||
represents one byte; a box like this:
|
||||
|
||||
@verbatim
|
||||
+==============+
|
||||
| |
|
||||
+==============+
|
||||
@end verbatim
|
||||
|
||||
represents a variable number of bytes.
|
||||
|
||||
@sp 1
|
||||
A lzip file consists of one or more independent "members" (compressed data
|
||||
sets). The members simply appear one after another in the file, with no
|
||||
additional information before, between, or after them. Each member can
|
||||
encode in compressed form up to @w{16 EiB - 1 byte} of uncompressed data.
|
||||
The size of a multimember file is unlimited.
|
||||
|
||||
Each member has the following structure:
|
||||
|
||||
@verbatim
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
|
||||
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
||||
@end verbatim
|
||||
|
||||
All multibyte values are stored in little endian order.
|
||||
|
||||
@table @samp
|
||||
@item ID string (the "magic" bytes)
|
||||
A four byte string, identifying the lzip format, with the value "LZIP"
|
||||
(0x4C, 0x5A, 0x49, 0x50).
|
||||
|
||||
@item VN (version number, 1 byte)
|
||||
Just in case something needs to be modified in the future. 1 for now.
|
||||
|
||||
@anchor{coded-dict-size}
|
||||
@item DS (coded dictionary size, 1 byte)
|
||||
The dictionary size is calculated by taking a power of 2 (the base size)
|
||||
and subtracting from it a fraction between 0/16 and 7/16 of the base size.@*
|
||||
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@*
|
||||
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
|
||||
from the base size to obtain the dictionary size.@*
|
||||
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
|
||||
Valid values for dictionary size range from 4 KiB to 512 MiB.
|
||||
|
||||
@item LZMA stream
|
||||
The LZMA stream, finished by an "End Of Stream" marker. Uses default values
|
||||
for encoder properties.
|
||||
@ifnothtml
|
||||
@xref{Stream format,,,lzip},
|
||||
@end ifnothtml
|
||||
@ifhtml
|
||||
See
|
||||
@uref{http://www.nongnu.org/lzip/manual/lzip_manual.html#Stream-format,,Stream format}
|
||||
@end ifhtml
|
||||
for a complete description.
|
||||
|
||||
@item CRC32 (4 bytes)
|
||||
Cyclic Redundancy Check (CRC) of the original uncompressed data.
|
||||
|
||||
@item Data size (8 bytes)
|
||||
Size of the original uncompressed data.
|
||||
|
||||
@item Member size (8 bytes)
|
||||
Total size of the member, including header and trailer. This field acts
|
||||
as a distributed index, improves the checking of stream integrity, and
|
||||
facilitates the safe recovery of undamaged members from multimember files.
|
||||
Lzip limits the member size to @w{2 PiB} to prevent the data size field from
|
||||
overflowing.
|
||||
|
||||
@end table
|
||||
|
||||
|
||||
@node Trailing data
|
||||
@chapter Extra data appended to the file
|
||||
@cindex trailing data
|
||||
|
|
@ -742,7 +794,7 @@ example when writing to a tape. It is safe to append any amount of
|
|||
padding zero bytes to a lzip file.
|
||||
|
||||
@item
|
||||
Useful data added by the user; an "End Of File" string (to check that the
|
||||
Useful data added by the user; an 'End Of File' string (to check that the
|
||||
file has not been truncated), a cryptographically secure hash, a description
|
||||
of file contents, etc. It is safe to append any amount of text to a lzip
|
||||
file as long as none of the first four bytes of the text matches the
|
||||
|
|
@ -794,9 +846,8 @@ compression can only be detected by comparing the compressed file with the
|
|||
original because the corruption happens before plzip compresses the RAM
|
||||
contents, resulting in a valid compressed file containing wrong data.
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 1: Extract all the files from archive @samp{foo.tar.lz}.
|
||||
Example 1: Extract all the files from archive @file{foo.tar.lz}.
|
||||
|
||||
@example
|
||||
tar -xf foo.tar.lz
|
||||
|
|
@ -804,43 +855,38 @@ or
|
|||
plzip -cd foo.tar.lz | tar -xf -
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 2: Replace a regular file with its compressed version @samp{file.lz}
|
||||
Example 2: Replace a regular file with its compressed version @file{file.lz}
|
||||
and show the compression ratio.
|
||||
|
||||
@example
|
||||
plzip -v file
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 3: Like example 2 but the created @samp{file.lz} has a block size of
|
||||
Example 3: Like example 2 but the created @file{file.lz} has a block size of
|
||||
@w{1 MiB}. The compression ratio is not shown.
|
||||
|
||||
@example
|
||||
plzip -B 1MiB file
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 4: Restore a regular file from its compressed version
|
||||
@samp{file.lz}. If the operation is successful, @samp{file.lz} is removed.
|
||||
@file{file.lz}. If the operation is successful, @file{file.lz} is removed.
|
||||
|
||||
@example
|
||||
plzip -d file.lz
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 5: Check the integrity of the compressed file @samp{file.lz} and
|
||||
Example 5: Check the integrity of the compressed file @file{file.lz} and
|
||||
show status.
|
||||
|
||||
@example
|
||||
plzip -tv file.lz
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@anchor{concat-example}
|
||||
@noindent
|
||||
Example 6: The right way of concatenating the decompressed output of two or
|
||||
|
|
@ -853,28 +899,25 @@ Do this instead
|
|||
plzip -cd file1.lz file2.lz file3.lz
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 7: Decompress @samp{file.lz} partially until @w{10 KiB} of
|
||||
Example 7: Decompress @file{file.lz} partially until @w{10 KiB} of
|
||||
decompressed data are produced.
|
||||
|
||||
@example
|
||||
plzip -cd file.lz | dd bs=1024 count=10
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 8: Decompress @samp{file.lz} partially from decompressed byte at
|
||||
Example 8: Decompress @file{file.lz} partially from decompressed byte at
|
||||
offset 10000 to decompressed byte at offset 14999 (5000 bytes are produced).
|
||||
|
||||
@example
|
||||
plzip -cd file.lz | dd bs=1000 skip=10 count=5
|
||||
@end example
|
||||
|
||||
@sp 1
|
||||
@noindent
|
||||
Example 9: Compress a whole device in /dev/sdc and send the output to
|
||||
@samp{file.lz}.
|
||||
@file{file.lz}.
|
||||
|
||||
@example
|
||||
plzip -c /dev/sdc > file.lz
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue