Merging upstream version 1.3~pre1.
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
parent
f04d94e9dd
commit
e4e17ab53e
GPG key ID:
FBB4F0E80A80222F
17 changed files with 387 additions and 259 deletions
145
doc/plzip.info
145
doc/plzip.info
|
|
@ -11,7 +11,7 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir)
|
|||
Plzip Manual
|
||||
************
|
||||
|
||||
This manual is for Plzip (version 1.2, 29 August 2014).
|
||||
This manual is for Plzip (version 1.3-pre1, 25 November 2014).
|
||||
|
||||
* Menu:
|
||||
|
||||
|
|
@ -19,6 +19,8 @@ This manual is for Plzip (version 1.2, 29 August 2014).
|
|||
* Program design:: Internal structure of plzip
|
||||
* Invoking plzip:: Command line interface
|
||||
* File format:: Detailed format of the compressed file
|
||||
* Memory requirements:: Memory required to compress and decompress
|
||||
* Minimum file sizes:: Minimum file sizes required for full speed
|
||||
* Problems:: Reporting bugs
|
||||
* Concept index:: Index of concepts
|
||||
|
||||
|
|
@ -40,16 +42,18 @@ the one of lzip, bzip2 or gzip.
|
|||
|
||||
Plzip can compress/decompress large files on multiprocessor machines
|
||||
much faster than lzip, at the cost of a slightly reduced compression
|
||||
ratio. Note that the number of usable threads is limited by file size;
|
||||
on files larger than a few GB plzip can use hundreds of processors, but
|
||||
on files of only a few MB plzip is no faster than lzip.
|
||||
ratio (0.4 to 2 percent larger compressed files). Note that the number
|
||||
of usable threads is limited by file size; on files larger than a few GB
|
||||
plzip can use hundreds of processors, but on files of only a few MB
|
||||
plzip is no faster than lzip (*note Minimum file sizes::).
|
||||
|
||||
Plzip uses the lzip file format; the files produced by plzip are
|
||||
fully compatible with lzip-1.4 or newer, and can be rescued with
|
||||
lziprecover.
|
||||
|
||||
The lzip file format is designed for long-term data archiving, taking
|
||||
into account both data integrity and decoder availability:
|
||||
The lzip file format is designed for data sharing and long-term
|
||||
archiving, taking into account both data integrity and decoder
|
||||
availability:
|
||||
|
||||
* The lzip format provides very safe integrity checking and some data
|
||||
recovery means. The lziprecover program can repair bit-flip errors
|
||||
|
|
@ -64,50 +68,23 @@ into account both data integrity and decoder availability:
|
|||
archaeologist to extract the data from a lzip file long after
|
||||
quantum computers eventually render LZMA obsolete.
|
||||
|
||||
* Additionally lzip is copylefted, which guarantees that it will
|
||||
remain free forever.
|
||||
* Additionally the lzip reference implementation is copylefted, which
|
||||
guarantees that it will remain free forever.
|
||||
|
||||
A nice feature of the lzip format is that a corrupt byte is easier to
|
||||
repair the nearer it is from the beginning of the file. Therefore, with
|
||||
the help of lziprecover, losing an entire archive just because of a
|
||||
corrupt byte near the beginning is a thing of the past.
|
||||
|
||||
The member trailer stores the 32-bit CRC of the original data, the
|
||||
size of the original data and the size of the member. These values,
|
||||
together with the value remaining in the range decoder and the
|
||||
end-of-stream marker, provide a 4 factor integrity checking which
|
||||
guarantees that the decompressed version of the data is identical to
|
||||
the original. This guards against corruption of the compressed data,
|
||||
and against undetected bugs in plzip (hopefully very unlikely). The
|
||||
chances of data corruption going undetected are microscopic. Be aware,
|
||||
though, that the check occurs upon decompression, so it can only tell
|
||||
you that something is wrong. It can't help you recover the original
|
||||
uncompressed data.
|
||||
|
||||
Plzip uses the same well-defined exit status values used by lzip and
|
||||
bzip2, which makes it safer than compressors returning ambiguous warning
|
||||
values (like gzip) when it is used as a back end for other programs like
|
||||
tar or zutils.
|
||||
|
||||
The amount of memory required *per thread* is approximately the
|
||||
following:
|
||||
|
||||
* For compression; 3 times the data size (*note --data-size::) plus
|
||||
11 times the dictionary size.
|
||||
|
||||
* For decompression or testing of a non-seekable file or of standard
|
||||
input; 2 times the dictionary size plus up to 32 MiB.
|
||||
|
||||
* For decompression of a regular file to a non-seekable file or to
|
||||
standard output; the dictionary size plus up to 32 MiB.
|
||||
|
||||
* For decompression of a regular file to another regular file, or for
|
||||
testing of a regular file; the dictionary size.
|
||||
|
||||
Plzip will automatically use the smallest possible dictionary size
|
||||
for each file without exceeding the given limit. Keep in mind that the
|
||||
decompression memory requirement is affected at compression time by the
|
||||
choice of dictionary size limit.
|
||||
choice of dictionary size limit (*note Memory requirements::).
|
||||
|
||||
When compressing, plzip replaces every file given in the command line
|
||||
with a compressed version of itself, with the name "original_name.lz".
|
||||
|
|
@ -245,8 +222,8 @@ The format for running plzip is:
|
|||
value.
|
||||
|
||||
Note that the number of usable threads is limited to
|
||||
ceil( file_size / data_size ) during compression (*note
|
||||
--data-size::), and to the number of members in the input during
|
||||
ceil( file_size / data_size ) during compression (*note Minimum
|
||||
file sizes::), and to the number of members in the input during
|
||||
decompression.
|
||||
|
||||
'-o FILE'
|
||||
|
|
@ -287,8 +264,8 @@ The format for running plzip is:
|
|||
When compressing, show the compression ratio for each file
|
||||
processed. A second '-v' shows the progress of compression.
|
||||
When decompressing or testing, further -v's (up to 4) increase the
|
||||
verbosity level, showing status, compression ratio, decompressed
|
||||
size, and compressed size.
|
||||
verbosity level, showing status, compression ratio, dictionary
|
||||
size, decompressed size, and compressed size.
|
||||
|
||||
'-1 .. -9'
|
||||
Set the compression parameters (dictionary size and match length
|
||||
|
|
@ -299,8 +276,7 @@ The format for running plzip is:
|
|||
linear scale optimal for all files. If your files are large, very
|
||||
repetitive, etc, you may need to use the '--match-length' and
|
||||
'--dictionary-size' options directly to achieve optimal
|
||||
performance. For example, '-9m64' usually compresses executables
|
||||
more (and faster) than '-9'.
|
||||
performance.
|
||||
|
||||
Level Dictionary size Match length limit
|
||||
-1 1 MiB 5 bytes
|
||||
|
|
@ -340,7 +316,7 @@ invalid input file, 3 for an internal consistency error (eg, bug) which
|
|||
caused plzip to panic.
|
||||
|
||||
|
||||
File: plzip.info, Node: File format, Next: Problems, Prev: Invoking plzip, Up: Top
|
||||
File: plzip.info, Node: File format, Next: Memory requirements, Prev: Invoking plzip, Up: Top
|
||||
|
||||
4 File format
|
||||
*************
|
||||
|
|
@ -413,9 +389,70 @@ additional information before, between, or after them.
|
|||
|
||||
|
||||
|
||||
File: plzip.info, Node: Problems, Next: Concept index, Prev: File format, Up: Top
|
||||
File: plzip.info, Node: Memory requirements, Next: Minimum file sizes, Prev: File format, Up: Top
|
||||
|
||||
5 Reporting bugs
|
||||
5 Memory required to compress and decompress
|
||||
********************************************
|
||||
|
||||
The amount of memory required *per thread* is approximately the
|
||||
following:
|
||||
|
||||
* For compression; 11 times the dictionary size plus 3 times the
|
||||
data size (*note --data-size::).
|
||||
|
||||
* For decompression of a regular (seekable) file to another regular
|
||||
file, or for testing of a regular file; the dictionary size. Note
|
||||
that regular files with more than 1024 bytes of trailing garbage
|
||||
are treated as non-seekable.
|
||||
|
||||
* For testing of a non-seekable file or of standard input; the
|
||||
dictionary size plus up to 5 MiB.
|
||||
|
||||
* For decompression of a regular file to a non-seekable file or to
|
||||
standard output; the dictionary size plus up to 32 MiB.
|
||||
|
||||
* For decompression of a non-seekable file or of standard input; the
|
||||
dictionary size plus up to 35 MiB.
|
||||
|
||||
|
||||
File: plzip.info, Node: Minimum file sizes, Next: Problems, Prev: Memory requirements, Up: Top
|
||||
|
||||
6 Minimum file sizes required for full compression speed
|
||||
********************************************************
|
||||
|
||||
When compressing, plzip divides the input file into chunks and
|
||||
compresses as many chunks simultaneously as worker threads are chosen,
|
||||
creating a multi-member compressed file.
|
||||
|
||||
For this to work as expected (and roughly multiply the compression
|
||||
speed by the number of available processors), the uncompressed file
|
||||
must be at least as large as the number of worker threads times the
|
||||
chunk size (*note --data-size::). Else some processors will not get any
|
||||
data to compress, and compression will be proportionally slower. The
|
||||
maximum speed increase achievable on a given file is limited by the
|
||||
ratio (file_size / data_size).
|
||||
|
||||
The following table shows the minimum uncompressed file size needed
|
||||
for full use of N processors at a given compression level, using the
|
||||
default data size for each level:
|
||||
|
||||
Processors 2 3 4 8 16 64
|
||||
-------------------------------------------------------------------------
|
||||
Level
|
||||
-1 4 MiB 6 MiB 8 MiB 16 MiB 32 MiB 128 MiB
|
||||
-2 6 MiB 9 MiB 12 MiB 24 MiB 48 MiB 192 MiB
|
||||
-3 8 MiB 12 MiB 16 MiB 32 MiB 64 MiB 256 MiB
|
||||
-4 12 MiB 18 MiB 24 MiB 48 MiB 96 MiB 384 MiB
|
||||
-5 16 MiB 24 MiB 32 MiB 64 MiB 128 MiB 512 MiB
|
||||
-6 32 MiB 48 MiB 64 MiB 128 MiB 256 MiB 1 GiB
|
||||
-7 64 MiB 96 MiB 128 MiB 256 MiB 512 MiB 2 GiB
|
||||
-8 96 MiB 144 MiB 192 MiB 384 MiB 768 MiB 3 GiB
|
||||
-9 128 MiB 192 MiB 256 MiB 512 MiB 1 GiB 4 GiB
|
||||
|
||||
|
||||
File: plzip.info, Node: Problems, Next: Concept index, Prev: Minimum file sizes, Up: Top
|
||||
|
||||
7 Reporting bugs
|
||||
****************
|
||||
|
||||
There are probably bugs in plzip. There are certainly errors and
|
||||
|
|
@ -441,6 +478,8 @@ Concept index
|
|||
* getting help: Problems. (line 6)
|
||||
* introduction: Introduction. (line 6)
|
||||
* invoking: Invoking plzip. (line 6)
|
||||
* memory requirements: Memory requirements. (line 6)
|
||||
* minimum file sizes: Minimum file sizes. (line 6)
|
||||
* options: Invoking plzip. (line 6)
|
||||
* program design: Program design. (line 6)
|
||||
* usage: Invoking plzip. (line 6)
|
||||
|
|
@ -450,13 +489,15 @@ Concept index
|
|||
|
||||
Tag Table:
|
||||
Node: Top221
|
||||
Node: Introduction847
|
||||
Node: Program design6279
|
||||
Node: Invoking plzip7868
|
||||
Ref: --data-size8313
|
||||
Node: File format13471
|
||||
Node: Problems15976
|
||||
Node: Concept index16505
|
||||
Node: Introduction994
|
||||
Node: Program design5290
|
||||
Node: Invoking plzip6879
|
||||
Ref: --data-size7324
|
||||
Node: File format12420
|
||||
Node: Memory requirements14936
|
||||
Node: Minimum file sizes15913
|
||||
Node: Problems17765
|
||||
Node: Concept index18301
|
||||
|
||||
End Tag Table
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue