lzip/plzip
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.2.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-24 04:05:33 +01:00
parent 844a3e48f2
commit 73f1304e10
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
19 changed files with 181 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

12
doc/plzip.1
View file

@ -1,10 +1,10 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1.
.TH PLZIP "1" "July 2014" "plzip 1.2-rc2" "User Commands"
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1.
.TH PLZIP "1" "August 2014" "plzip 1.2" "User Commands"
.SH NAME
plzip \- reduces the size of files
.SH SYNOPSIS
.B plzip
[\fIoptions\fR] [\fIfiles\fR]
[\fI\,options\/\fR] [\fI\,files\/\fR]
.SH DESCRIPTION
Plzip \- Parallel compressor compatible with lzip.
.SH OPTIONS
@ -16,7 +16,7 @@ display this help and exit
output version information and exit
.TP
\fB\-B\fR, \fB\-\-data\-size=\fR<bytes>
set input data block size in bytes
set size of input data blocks, in bytes
.TP
\fB\-c\fR, \fB\-\-stdout\fR
send output to standard output
@ -37,7 +37,7 @@ keep (don't delete) input files
set match length limit in bytes [36]
.TP
\fB\-n\fR, \fB\-\-threads=\fR<n>
set number of (de)compression threads [1]
set number of (de)compression threads [2]
.TP
\fB\-o\fR, \fB\-\-output=\fR<file>
if reading stdin, place the output into <file>
@ -85,7 +85,7 @@ Plzip home page: http://www.nongnu.org/lzip/plzip.html
Copyright \(co 2009 Laszlo Ersek.
.br
Copyright \(co 2014 Antonio Diaz Diaz.
Using Lzlib 1.6\-rc2
Using Lzlib 1.6
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>
.br
This is free software: you are free to change and redistribute it.

80
doc/plzip.info
View file

@ -11,7 +11,7 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir)
Plzip Manual
************
This manual is for Plzip (version 1.2-rc2, 1 July 2014).
This manual is for Plzip (version 1.2, 29 August 2014).
* Menu:
@ -23,7 +23,7 @@ This manual is for Plzip (version 1.2-rc2, 1 July 2014).
* Concept index:: Index of concepts
Copyright (C) 2009, 2010, 2011, 2012, 2013, 2014 Antonio Diaz Diaz.
Copyright (C) 2009-2014 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission to
copy, distribute and modify it.
@ -34,15 +34,15 @@ File: plzip.info, Node: Introduction, Next: Program design, Prev: Top, Up: T
1 Introduction
**************
Plzip is a massively parallel (multi-threaded), lossless data compressor
Plzip is a massively parallel (multi-threaded) lossless data compressor
based on the lzlib compression library, with a user interface similar to
the one of lzip, bzip2 or gzip.
Plzip can compress/decompress large files on multiprocessor machines
much faster than lzip, at the cost of a slightly reduced compression
ratio. Note that the number of usable threads is limited by file size,
so on files larger than a few GB plzip can use hundreds of processors,
but on files of only a few MB plzip is no faster than lzip.
ratio. Note that the number of usable threads is limited by file size;
on files larger than a few GB plzip can use hundreds of processors, but
on files of only a few MB plzip is no faster than lzip.
Plzip uses the lzip file format; the files produced by plzip are
fully compatible with lzip-1.4 or newer, and can be rescued with
@ -67,6 +67,11 @@ into account both data integrity and decoder availability:
* Additionally lzip is copylefted, which guarantees that it will
remain free forever.
A nice feature of the lzip format is that a corrupt byte is easier to
repair the nearer it is from the beginning of the file. Therefore, with
the help of lziprecover, losing an entire archive just because of a
corrupt byte near the beginning is a thing of the past.
The member trailer stores the 32-bit CRC of the original data, the
size of the original data and the size of the member. These values,
together with the value remaining in the range decoder and the
@ -81,7 +86,23 @@ uncompressed data.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer than compressors returning ambiguous warning
values (like gzip) when it is used as a back end for tar or zutils.
values (like gzip) when it is used as a back end for other programs like
tar or zutils.
The amount of memory required *per thread* is approximately the
following:
* For compression; 3 times the data size (*note --data-size::) plus
11 times the dictionary size.
* For decompression or testing of a non-seekable file or of standard
input; 2 times the dictionary size plus up to 32 MiB.
* For decompression of a regular file to a non-seekable file or to
standard output; the dictionary size plus up to 32 MiB.
* For decompression of a regular file to another regular file, or for
testing of a regular file; the dictionary size.
Plzip will automatically use the smallest possible dictionary size
for each file without exceeding the given limit. Keep in mind that the
@ -129,7 +150,17 @@ File: plzip.info, Node: Program design, Next: Invoking plzip, Prev: Introduct
2 Program design
****************
For each input file, a splitter thread and several worker threads are
When compressing, plzip divides the input file into chunks and
compresses as many chunks simultaneously as worker threads are chosen,
creating a multi-member compressed file.
When decompressing, plzip decompresses as many members
simultaneously as worker threads are chosen. Files that were compressed
with lzip will not be decompressed faster than using lzip (unless the
'-b' option was used) because lzip usually produces single-member
files, which can't be decompressed in parallel.
For each input file, a splitter thread and several worker threads are
created, acting the main thread as muxer (multiplexer) thread. A "packet
courier" takes care of data transfers among threads and limits the
maximum number of data blocks (packets) being processed simultaneously.
@ -141,10 +172,11 @@ writes them to the output file.
When decompressing from a regular file, the splitter is removed and
the workers read directly from the input file. If the output file is
also a regular file, the muxer is also removed, and the workers write
directly to the output file. With these optimizations, decompression
speed of large files with many members is only limited by the number of
processors available and by I/O speed.
also a regular file, the muxer is also removed and the workers write
directly to the output file. With these optimizations, the use of RAM
is greatly reduced and the decompression speed of large files with many
members is only limited by the number of processors available and by
I/O speed.

File: plzip.info, Node: Invoking plzip, Next: File format, Prev: Program design, Up: Top
@ -168,11 +200,11 @@ The format for running plzip is:
'-B BYTES'
'--data-size=BYTES'
Set the input data block size in bytes. The input file will be
divided in chunks of this size before compression is performed.
Valid values range from 8 KiB to 1 GiB. Default value is two times
the dictionary size. Plzip will reduce the dictionary size if it
is larger than the chosen data size.
Set the size of the input data blocks, in bytes. The input file
will be divided in chunks of this size before compression is
performed. Valid values range from 8 KiB to 1 GiB. Default value
is two times the dictionary size. Plzip will reduce the dictionary
size if it is larger than the chosen data size.
'-c'
'--stdout'
@ -418,13 +450,13 @@ Concept index

Tag Table:
Node: Top221
Node: Introduction873
Node: Program design5442
Node: Invoking plzip6496
Ref: --data-size6941
Node: File format12090
Node: Problems14595
Node: Concept index15124
Node: Introduction847
Node: Program design6279
Node: Invoking plzip7868
Ref: --data-size8313
Node: File format13471
Node: Problems15976
Node: Concept index16505

End Tag Table

72
doc/plzip.texi
View file

@ -6,8 +6,8 @@
@finalout
@c %**end of header
@set UPDATED 1 July 2014
@set VERSION 1.2-rc2
@set UPDATED 29 August 2014
@set VERSION 1.2
@dircategory Data Compression
@direntry
@ -44,8 +44,7 @@ This manual is for Plzip (version @value{VERSION}, @value{UPDATED}).
@end menu
@sp 1
Copyright @copyright{} 2009, 2010, 2011, 2012, 2013, 2014
Antonio Diaz Diaz.
Copyright @copyright{} 2009-2014 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission
to copy, distribute and modify it.
@ -55,15 +54,15 @@ to copy, distribute and modify it.
@chapter Introduction
@cindex introduction
Plzip is a massively parallel (multi-threaded), lossless data compressor
Plzip is a massively parallel (multi-threaded) lossless data compressor
based on the lzlib compression library, with a user interface similar to
the one of lzip, bzip2 or gzip.
Plzip can compress/decompress large files on multiprocessor machines
much faster than lzip, at the cost of a slightly reduced compression
ratio. Note that the number of usable threads is limited by file size,
so on files larger than a few GB plzip can use hundreds of processors,
but on files of only a few MB plzip is no faster than lzip.
ratio. Note that the number of usable threads is limited by file size;
on files larger than a few GB plzip can use hundreds of processors, but
on files of only a few MB plzip is no faster than lzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
@ -92,6 +91,11 @@ Additionally lzip is copylefted, which guarantees that it will remain
free forever.
@end itemize
A nice feature of the lzip format is that a corrupt byte is easier to
repair the nearer it is from the beginning of the file. Therefore, with
the help of lziprecover, losing an entire archive just because of a
corrupt byte near the beginning is a thing of the past.
The member trailer stores the 32-bit CRC of the original data, the size
of the original data and the size of the member. These values, together
with the value remaining in the range decoder and the end-of-stream
@ -105,7 +109,29 @@ wrong. It can't help you recover the original uncompressed data.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer than compressors returning ambiguous warning
values (like gzip) when it is used as a back end for tar or zutils.
values (like gzip) when it is used as a back end for other programs like
tar or zutils.
The amount of memory required @strong{per thread} is approximately the
following:
@itemize @bullet
@item
For compression; 3 times the data size (@pxref{--data-size}) plus 11
times the dictionary size.
@item
For decompression or testing of a non-seekable file or of standard
input; 2 times the dictionary size plus up to 32 MiB.
@item
For decompression of a regular file to a non-seekable file or to
standard output; the dictionary size plus up to 32 MiB.
@item
For decompression of a regular file to another regular file, or for
testing of a regular file; the dictionary size.
@end itemize
Plzip will automatically use the smallest possible dictionary size for
each file without exceeding the given limit. Keep in mind that the
@ -154,6 +180,16 @@ you verify the compressed file with a command like
@chapter Program design
@cindex program design
When compressing, plzip divides the input file into chunks and
compresses as many chunks simultaneously as worker threads are chosen,
creating a multi-member compressed file.
When decompressing, plzip decompresses as many members simultaneously as
worker threads are chosen. Files that were compressed with lzip will not
be decompressed faster than using lzip (unless the @samp{-b} option was
used) because lzip usually produces single-member files, which can't be
decompressed in parallel.
For each input file, a splitter thread and several worker threads are
created, acting the main thread as muxer (multiplexer) thread. A "packet
courier" takes care of data transfers among threads and limits the
@ -166,10 +202,10 @@ writes them to the output file.
When decompressing from a regular file, the splitter is removed and the
workers read directly from the input file. If the output file is also a
regular file, the muxer is also removed, and the workers write directly
to the output file. With these optimizations, decompression speed of
large files with many members is only limited by the number of
processors available and by I/O speed.
regular file, the muxer is also removed and the workers write directly
to the output file. With these optimizations, the use of RAM is greatly
reduced and the decompression speed of large files with many members is
only limited by the number of processors available and by I/O speed.
@node Invoking plzip
@ -199,11 +235,11 @@ Print the version number of plzip on the standard output and exit.
@item -B @var{bytes}
@itemx --data-size=@var{bytes}
@anchor{--data-size}
Set the input data block size in bytes. The input file will be divided
in chunks of this size before compression is performed. Valid values
range from 8 KiB to 1 GiB. Default value is two times the dictionary
size. Plzip will reduce the dictionary size if it is larger than the
chosen data size.
Set the size of the input data blocks, in bytes. The input file will be
divided in chunks of this size before compression is performed. Valid
values range from 8 KiB to 1 GiB. Default value is two times the
dictionary size. Plzip will reduce the dictionary size if it is larger
than the chosen data size.
@item -c
@itemx --stdout