lzip/plzip
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.2~rc1.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-24 04:04:02 +01:00
parent 79610bce87
commit 945344fb87
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
17 changed files with 288 additions and 217 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
ChangeLog
View file

@ -1,3 +1,8 @@
2014-05-08 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.2-rc1 released.
* Minor changes.
2014-01-20 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.2-pre1 released.

12
INSTALL
View file

@ -45,12 +45,12 @@ the main archive.
Another way
-----------
You can also compile plzip into a separate directory. To do this, you
must use a version of 'make' that supports the 'VPATH' variable, such
as GNU 'make'. 'cd' to the directory where you want the object files
and executables to go and run the 'configure' script. 'configure'
automatically checks for the source code in '.', in '..' and in the
directory that 'configure' is in.
You can also compile plzip into a separate directory.
To do this, you must use a version of 'make' that supports the 'VPATH'
variable, such as GNU 'make'. 'cd' to the directory where you want the
object files and executables to go and run the 'configure' script.
'configure' automatically checks for the source code in '.', in '..' and
in the directory that 'configure' is in.
'configure' recognizes the option '--srcdir=DIR' to control where to
look for the sources. Usually 'configure' can determine that directory

8
Makefile.in
View file

@ -18,16 +18,16 @@ objs = arg_parser.o file_index.o compress.o dec_stdout.o dec_stream.o \
all : $(progname)
$(progname) : $(objs)
$(CXX) $(LDFLAGS) -o $@ $(objs) $(LIBS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ $(objs) $(LIBS)
$(progname)_profiled : $(objs)
$(CXX) $(LDFLAGS) -pg -o $@ $(objs) $(LIBS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) -pg -o $@ $(objs) $(LIBS)
main.o : main.cc
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -DPROGVERSION=\"$(pkgversion)\" -c -o $@ $<
$(CXX) $(CXXFLAGS) $(CPPFLAGS) -DPROGVERSION=\"$(pkgversion)\" -c -o $@ $<
%.o : %.cc
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c -o $@ $<
$(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -o $@ $<
$(objs) : Makefile
arg_parser.o : arg_parser.h

39
README
View file

@ -13,29 +13,34 @@ but on files of only a few MB plzip is no faster than lzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
The lzip file format is designed for long-term data archiving and
provides very safe integrity checking. The member trailer stores the
32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
The lzip file format is designed for long-term data archiving, taking
into account both data integrity and decoder availability:
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
* The lzip format provides very safe integrity checking and some data
recovery means. The lziprecover program can repair bit-flip errors
(one of the most common forms of data corruption) in lzip files,
and provides data recovery capabilities, including error-checked
merging of damaged copies of a file.
* The lzip format is as simple as possible (but not simpler). The
lzip manual provides the code of a simple decompressor along with a
detailed explanation of how it works, so that with the only help of
the lzip manual it would be possible for a digital archaeologist to
extract the data from a lzip file long after quantum computers
eventually render LZMA obsolete.
* Additionally lzip is copylefted, which guarantees that it will
remain free forever.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer than compressors returning ambiguous warning
values (like gzip) when it is used as a back end for tar or zutils.
Plzip will automatically use the smallest possible dictionary size for
each file without exceeding the given limit. Keep in mind that the
decompression memory requirement is affected at compression time by the
choice of dictionary size limit.
When compressing, plzip replaces every file given in the command line
with a compressed version of itself, with the name "original_name.lz".
When decompressing, plzip attempts to guess the name for the decompressed

5
arg_parser.cc
View file

@ -120,7 +120,7 @@ bool Arg_parser::parse_short_option( const char * const opt, const char * const
if( index < 0 )
{
error_ = "invalid option -- "; error_ += c;
error_ = "invalid option -- '"; error_ += c; error_ += '\'';
return false;
}
@ -135,7 +135,8 @@ bool Arg_parser::parse_short_option( const char * const opt, const char * const
{
if( !arg || !arg[0] )
{
error_ = "option requires an argument -- "; error_ += c;
error_ = "option requires an argument -- '"; error_ += c;
error_ += '\'';
return false;
}
data.back().argument = arg; ++argind; cind = 0;

8
compress.cc
View file

@ -52,7 +52,7 @@ int readblock( const int fd, uint8_t * const buf, const int size )
const int n = read( fd, buf + size - rest, rest );
if( n > 0 ) rest -= n;
else if( n == 0 ) break; // EOF
else if( errno != EINTR && errno != EAGAIN ) break;
else if( errno != EINTR ) break;
errno = 0;
}
return size - rest;
@ -70,7 +70,7 @@ int writeblock( const int fd, const uint8_t * const buf, const int size )
{
const int n = write( fd, buf + size - rest, rest );
if( n > 0 ) rest -= n;
else if( n < 0 && errno != EINTR && errno != EAGAIN ) break;
else if( n < 0 && errno != EINTR ) break;
errno = 0;
}
return size - rest;
@ -421,7 +421,7 @@ extern "C" void * cworker( void * arg )
}
if( LZ_compress_close( encoder ) < 0 )
{ pp( "LZ_compress_close failed" ); cleanup_and_fail(); }
{ pp( "LZ_compress_close failed." ); cleanup_and_fail(); }
if( verbosity >= 2 && packet->size > 0 ) show_progress( packet->size );
packet->data = new_data;
@ -440,7 +440,7 @@ void muxer( Packet_courier & courier, const Pretty_print & pp, const int outfd )
while( true )
{
courier.deliver_packets( packet_vector );
if( packet_vector.size() == 0 ) break; // all workers exited
if( packet_vector.empty() ) break; // all workers exited
for( unsigned i = 0; i < packet_vector.size(); ++i )
{

2
configure vendored
View file

@ -6,7 +6,7 @@
# to copy, distribute and modify it.
pkgname=plzip
pkgversion=1.2-pre1
pkgversion=1.2-rc1
progname=plzip
srctrigger=doc/${pkgname}.texi

18
dec_stdout.cc
View file

@ -172,10 +172,10 @@ extern "C" void * dworker_o( void * arg )
LZ_Decoder * const decoder = LZ_decompress_open();
if( !new_data || !ibuffer || !decoder ||
LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
int new_pos = 0;
for( int i = worker_id; i < file_index.members(); i += num_workers )
for( long i = worker_id; i < file_index.members(); i += num_workers )
{
long long member_pos = file_index.mblock( i ).pos();
long long member_rest = file_index.mblock( i ).size();
@ -193,7 +193,7 @@ extern "C" void * dworker_o( void * arg )
member_pos += size;
member_rest -= size;
if( LZ_decompress_write( decoder, ibuffer, size ) != size )
internal_error( "library error (LZ_decompress_write)" );
internal_error( "library error (LZ_decompress_write)." );
}
if( member_rest <= 0 ) { LZ_decompress_finish( decoder ); break; }
}
@ -205,7 +205,7 @@ extern "C" void * dworker_o( void * arg )
cleanup_and_fail( decompress_read_error( decoder, pp, worker_id ) );
new_pos += rd;
if( new_pos > max_packet_size )
internal_error( "opacket size exceeded in worker" );
internal_error( "opacket size exceeded in worker." );
if( new_pos == max_packet_size ||
LZ_decompress_finished( decoder ) == 1 )
{
@ -217,7 +217,7 @@ extern "C" void * dworker_o( void * arg )
courier.collect_packet( opacket, worker_id );
new_pos = 0;
new_data = new( std::nothrow ) uint8_t[max_packet_size];
if( !new_data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !new_data ) { pp( "Not enough memory." ); cleanup_and_fail(); }
}
if( LZ_decompress_finished( decoder ) == 1 )
{
@ -236,9 +236,9 @@ extern "C" void * dworker_o( void * arg )
delete[] ibuffer; delete[] new_data;
if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
{ pp( "Error, some data remains in decoder." ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
{ pp( "LZ_decompress_close failed." ); cleanup_and_fail(); }
courier.worker_finished();
return 0;
}
@ -278,7 +278,7 @@ int dec_stdout( const int num_workers, const int infd, const int outfd,
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i )
{
worker_args[i].file_index = &file_index;
@ -324,6 +324,6 @@ int dec_stdout( const int num_workers, const int infd, const int outfd,
courier.ocheck_counter,
courier.owait_counter );
if( !courier.finished() ) internal_error( "courier not finished" );
if( !courier.finished() ) internal_error( "courier not finished." );
return 0;
}

30
dec_stream.cc
View file

@ -249,7 +249,7 @@ extern "C" void * dsplitter_s( void * arg )
const int buffer_size = max_packet_size;
const int base_buffer_size = tsize + buffer_size + hsize;
uint8_t * const base_buffer = new( std::nothrow ) uint8_t[base_buffer_size];
if( !base_buffer ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !base_buffer ) { pp( "Not enough memory." ); cleanup_and_fail(); }
uint8_t * const buffer = base_buffer + tsize;
int size = readblock( infd, buffer, buffer_size + hsize ) - hsize;
@ -257,10 +257,10 @@ extern "C" void * dsplitter_s( void * arg )
if( size != buffer_size && errno )
{ pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
if( size + hsize < min_member_size )
{ pp( "Input file is too short" ); cleanup_and_fail( 2 ); }
{ pp( "Input file is too short." ); cleanup_and_fail( 2 ); }
const File_header & header = *(File_header *)buffer;
if( !header.verify_magic() )
{ pp( "Bad magic number (file not in lzip format)" ); cleanup_and_fail( 2 ); }
{ pp( "Bad magic number (file not in lzip format)." ); cleanup_and_fail( 2 ); }
if( !header.verify_version() )
{
if( verbosity >= 0 )
@ -293,7 +293,7 @@ extern "C" void * dsplitter_s( void * arg )
cleanup_and_fail( 2 );
}
uint8_t * const data = new( std::nothrow ) uint8_t[newpos - pos];
if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !data ) { pp( "Not enough memory." ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, newpos - pos );
courier.receive_packet( data, newpos - pos );
courier.receive_packet( 0, 0 ); // end of member token
@ -306,7 +306,7 @@ extern "C" void * dsplitter_s( void * arg )
if( at_stream_end )
{
uint8_t * data = new( std::nothrow ) uint8_t[size + hsize - pos];
if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !data ) { pp( "Not enough memory." ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, size + hsize - pos );
courier.receive_packet( data, size + hsize - pos );
courier.receive_packet( 0, 0 ); // end of member token
@ -316,7 +316,7 @@ extern "C" void * dsplitter_s( void * arg )
{
partial_member_size += buffer_size - pos;
uint8_t * data = new( std::nothrow ) uint8_t[buffer_size - pos];
if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !data ) { pp( "Not enough memory." ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, buffer_size - pos );
courier.receive_packet( data, buffer_size - pos );
}
@ -352,7 +352,7 @@ extern "C" void * dworker_s( void * arg )
uint8_t * new_data = new( std::nothrow ) uint8_t[max_packet_size];
LZ_Decoder * const decoder = LZ_decompress_open();
if( !new_data || !decoder || LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
int new_pos = 0;
bool trailing_garbage_found = false;
@ -369,10 +369,10 @@ extern "C" void * dworker_s( void * arg )
{
const int wr = LZ_decompress_write( decoder, ipacket->data + written,
ipacket->size - written );
if( wr < 0 ) internal_error( "library error (LZ_decompress_write)" );
if( wr < 0 ) internal_error( "library error (LZ_decompress_write)." );
written += wr;
if( written > ipacket->size )
internal_error( "ipacket size exceeded in worker" );
internal_error( "ipacket size exceeded in worker." );
}
while( !trailing_garbage_found ) // read and pack decompressed data
{
@ -387,7 +387,7 @@ extern "C" void * dworker_s( void * arg )
}
else new_pos += rd;
if( new_pos > max_packet_size )
internal_error( "opacket size exceeded in worker" );
internal_error( "opacket size exceeded in worker." );
if( new_pos == max_packet_size || trailing_garbage_found ||
LZ_decompress_finished( decoder ) == 1 )
{
@ -399,7 +399,7 @@ extern "C" void * dworker_s( void * arg )
courier.collect_packet( opacket, worker_id );
new_pos = 0;
new_data = new( std::nothrow ) uint8_t[max_packet_size];
if( !new_data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
if( !new_data ) { pp( "Not enough memory." ); cleanup_and_fail(); }
}
if( trailing_garbage_found ||
LZ_decompress_finished( decoder ) == 1 )
@ -422,9 +422,9 @@ extern "C" void * dworker_s( void * arg )
delete[] new_data;
if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
{ pp( "Error, some data remains in decoder." ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
{ pp( "LZ_decompress_close failed." ); cleanup_and_fail(); }
return 0;
}
@ -481,7 +481,7 @@ int dec_stream( const int num_workers, const int infd, const int outfd,
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i )
{
worker_args[i].courier = &courier;
@ -529,6 +529,6 @@ int dec_stream( const int num_workers, const int infd, const int outfd,
courier.ocheck_counter,
courier.owait_counter );
if( !courier.finished() ) internal_error( "courier not finished" );
if( !courier.finished() ) internal_error( "courier not finished." );
return 0;
}

20
decompress.cc
View file

@ -50,7 +50,7 @@ int preadblock( const int fd, uint8_t * const buf, const int size,
const int n = pread( fd, buf + size - rest, rest, pos + size - rest );
if( n > 0 ) rest -= n;
else if( n == 0 ) break; // EOF
else if( errno != EINTR && errno != EAGAIN ) break;
else if( errno != EINTR ) break;
errno = 0;
}
return size - rest;
@ -69,7 +69,7 @@ int pwriteblock( const int fd, const uint8_t * const buf, const int size,
{
const int n = pwrite( fd, buf + size - rest, rest, pos + size - rest );
if( n > 0 ) rest -= n;
else if( n < 0 && errno != EINTR && errno != EAGAIN ) break;
else if( n < 0 && errno != EINTR ) break;
errno = 0;
}
return size - rest;
@ -122,9 +122,9 @@ extern "C" void * dworker( void * arg )
LZ_Decoder * const decoder = LZ_decompress_open();
if( !ibuffer || !obuffer || !decoder ||
LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
for( int i = worker_id; i < file_index.members(); i += num_workers )
for( long i = worker_id; i < file_index.members(); i += num_workers )
{
long long data_pos = file_index.dblock( i ).pos();
long long data_rest = file_index.dblock( i ).size();
@ -144,7 +144,7 @@ extern "C" void * dworker( void * arg )
member_pos += size;
member_rest -= size;
if( LZ_decompress_write( decoder, ibuffer, size ) != size )
internal_error( "library error (LZ_decompress_write)" );
internal_error( "library error (LZ_decompress_write)." );
}
if( member_rest <= 0 ) { LZ_decompress_finish( decoder ); break; }
}
@ -160,7 +160,7 @@ extern "C" void * dworker( void * arg )
{
pp();
if( verbosity >= 0 )
std::fprintf( stderr, "Write error in worker %d: %s\n",
std::fprintf( stderr, "Write error in worker %d: %s.\n",
worker_id, std::strerror( errno ) );
cleanup_and_fail();
}
@ -173,7 +173,7 @@ extern "C" void * dworker( void * arg )
if( LZ_decompress_finished( decoder ) == 1 )
{
if( data_rest != 0 )
internal_error( "final data_rest != 0" );
internal_error( "final data_rest is not zero." );
LZ_decompress_reset( decoder ); // prepare for new member
break;
}
@ -184,9 +184,9 @@ extern "C" void * dworker( void * arg )
delete[] obuffer; delete[] ibuffer;
if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
{ pp( "Error, some data remains in decoder." ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
{ pp( "LZ_decompress_close failed." ); cleanup_and_fail(); }
return 0;
}
@ -224,7 +224,7 @@ int decompress( int num_workers, const int infd, const int outfd,
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); cleanup_and_fail(); }
{ pp( "Not enough memory." ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i )
{
worker_args[i].file_index = &file_index;

12
doc/plzip.1
View file

@ -1,7 +1,7 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1.
.TH PLZIP "1" "January 2014" "Plzip 1.2-pre1" "User Commands"
.TH PLZIP "1" "May 2014" "plzip 1.2-rc1" "User Commands"
.SH NAME
Plzip \- reduces the size of files
plzip \- reduces the size of files
.SH SYNOPSIS
.B plzip
[\fIoptions\fR] [\fIfiles\fR]
@ -85,20 +85,20 @@ Plzip home page: http://www.nongnu.org/lzip/plzip.html
Copyright \(co 2009 Laszlo Ersek.
.br
Copyright \(co 2014 Antonio Diaz Diaz.
Using Lzlib 1.6\-pre1
Using Lzlib 1.6\-rc1
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
.br
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
.SH "SEE ALSO"
The full documentation for
.B Plzip
.B plzip
is maintained as a Texinfo manual. If the
.B info
and
.B Plzip
.B plzip
programs are properly installed at your site, the command
.IP
.B info Plzip
.B info plzip
.PP
should give you access to the complete manual.

69
doc/plzip.info
View file

@ -11,7 +11,7 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir)
Plzip Manual
************
This manual is for Plzip (version 1.2-pre1, 20 January 2014).
This manual is for Plzip (version 1.2-rc1, 8 May 2014).
* Menu:
@ -48,29 +48,46 @@ but on files of only a few MB plzip is no faster than lzip.
fully compatible with lzip-1.4 or newer, and can be rescued with
lziprecover.
The lzip file format is designed for long-term data archiving and
provides very safe integrity checking. The member trailer stores the
32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
The lzip file format is designed for long-term data archiving, taking
into account both data integrity and decoder availability:
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
* The lzip format provides very safe integrity checking and some data
recovery means. The lziprecover program can repair bit-flip errors
(one of the most common forms of data corruption) in lzip files,
and provides data recovery capabilities, including error-checked
merging of damaged copies of a file.
* The lzip format is as simple as possible (but not simpler). The
lzip manual provides the code of a simple decompressor along with
a detailed explanation of how it works, so that with the only help
of the lzip manual it would be possible for a digital
archaeologist to extract the data from a lzip file long after
quantum computers eventually render LZMA obsolete.
* Additionally lzip is copylefted, which guarantees that it will
remain free forever.
The member trailer stores the 32-bit CRC of the original data, the
size of the original data and the size of the member. These values,
together with the value remaining in the range decoder and the
end-of-stream marker, provide a 4 factor integrity checking which
guarantees that the decompressed version of the data is identical to
the original. This guards against corruption of the compressed data,
and against undetected bugs in plzip (hopefully very unlikely). The
chances of data corruption going undetected are microscopic. Be aware,
though, that the check occurs upon decompression, so it can only tell
you that something is wrong. It can't help you recover the original
uncompressed data.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer than compressors returning ambiguous warning
values (like gzip) when it is used as a back end for tar or zutils.
Plzip will automatically use the smallest possible dictionary size
for each file without exceeding the given limit. Keep in mind that the
decompression memory requirement is affected at compression time by the
choice of dictionary size limit.
When compressing, plzip replaces every file given in the command line
with a compressed version of itself, with the name "original_name.lz".
When decompressing, plzip attempts to guess the name for the
@ -215,7 +232,7 @@ The format for running plzip is:
'--dictionary-size=BYTES'
Set the dictionary size limit in bytes. Valid values range from 4
KiB to 512 MiB. Plzip will use the smallest possible dictionary
size for each member without exceeding this limit. Note that
size for each file without exceeding this limit. Note that
dictionary sizes are quantized. If the specified size does not
match one of the valid sizes, it will be rounded upwards by adding
up to (BYTES / 16) to it.
@ -401,13 +418,13 @@ Concept index

Tag Table:
Node: Top221
Node: Introduction878
Node: Program design4650
Node: Invoking plzip5704
Ref: --data-size6149
Node: File format11300
Node: Problems13805
Node: Concept index14334
Node: Introduction872
Node: Program design5441
Node: Invoking plzip6495
Ref: --data-size6940
Node: File format12089
Node: Problems14594
Node: Concept index15123

End Tag Table

61
doc/plzip.texi
View file

@ -6,8 +6,8 @@
@finalout
@c %**end of header
@set UPDATED 20 January 2014
@set VERSION 1.2-pre1
@set UPDATED 8 May 2014
@set VERSION 1.2-rc1
@dircategory Data Compression
@direntry
@ -68,29 +68,50 @@ but on files of only a few MB plzip is no faster than lzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
The lzip file format is designed for long-term data archiving and
provides very safe integrity checking. The member trailer stores the
32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
The lzip file format is designed for long-term data archiving, taking
into account both data integrity and decoder availability:
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
@itemize @bullet
@item
The lzip format provides very safe integrity checking and some data
recovery means. The lziprecover program can repair bit-flip errors (one
of the most common forms of data corruption) in lzip files, and provides
data recovery capabilities, including error-checked merging of damaged
copies of a file.
@item
The lzip format is as simple as possible (but not simpler). The lzip
manual provides the code of a simple decompressor along with a detailed
explanation of how it works, so that with the only help of the lzip
manual it would be possible for a digital archaeologist to extract the
data from a lzip file long after quantum computers eventually render
LZMA obsolete.
@item
Additionally lzip is copylefted, which guarantees that it will remain
free forever.
@end itemize
The member trailer stores the 32-bit CRC of the original data, the size
of the original data and the size of the member. These values, together
with the value remaining in the range decoder and the end-of-stream
marker, provide a 4 factor integrity checking which guarantees that the
decompressed version of the data is identical to the original. This
guards against corruption of the compressed data, and against undetected
bugs in plzip (hopefully very unlikely). The chances of data corruption
going undetected are microscopic. Be aware, though, that the check
occurs upon decompression, so it can only tell you that something is
wrong. It can't help you recover the original uncompressed data.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer than compressors returning ambiguous warning
values (like gzip) when it is used as a back end for tar or zutils.
Plzip will automatically use the smallest possible dictionary size for
each file without exceeding the given limit. Keep in mind that the
decompression memory requirement is affected at compression time by the
choice of dictionary size limit.
When compressing, plzip replaces every file given in the command line
with a compressed version of itself, with the name "original_name.lz".
When decompressing, plzip attempts to guess the name for the decompressed
@ -238,7 +259,7 @@ Quiet operation. Suppress all messages.
@itemx --dictionary-size=@var{bytes}
Set the dictionary size limit in bytes. Valid values range from 4 KiB to
512 MiB. Plzip will use the smallest possible dictionary size for each
member without exceeding this limit. Note that dictionary sizes are
file without exceeding this limit. Note that dictionary sizes are
quantized. If the specified size does not match one of the valid sizes,
it will be rounded upwards by adding up to (@var{bytes} / 16) to it.

89
file_index.cc
View file

@ -38,95 +38,84 @@ int seek_read( const int fd, uint8_t * const buf, const int size,
}
const char * format_num( unsigned long long num,
unsigned long long limit = -1ULL,
const int set_prefix = 0 )
void File_index::set_errno_error( const char * const msg )
{
const char * const si_prefix[8] =
{ "k", "M", "G", "T", "P", "E", "Z", "Y" };
const char * const binary_prefix[8] =
{ "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" };
static bool si = true;
static char buf[32];
error_ = msg; error_ += std::strerror( errno ); error_ += '.';
retval_ = 1;
}
if( set_prefix ) si = ( set_prefix > 0 );
const unsigned factor = ( si ? 1000 : 1024 );
const char * const * prefix = ( si ? si_prefix : binary_prefix );
const char * p = "";
bool exact = ( num % factor == 0 );
for( int i = 0; i < 8 && ( num > limit || ( exact && num >= factor ) ); ++i )
{ num /= factor; if( num % factor != 0 ) exact = false; p = prefix[i]; }
snprintf( buf, sizeof buf, "%llu %s", num, p );
return buf;
void File_index::set_num_error( const char * const msg1, unsigned long long num,
const char * const msg2 )
{
char buf[80];
snprintf( buf, sizeof buf, "%s%llu%s", msg1, num, msg2 );
error_ = buf;
retval_ = 2;
}
File_index::File_index( const int infd ) : retval_( 0 )
File_index::File_index( const int infd )
: retval_( 0 )
{
const long long isize = lseek( infd, 0, SEEK_END );
if( isize < 0 )
{ error_ = "Input file is not seekable :";
error_ += std::strerror( errno ); retval_ = 1; return; }
if( isize > INT64_MAX )
{ error_ = "Input file is too long (2^63 bytes or more)";
retval_ = 2; return; }
long long pos = isize; // always points to a header or EOF
File_header header;
File_trailer trailer;
{ set_errno_error( "Input file is not seekable :" ); return; }
if( isize < min_member_size )
{ error_ = "Input file is too short"; retval_ = 2; return; }
{ error_ = "Input file is too short."; retval_ = 2; return; }
if( isize > INT64_MAX )
{ error_ = "Input file is too long (2^63 bytes or more).";
retval_ = 2; return; }
File_header header;
if( seek_read( infd, header.data, File_header::size, 0 ) != File_header::size )
{ error_ = "Error reading member header :";
error_ += std::strerror( errno ); retval_ = 1; return; }
{ set_errno_error( "Error reading member header :" ); return; }
if( !header.verify_magic() )
{ error_ = "Bad magic number (file not in lzip format)";
{ error_ = "Bad magic number (file not in lzip format).";
retval_ = 2; return; }
if( !header.verify_version() )
{ error_ = "Version "; error_ += format_num( header.version() );
error_ += "member format not supported"; retval_ = 2; return; }
{ set_num_error( "Version ", header.version(),
" member format not supported." ); return; }
long long pos = isize; // always points to a header or to EOF
while( pos >= min_member_size )
{
File_trailer trailer;
if( seek_read( infd, trailer.data, File_trailer::size,
pos - File_trailer::size ) != File_trailer::size )
{ error_ = "Error reading member trailer :";
error_ += std::strerror( errno ); retval_ = 1; break; }
{ set_errno_error( "Error reading member trailer :" ); break; }
const long long member_size = trailer.member_size();
if( member_size < min_member_size || member_size > pos )
{
if( member_vector.size() == 0 ) // maybe trailing garbage
if( member_vector.empty() ) // maybe trailing garbage
{ --pos; continue; }
error_ = "Member size in trailer is corrupt at pos ";
error_ += format_num( pos - 8 ); retval_ = 2; break;
set_num_error( "Member size in trailer is corrupt at pos ", pos - 8 );
break;
}
if( seek_read( infd, header.data, File_header::size,
pos - member_size ) != File_header::size )
{ error_ = "Error reading member header :";
error_ += std::strerror( errno ); retval_ = 1; break; }
{ set_errno_error( "Error reading member header :" ); break; }
if( !header.verify_magic() || !header.verify_version() )
{
if( member_vector.size() == 0 ) // maybe trailing garbage
if( member_vector.empty() ) // maybe trailing garbage
{ --pos; continue; }
error_ = "Bad header at pos ";
error_ += format_num( pos - member_size ); retval_ = 2; break;
set_num_error( "Bad header at pos ", pos - member_size );
break;
}
if( member_vector.size() == 0 && isize - pos > File_header::size &&
if( member_vector.empty() && isize - pos > File_header::size &&
seek_read( infd, header.data, File_header::size, pos ) == File_header::size &&
header.verify_magic() && header.verify_version() )
{
error_ = "Last member in input file is truncated or corrupt";
error_ = "Last member in input file is truncated or corrupt.";
retval_ = 2; break;
}
pos -= member_size;
member_vector.push_back( Member( 0, trailer.data_size(),
pos, member_size ) );
}
if( pos != 0 || member_vector.size() == 0 )
if( pos != 0 || member_vector.empty() )
{
member_vector.clear();
if( retval_ == 0 ) { error_ = "Can't create file index"; retval_ = 2; }
if( retval_ == 0 ) { error_ = "Can't create file index."; retval_ = 2; }
return;
}
std::reverse( member_vector.begin(), member_vector.end() );
@ -136,7 +125,7 @@ File_index::File_index( const int infd ) : retval_( 0 )
if( end < 0 || end > INT64_MAX )
{
member_vector.clear();
error_ = "Data in input file is too long (2^63 bytes or more)";
error_ = "Data in input file is too long (2^63 bytes or more).";
retval_ = 2; return;
}
member_vector[i+1].dblock.pos( end );

14
file_index.h
View file

@ -55,9 +55,14 @@ class File_index
std::string error_;
int retval_;
public:
File_index( const int infd );
void set_errno_error( const char * const msg );
void set_num_error( const char * const msg1, unsigned long long num,
const char * const msg2 = "." );
public:
explicit File_index( const int infd );
long members() const { return member_vector.size(); }
const std::string & error() const { return error_; }
int retval() const { return retval_; }
@ -69,9 +74,8 @@ public:
{ if( member_vector.size() ) return member_vector.back().mblock.end();
else return 0; }
const Block & dblock( const int i ) const
const Block & dblock( const long i ) const
{ return member_vector[i].dblock; }
const Block & mblock( const int i ) const
const Block & mblock( const long i ) const
{ return member_vector[i].mblock; }
int members() const { return (int)member_vector.size(); }
};

27
main.cc
View file

@ -143,7 +143,7 @@ void show_help( const long num_online )
void show_version()
{
std::printf( "%s %s\n", Program_name, PROGVERSION );
std::printf( "%s %s\n", program_name, PROGVERSION );
std::printf( "Copyright (C) 2009 Laszlo Ersek.\n"
"Copyright (C) %s Antonio Diaz Diaz.\n", program_year );
std::printf( "Using Lzlib %s\n", LZ_version() );
@ -245,8 +245,7 @@ int open_instream( const char * const name, struct stat * const in_statsp,
}
else
{
do infd = open( name, O_RDONLY | O_BINARY );
while( infd < 0 && errno == EINTR );
infd = open( name, O_RDONLY | O_BINARY );
if( infd < 0 )
{
if( verbosity >= 0 )
@ -308,8 +307,7 @@ bool open_outstream( const bool force )
int flags = O_CREAT | O_WRONLY | O_BINARY;
if( force ) flags |= O_TRUNC; else flags |= O_EXCL;
do outfd = open( output_filename.c_str(), flags, outfd_mode );
while( outfd < 0 && errno == EINTR );
outfd = open( output_filename.c_str(), flags, outfd_mode );
if( outfd < 0 && verbosity >= 0 )
{
if( errno == EEXIST )
@ -402,7 +400,7 @@ void Pretty_print::operator()( const char * const msg ) const
std::fprintf( stderr, " " );
if( !msg ) std::fflush( stderr );
}
if( msg ) std::fprintf( stderr, "%s.\n", msg );
if( msg ) std::fprintf( stderr, "%s\n", msg );
}
}
@ -415,7 +413,7 @@ void show_error( const char * const msg, const int errcode, const bool help )
{
std::fprintf( stderr, "%s: %s", program_name, msg );
if( errcode > 0 )
std::fprintf( stderr, ": %s", std::strerror( errcode ) );
std::fprintf( stderr, ": %s.", std::strerror( errcode ) );
std::fprintf( stderr, "\n" );
}
if( help )
@ -428,7 +426,7 @@ void show_error( const char * const msg, const int errcode, const bool help )
void internal_error( const char * const msg )
{
if( verbosity >= 0 )
std::fprintf( stderr, "%s: internal error: %s.\n", program_name, msg );
std::fprintf( stderr, "%s: internal error: %s\n", program_name, msg );
std::exit( 3 );
}
@ -467,7 +465,7 @@ void show_progress( const int packet_size,
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
if( p ) // initialize static vars
{ csize = cfile_size; pos = 0; pp = p; return; }
{ csize = cfile_size; pos = 0; pp = p; }
if( pp )
{
xlock( &mutex );
@ -513,7 +511,7 @@ int main( const int argc, const char * const argv[] )
invocation_name = argv[0];
if( LZ_version()[0] != LZ_version_string[0] )
internal_error( "bad library version" );
internal_error( "bad library version." );
const long num_online = std::max( 1L, sysconf( _SC_NPROCESSORS_ONLN ) );
long max_workers = sysconf( _SC_THREAD_THREADS_MAX );
@ -589,7 +587,7 @@ int main( const int argc, const char * const argv[] )
case 't': program_mode = m_test; break;
case 'v': if( verbosity < 4 ) ++verbosity; break;
case 'V': show_version(); return 0;
default : internal_error( "uncaught option" );
default : internal_error( "uncaught option." );
}
} // end process options
@ -604,7 +602,8 @@ int main( const int argc, const char * const argv[] )
if( data_size <= 0 )
data_size = 2 * std::max( 65536, encoder_options.dictionary_size );
else if( data_size < encoder_options.dictionary_size )
encoder_options.dictionary_size = std::max( data_size, LZ_min_dictionary_size() );
encoder_options.dictionary_size =
std::max( data_size, LZ_min_dictionary_size() );
if( num_workers <= 0 )
num_workers = std::min( num_online, max_workers );
@ -629,13 +628,13 @@ int main( const int argc, const char * const argv[] )
struct stat in_stats;
output_filename.clear();
if( !filenames[i].size() || filenames[i] == "-" )
if( filenames[i].empty() || filenames[i] == "-" )
{
input_filename.clear();
infd = STDIN_FILENO;
if( program_mode != m_test )
{
if( to_stdout || !default_output_filename.size() )
if( to_stdout || default_output_filename.empty() )
outfd = STDOUT_FILENO;
else
{

86
testsuite/check.sh
View file

@ -12,7 +12,7 @@ testdir=`cd "$1" ; pwd`
LZIP="${objdir}"/plzip
framework_failure() { echo "failure in testing framework" ; exit 1 ; }
if [ ! -x "${LZIP}" ] ; then
if [ ! -f "${LZIP}" ] || [ ! -x "${LZIP}" ] ; then
echo "${LZIP}: cannot execute"
exit 1
fi
@ -27,22 +27,29 @@ fail=0
printf "testing plzip-%s..." "$2"
"${LZIP}" -cqs-1 in > /dev/null
if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs0 in > /dev/null
if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs4095 in > /dev/null
if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqm4 in > /dev/null
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cqm274 in > /dev/null
if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cqs-1 in > /dev/null
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cqs0 in > /dev/null
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cqs4095 in > /dev/null
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cqs513MiB in > /dev/null
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
printf " in: Bad magic number (file not in lzip format).\n" > msg
"${LZIP}" -t in 2> out
if [ $? = 2 ] && cmp out msg ; then printf . ; else printf - ; fail=1 ; fi
printf " (stdin): Bad magic number (file not in lzip format).\n" > msg
"${LZIP}" -t < in 2> out
if [ $? = 2 ] && cmp out msg ; then printf . ; else printf - ; fail=1 ; fi
rm -f out msg
"${LZIP}" -cdq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cdq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
dd if="${in_lz}" bs=1 count=6 2> /dev/null | "${LZIP}" -tq
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
dd if="${in_lz}" bs=1 count=20 2> /dev/null | "${LZIP}" -tq
@ -53,8 +60,38 @@ if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
cmp in copy || fail=1
printf .
cat "${in_lz}" > copy.lz || framework_failure
printf "to be overwritten" > copy || framework_failure
"${LZIP}" -df copy.lz || fail=1
cmp in copy || fail=1
printf .
printf "to be overwritten" > copy || framework_failure
"${LZIP}" -df -o copy < "${in_lz}" || fail=1
cmp in copy || fail=1
printf .
"${LZIP}" < in > anyothername || fail=1
"${LZIP}" -d anyothername || fail=1
cmp in anyothername.out || fail=1
printf .
cat in in > in2 || framework_failure
"${LZIP}" -o copy2 < in2 || fail=1
"${LZIP}" -t copy2.lz || fail=1
printf .
"${LZIP}" -cd copy2.lz > copy2 || fail=1
cmp in2 copy2 || fail=1
printf .
printf "garbage" >> copy2.lz || framework_failure
printf "to be overwritten" > copy2 || framework_failure
"${LZIP}" -df copy2.lz || fail=1
cmp in2 copy2 || fail=1
printf .
"${LZIP}" -cfq "${in_lz}" > out
if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 1 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cF "${in_lz}" > out || fail=1
"${LZIP}" -cd out | "${LZIP}" -d > copy || fail=1
cmp in copy || fail=1
@ -79,7 +116,6 @@ printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
"${LZIP}" -$i < in > out || fail=1
printf "garbage" >> out || fail=1
"${LZIP}" -d < out > copy || fail=1
cmp in copy || fail=1
done
@ -87,17 +123,11 @@ printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
"${LZIP}" -f -$i -o out < in || fail=1
printf "g" >> out.lz || fail=1
"${LZIP}" -df -o copy < out.lz || fail=1
cmp in copy || fail=1
done
printf .
"${LZIP}" < in > anyothername || fail=1
"${LZIP}" -d anyothername || fail=1
cmp in anyothername.out || fail=1
printf .
cat in in in in > in4 || framework_failure
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ; do
"${LZIP}" -c -s4Ki -B8Ki -n$i in4 > out4.lz || fail=1
@ -122,9 +152,9 @@ cat "${in_lz}" > ingin.lz || framework_failure
printf "g" >> ingin.lz || framework_failure
cat "${in_lz}" >> ingin.lz || framework_failure
"${LZIP}" -tq ingin.lz
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cdq ingin.lz > out
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -t < ingin.lz || fail=1
"${LZIP}" -d < ingin.lz > copy || fail=1
cmp in copy || fail=1
@ -132,13 +162,13 @@ printf .
dd if="${in_lz}" bs=1024 count=10 > trunc.lz 2> /dev/null || framework_failure
"${LZIP}" -tq trunc.lz
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -cdq trunc.lz > out
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -tq < trunc.lz
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -dq < trunc.lz > out
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
echo
if [ ${fail} = 0 ] ; then