lzip/plzip
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.1~pre1.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-24 04:01:20 +01:00
parent 4bce01c02a
commit e7c68f81ff
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
17 changed files with 356 additions and 277 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
ChangeLog
View file

@ -1,3 +1,9 @@
2013-07-20 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.1-pre1 released.
* Show progress of compression at verbosity level 2 (-vv).
* SIGUSR1 and SIGUSR2 are no more used to signal a fatal error.
2013-05-29 Antonio Diaz Diaz <antonio@gnu.org> 2013-05-29 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.0 released. * Version 1.0 released.

6
INSTALL
View file

@ -1,7 +1,7 @@
Requirements Requirements
------------ ------------
You will need a C++ compiler and the lzlib compression library installed. You will need a C++ compiler and the lzlib compression library installed.
I use gcc 4.8.0 and 3.3.6, but the code should compile with any I use gcc 4.8.1 and 3.3.6, but the code should compile with any
standards compliant compiler. standards compliant compiler.
Lzlib must be version 1.0 or newer. Lzlib must be version 1.0 or newer.
Gcc is available at http://gcc.gnu.org. Gcc is available at http://gcc.gnu.org.
@ -12,9 +12,9 @@ Procedure
--------- ---------
1. Unpack the archive if you have not done so already: 1. Unpack the archive if you have not done so already:
lzip -cd plzip[version].tar.lz | tar -xf - tar -xf plzip[version].tar.lz
or or
gzip -cd plzip[version].tar.gz | tar -xf - lzip -cd plzip[version].tar.lz | tar -xf -
This creates the directory ./plzip[version] containing the source from This creates the directory ./plzip[version] containing the source from
the main archive. the main archive.

16
NEWS
View file

@ -1,15 +1,5 @@
Changes in version 1.0: Changes in version 1.1:
Scalability of compression (max number of useful worker threads) has Plzip now shows the progress of compression at verbosity level 2 (-vv).
been increased.
Scalability when decompressing from/to regular files has been increased. Signals "SIGUSR1" and "SIGUSR2" are no more used to signal a fatal error.
The number of worker threads is now limited to the number of members in
the input file when decompressing from a regular file.
"configure" now accepts options with a separate argument.
The target "install-as-lzip" has been added to the Makefile.
The target "install-bin" has been added to the Makefile.

49
README
View file

@ -1,21 +1,40 @@
Description Description
Plzip is a massively parallel (multi-threaded), lossless data compressor Plzip is a massively parallel (multi-threaded), lossless data compressor
based on the lzlib compression library, with very safe integrity based on the lzlib compression library, with a user interface similar to
checking and a user interface similar to the one of bzip2, gzip or lzip. the one of lzip, bzip2 or gzip.
Plzip is intended for faster compression/decompression of big files on Plzip can compress/decompress large files on multiprocessor machines
multiprocessor machines, which makes it specially well suited for much faster than lzip, at the cost of a slightly reduced compression
distribution of big software files and large scale data archiving. On ratio. On files large enough (several GB), plzip can use hundreds of
files big enough (several GB), plzip can use hundreds of processors. processors. On files of only a few MB it is better to use lzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
Plzip uses the same well-defined exit status values used by lzip and Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer when used in pipes or scripts than bzip2, which makes it safer when used in pipes or scripts than
compressors returning ambiguous warning values, like gzip. compressors returning ambiguous warning values, like gzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
The lzip file format is designed for long-term data archiving and
provides very safe integrity checking. The member trailer stores the
32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
Plzip replaces every file given in the command line with a compressed Plzip replaces every file given in the command line with a compressed
version of itself, with the name "original_name.lz". Each compressed version of itself, with the name "original_name.lz". Each compressed
file has the same modification date, permissions, and, when possible, file has the same modification date, permissions, and, when possible,
@ -33,18 +52,6 @@ or more compressed files. The result is the concatenation of the
corresponding uncompressed files. Integrity testing of concatenated corresponding uncompressed files. Integrity testing of concatenated
compressed files is also supported. compressed files is also supported.
As a self-check for your protection, plzip stores in the member trailer
the 32-bit CRC of the original data, the size of the original data and
the size of the member. These values, together with the value remaining
in the range decoder and the end-of-stream marker, provide a very safe 4
factor integrity checking which guarantees that the decompressed version
of the data is identical to the original. This guards against corruption
of the compressed data, and against undetected bugs in plzip (hopefully
very unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.

54
compress.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009 Laszlo Ersek. Copyright (C) 2009 Laszlo Ersek.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -80,61 +80,70 @@ int writeblock( const int fd, const uint8_t * const buf, const int size )
void xinit( pthread_mutex_t * const mutex ) void xinit( pthread_mutex_t * const mutex )
{ {
const int errcode = pthread_mutex_init( mutex, 0 ); const int errcode = pthread_mutex_init( mutex, 0 );
if( errcode ) { show_error( "pthread_mutex_init", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_mutex_init", errcode ); cleanup_and_fail(); }
} }
void xinit( pthread_cond_t * const cond ) void xinit( pthread_cond_t * const cond )
{ {
const int errcode = pthread_cond_init( cond, 0 ); const int errcode = pthread_cond_init( cond, 0 );
if( errcode ) { show_error( "pthread_cond_init", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_cond_init", errcode ); cleanup_and_fail(); }
} }
void xdestroy( pthread_mutex_t * const mutex ) void xdestroy( pthread_mutex_t * const mutex )
{ {
const int errcode = pthread_mutex_destroy( mutex ); const int errcode = pthread_mutex_destroy( mutex );
if( errcode ) { show_error( "pthread_mutex_destroy", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_mutex_destroy", errcode ); cleanup_and_fail(); }
} }
void xdestroy( pthread_cond_t * const cond ) void xdestroy( pthread_cond_t * const cond )
{ {
const int errcode = pthread_cond_destroy( cond ); const int errcode = pthread_cond_destroy( cond );
if( errcode ) { show_error( "pthread_cond_destroy", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_cond_destroy", errcode ); cleanup_and_fail(); }
} }
void xlock( pthread_mutex_t * const mutex ) void xlock( pthread_mutex_t * const mutex )
{ {
const int errcode = pthread_mutex_lock( mutex ); const int errcode = pthread_mutex_lock( mutex );
if( errcode ) { show_error( "pthread_mutex_lock", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_mutex_lock", errcode ); cleanup_and_fail(); }
} }
void xunlock( pthread_mutex_t * const mutex ) void xunlock( pthread_mutex_t * const mutex )
{ {
const int errcode = pthread_mutex_unlock( mutex ); const int errcode = pthread_mutex_unlock( mutex );
if( errcode ) { show_error( "pthread_mutex_unlock", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_mutex_unlock", errcode ); cleanup_and_fail(); }
} }
void xwait( pthread_cond_t * const cond, pthread_mutex_t * const mutex ) void xwait( pthread_cond_t * const cond, pthread_mutex_t * const mutex )
{ {
const int errcode = pthread_cond_wait( cond, mutex ); const int errcode = pthread_cond_wait( cond, mutex );
if( errcode ) { show_error( "pthread_cond_wait", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_cond_wait", errcode ); cleanup_and_fail(); }
} }
void xsignal( pthread_cond_t * const cond ) void xsignal( pthread_cond_t * const cond )
{ {
const int errcode = pthread_cond_signal( cond ); const int errcode = pthread_cond_signal( cond );
if( errcode ) { show_error( "pthread_cond_signal", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_cond_signal", errcode ); cleanup_and_fail(); }
} }
void xbroadcast( pthread_cond_t * const cond ) void xbroadcast( pthread_cond_t * const cond )
{ {
const int errcode = pthread_cond_broadcast( cond ); const int errcode = pthread_cond_broadcast( cond );
if( errcode ) { show_error( "pthread_cond_broadcast", errcode ); fatal(); } if( errcode )
{ show_error( "pthread_cond_broadcast", errcode ); cleanup_and_fail(); }
} }
@ -317,10 +326,10 @@ extern "C" void * csplitter( void * arg )
for( bool first_post = true; ; first_post = false ) for( bool first_post = true; ; first_post = false )
{ {
uint8_t * const data = new( std::nothrow ) uint8_t[data_size]; uint8_t * const data = new( std::nothrow ) uint8_t[data_size];
if( !data ) { pp( mem_msg ); fatal(); } if( !data ) { pp( mem_msg ); cleanup_and_fail(); }
const int size = readblock( infd, data, data_size ); const int size = readblock( infd, data, data_size );
if( size != data_size && errno ) if( size != data_size && errno )
{ pp(); show_error( "Read error", errno ); fatal(); } { pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
if( size > 0 || first_post ) // first packet may be empty if( size > 0 || first_post ) // first packet may be empty
{ {
@ -365,7 +374,7 @@ extern "C" void * cworker( void * arg )
const int max_compr_size = 42 + packet->size + ( ( packet->size + 7 ) / 8 ); const int max_compr_size = 42 + packet->size + ( ( packet->size + 7 ) / 8 );
uint8_t * const new_data = new( std::nothrow ) uint8_t[max_compr_size]; uint8_t * const new_data = new( std::nothrow ) uint8_t[max_compr_size];
if( !new_data ) { pp( mem_msg ); fatal(); } if( !new_data ) { pp( mem_msg ); cleanup_and_fail(); }
const int dict_size = std::max( LZ_min_dictionary_size(), const int dict_size = std::max( LZ_min_dictionary_size(),
std::min( dictionary_size, packet->size ) ); std::min( dictionary_size, packet->size ) );
LZ_Encoder * const encoder = LZ_Encoder * const encoder =
@ -376,7 +385,7 @@ extern "C" void * cworker( void * arg )
pp( mem_msg ); pp( mem_msg );
else else
internal_error( "invalid argument to encoder" ); internal_error( "invalid argument to encoder" );
fatal(); cleanup_and_fail();
} }
int written = 0; int written = 0;
@ -403,7 +412,7 @@ extern "C" void * cworker( void * arg )
if( verbosity >= 0 ) if( verbosity >= 0 )
std::fprintf( stderr, "LZ_compress_read error: %s.\n", std::fprintf( stderr, "LZ_compress_read error: %s.\n",
LZ_strerror( LZ_compress_errno( encoder ) ) ); LZ_strerror( LZ_compress_errno( encoder ) ) );
fatal(); cleanup_and_fail();
} }
new_pos += rd; new_pos += rd;
if( new_pos > max_compr_size ) if( new_pos > max_compr_size )
@ -412,8 +421,9 @@ extern "C" void * cworker( void * arg )
} }
if( LZ_compress_close( encoder ) < 0 ) if( LZ_compress_close( encoder ) < 0 )
{ pp( "LZ_compress_close failed" ); fatal(); } { pp( "LZ_compress_close failed" ); cleanup_and_fail(); }
if( verbosity >= 2 && packet->size > 0 ) show_progress( packet->size );
packet->data = new_data; packet->data = new_data;
packet->size = new_pos; packet->size = new_pos;
courier.collect_packet( packet ); courier.collect_packet( packet );
@ -441,7 +451,7 @@ void muxer( Packet_courier & courier, const Pretty_print & pp, const int outfd )
{ {
const int wr = writeblock( outfd, opacket->data, opacket->size ); const int wr = writeblock( outfd, opacket->data, opacket->size );
if( wr != opacket->size ) if( wr != opacket->size )
{ pp(); show_error( "Write error", errno ); fatal(); } { pp(); show_error( "Write error", errno ); cleanup_and_fail(); }
} }
delete[] opacket->data; delete[] opacket->data;
delete opacket; delete opacket;
@ -475,7 +485,7 @@ int compress( const int data_size, const int dictionary_size,
pthread_t splitter_thread; pthread_t splitter_thread;
int errcode = pthread_create( &splitter_thread, 0, csplitter, &splitter_arg ); int errcode = pthread_create( &splitter_thread, 0, csplitter, &splitter_arg );
if( errcode ) if( errcode )
{ show_error( "Can't create splitter thread", errcode ); fatal(); } { show_error( "Can't create splitter thread", errcode ); cleanup_and_fail(); }
Worker_arg worker_arg; Worker_arg worker_arg;
worker_arg.courier = &courier; worker_arg.courier = &courier;
@ -484,12 +494,12 @@ int compress( const int data_size, const int dictionary_size,
worker_arg.match_len_limit = match_len_limit; worker_arg.match_len_limit = match_len_limit;
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers]; pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_threads ) { pp( mem_msg ); fatal(); } if( !worker_threads ) { pp( mem_msg ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i ) for( int i = 0; i < num_workers; ++i )
{ {
errcode = pthread_create( worker_threads + i, 0, cworker, &worker_arg ); errcode = pthread_create( worker_threads + i, 0, cworker, &worker_arg );
if( errcode ) if( errcode )
{ show_error( "Can't create worker threads", errcode ); fatal(); } { show_error( "Can't create worker threads", errcode ); cleanup_and_fail(); }
} }
muxer( courier, pp, outfd ); muxer( courier, pp, outfd );
@ -498,13 +508,13 @@ int compress( const int data_size, const int dictionary_size,
{ {
errcode = pthread_join( worker_threads[i], 0 ); errcode = pthread_join( worker_threads[i], 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join worker threads", errcode ); fatal(); } { show_error( "Can't join worker threads", errcode ); cleanup_and_fail(); }
} }
delete[] worker_threads; delete[] worker_threads;
errcode = pthread_join( splitter_thread, 0 ); errcode = pthread_join( splitter_thread, 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join splitter thread", errcode ); fatal(); } { show_error( "Can't join splitter thread", errcode ); cleanup_and_fail(); }
if( verbosity >= 1 ) if( verbosity >= 1 )
{ {

18
configure vendored
View file

@ -1,14 +1,14 @@
#! /bin/sh #! /bin/sh
# configure script for Plzip - A parallel compressor compatible with lzip # configure script for Plzip - Parallel compressor compatible with lzip
# Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. # Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
# #
# This configure script is free software: you have unlimited permission # This configure script is free software: you have unlimited permission
# to copy, distribute and modify it. # to copy, distribute and modify it.
pkgname=plzip pkgname=plzip
pkgversion=1.0 pkgversion=1.1-pre1
progname=plzip progname=plzip
srctrigger=doc/plzip.texinfo srctrigger=doc/${pkgname}.texinfo
# clear some things potentially inherited from environment. # clear some things potentially inherited from environment.
LC_ALL=C LC_ALL=C
@ -100,14 +100,14 @@ while [ $# != 0 ] ; do
*=* | *-*-*) ;; *=* | *-*-*) ;;
*) *)
echo "configure: unrecognized option: '${option}'" 1>&2 echo "configure: unrecognized option: '${option}'" 1>&2
echo "Try 'configure --help' for more information." echo "Try 'configure --help' for more information." 1>&2
exit 1 ;; exit 1 ;;
esac esac
# Check if the option took a separate argument # Check if the option took a separate argument
if [ "${arg2}" = yes ] ; then if [ "${arg2}" = yes ] ; then
if [ $# != 0 ] ; then args="${args} \"$1\"" ; shift if [ $# != 0 ] ; then args="${args} \"$1\"" ; shift
else echo "configure: Missing argument to \"${option}\"" 1>&2 else echo "configure: Missing argument to '${option}'" 1>&2
exit 1 exit 1
fi fi
fi fi
@ -125,10 +125,8 @@ if [ -z "${srcdir}" ] ; then
fi fi
if [ ! -r "${srcdir}/${srctrigger}" ] ; then if [ ! -r "${srcdir}/${srctrigger}" ] ; then
exec 1>&2 echo "configure: Can't find sources in ${srcdir} ${srcdirtext}" 1>&2
echo echo "configure: (At least ${srctrigger} is missing)." 1>&2
echo "configure: Can't find sources in ${srcdir} ${srcdirtext}"
echo "configure: (At least ${srctrigger} is missing)."
exit 1 exit 1
fi fi
@ -166,7 +164,7 @@ echo "CXXFLAGS = ${CXXFLAGS}"
echo "LDFLAGS = ${LDFLAGS}" echo "LDFLAGS = ${LDFLAGS}"
rm -f Makefile rm -f Makefile
cat > Makefile << EOF cat > Makefile << EOF
# Makefile for Plzip - A parallel compressor compatible with lzip # Makefile for Plzip - Parallel compressor compatible with lzip
# Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. # Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
# This file was generated automatically by configure. Do not edit. # This file was generated automatically by configure. Do not edit.
# #

22
dec_stdout.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009 Laszlo Ersek. Copyright (C) 2009 Laszlo Ersek.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -171,7 +171,7 @@ extern "C" void * dworker_o( void * arg )
LZ_Decoder * const decoder = LZ_decompress_open(); LZ_Decoder * const decoder = LZ_decompress_open();
if( !new_data || !ibuffer || !decoder || if( !new_data || !ibuffer || !decoder ||
LZ_decompress_errno( decoder ) != LZ_ok ) LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
int new_pos = 0; int new_pos = 0;
for( int i = worker_id; i < file_index.members(); i += num_workers ) for( int i = worker_id; i < file_index.members(); i += num_workers )
@ -188,7 +188,7 @@ extern "C" void * dworker_o( void * arg )
if( size > 0 ) if( size > 0 )
{ {
if( preadblock( infd, ibuffer, size, member_pos ) != size ) if( preadblock( infd, ibuffer, size, member_pos ) != size )
{ pp(); show_error( "Read error", errno ); fatal(); } { pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
member_pos += size; member_pos += size;
member_rest -= size; member_rest -= size;
if( LZ_decompress_write( decoder, ibuffer, size ) != size ) if( LZ_decompress_write( decoder, ibuffer, size ) != size )
@ -201,7 +201,7 @@ extern "C" void * dworker_o( void * arg )
const int rd = LZ_decompress_read( decoder, new_data + new_pos, const int rd = LZ_decompress_read( decoder, new_data + new_pos,
max_packet_size - new_pos ); max_packet_size - new_pos );
if( rd < 0 ) if( rd < 0 )
fatal( decompress_read_error( decoder, pp, worker_id ) ); cleanup_and_fail( decompress_read_error( decoder, pp, worker_id ) );
new_pos += rd; new_pos += rd;
if( new_pos > max_packet_size ) if( new_pos > max_packet_size )
internal_error( "opacket size exceeded in worker" ); internal_error( "opacket size exceeded in worker" );
@ -216,7 +216,7 @@ extern "C" void * dworker_o( void * arg )
courier.collect_packet( opacket, worker_id ); courier.collect_packet( opacket, worker_id );
new_pos = 0; new_pos = 0;
new_data = new( std::nothrow ) uint8_t[max_packet_size]; new_data = new( std::nothrow ) uint8_t[max_packet_size];
if( !new_data ) { pp( "Not enough memory" ); fatal(); } if( !new_data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
} }
if( LZ_decompress_finished( decoder ) == 1 ) if( LZ_decompress_finished( decoder ) == 1 )
{ {
@ -235,9 +235,9 @@ extern "C" void * dworker_o( void * arg )
delete[] ibuffer; delete[] new_data; delete[] ibuffer; delete[] new_data;
if( LZ_decompress_member_position( decoder ) != 0 ) if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); fatal(); } { pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 ) if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); fatal(); } { pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
courier.worker_finished(); courier.worker_finished();
return 0; return 0;
} }
@ -256,7 +256,7 @@ void muxer( Packet_courier & courier, const Pretty_print & pp, const int outfd )
{ {
const int wr = writeblock( outfd, opacket->data, opacket->size ); const int wr = writeblock( outfd, opacket->data, opacket->size );
if( wr != opacket->size ) if( wr != opacket->size )
{ pp(); show_error( "Write error", errno ); fatal(); } { pp(); show_error( "Write error", errno ); cleanup_and_fail(); }
} }
delete[] opacket->data; delete[] opacket->data;
delete opacket; delete opacket;
@ -280,7 +280,7 @@ int dec_stdout( const int num_workers, const int infd, const int outfd,
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers]; Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers]; pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads ) if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i ) for( int i = 0; i < num_workers; ++i )
{ {
worker_args[i].file_index = &file_index; worker_args[i].file_index = &file_index;
@ -292,7 +292,7 @@ int dec_stdout( const int num_workers, const int infd, const int outfd,
const int errcode = const int errcode =
pthread_create( &worker_threads[i], 0, dworker_o, &worker_args[i] ); pthread_create( &worker_threads[i], 0, dworker_o, &worker_args[i] );
if( errcode ) if( errcode )
{ show_error( "Can't create worker threads", errcode ); fatal(); } { show_error( "Can't create worker threads", errcode ); cleanup_and_fail(); }
} }
muxer( courier, pp, outfd ); muxer( courier, pp, outfd );
@ -301,7 +301,7 @@ int dec_stdout( const int num_workers, const int infd, const int outfd,
{ {
const int errcode = pthread_join( worker_threads[i], 0 ); const int errcode = pthread_join( worker_threads[i], 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join worker threads", errcode ); fatal(); } { show_error( "Can't join worker threads", errcode ); cleanup_and_fail(); }
} }
delete[] worker_threads; delete[] worker_threads;
delete[] worker_args; delete[] worker_args;

71
dec_stream.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009 Laszlo Ersek. Copyright (C) 2009 Laszlo Ersek.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -248,22 +248,31 @@ extern "C" void * dsplitter_s( void * arg )
Packet_courier & courier = *tmp.courier; Packet_courier & courier = *tmp.courier;
const Pretty_print & pp = *tmp.pp; const Pretty_print & pp = *tmp.pp;
const int infd = tmp.infd; const int infd = tmp.infd;
const int hsize = 6; // header size const int hsize = File_header::size;
const int tsize = 20; // trailer size const int tsize = File_trailer::size;
const int buffer_size = max_packet_size; const int buffer_size = max_packet_size;
const int base_buffer_size = tsize + buffer_size + hsize; const int base_buffer_size = tsize + buffer_size + hsize;
uint8_t * const base_buffer = new( std::nothrow ) uint8_t[base_buffer_size]; uint8_t * const base_buffer = new( std::nothrow ) uint8_t[base_buffer_size];
if( !base_buffer ) { pp( "Not enough memory" ); fatal(); } if( !base_buffer ) { pp( "Not enough memory" ); cleanup_and_fail(); }
uint8_t * const buffer = base_buffer + tsize; uint8_t * const buffer = base_buffer + tsize;
int size = readblock( infd, buffer, buffer_size + hsize ) - hsize; int size = readblock( infd, buffer, buffer_size + hsize ) - hsize;
bool at_stream_end = ( size < buffer_size ); bool at_stream_end = ( size < buffer_size );
if( size != buffer_size && errno ) if( size != buffer_size && errno )
{ pp(); show_error( "Read error", errno ); fatal(); } { pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
if( size <= tsize ) if( size + hsize < min_member_size )
{ pp( "Error reading member header" ); fatal(); } { pp( "Input file is too short" ); cleanup_and_fail( 2 ); }
if( find_magic( buffer, 0, 4 ) != 0 ) const File_header & header = *(File_header *)buffer;
{ pp( "Bad magic number (file not in lzip format)" ); fatal(); } if( !header.verify_magic() )
{ pp( "Bad magic number (file not in lzip format)" ); cleanup_and_fail( 2 ); }
if( !header.verify_version() )
{
if( verbosity >= 0 )
{ pp();
std::fprintf( stderr, "Version %d member format not supported.\n",
header.version() ); }
cleanup_and_fail( 2 );
}
unsigned long long partial_member_size = 0; unsigned long long partial_member_size = 0;
while( true ) while( true )
@ -274,13 +283,21 @@ extern "C" void * dsplitter_s( void * arg )
newpos = find_magic( buffer, newpos, size + 4 - newpos ); newpos = find_magic( buffer, newpos, size + 4 - newpos );
if( newpos <= size ) if( newpos <= size )
{ {
unsigned long long member_size = 0; const File_trailer & trailer = *(File_trailer *)(buffer + newpos - tsize);
for( int i = 1; i <= 8; ++i ) const unsigned long long member_size = trailer.member_size();
{ member_size <<= 8; member_size += base_buffer[tsize+newpos-i]; }
if( partial_member_size + newpos - pos == member_size ) if( partial_member_size + newpos - pos == member_size )
{ // header found { // header found
const File_header & header = *(File_header *)(buffer + newpos);
if( !header.verify_version() )
{
if( verbosity >= 0 )
{ pp();
std::fprintf( stderr, "Version %d member format not supported.\n",
header.version() ); }
cleanup_and_fail( 2 );
}
uint8_t * const data = new( std::nothrow ) uint8_t[newpos - pos]; uint8_t * const data = new( std::nothrow ) uint8_t[newpos - pos];
if( !data ) { pp( "Not enough memory" ); fatal(); } if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, newpos - pos ); std::memcpy( data, buffer + pos, newpos - pos );
courier.receive_packet( data, newpos - pos ); courier.receive_packet( data, newpos - pos );
courier.receive_packet( 0, 0 ); // end of member token courier.receive_packet( 0, 0 ); // end of member token
@ -293,7 +310,7 @@ extern "C" void * dsplitter_s( void * arg )
if( at_stream_end ) if( at_stream_end )
{ {
uint8_t * data = new( std::nothrow ) uint8_t[size + hsize - pos]; uint8_t * data = new( std::nothrow ) uint8_t[size + hsize - pos];
if( !data ) { pp( "Not enough memory" ); fatal(); } if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, size + hsize - pos ); std::memcpy( data, buffer + pos, size + hsize - pos );
courier.receive_packet( data, size + hsize - pos ); courier.receive_packet( data, size + hsize - pos );
courier.receive_packet( 0, 0 ); // end of member token courier.receive_packet( 0, 0 ); // end of member token
@ -303,7 +320,7 @@ extern "C" void * dsplitter_s( void * arg )
{ {
partial_member_size += buffer_size - pos; partial_member_size += buffer_size - pos;
uint8_t * data = new( std::nothrow ) uint8_t[buffer_size - pos]; uint8_t * data = new( std::nothrow ) uint8_t[buffer_size - pos];
if( !data ) { pp( "Not enough memory" ); fatal(); } if( !data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
std::memcpy( data, buffer + pos, buffer_size - pos ); std::memcpy( data, buffer + pos, buffer_size - pos );
courier.receive_packet( data, buffer_size - pos ); courier.receive_packet( data, buffer_size - pos );
} }
@ -311,7 +328,7 @@ extern "C" void * dsplitter_s( void * arg )
size = readblock( infd, buffer + hsize, buffer_size ); size = readblock( infd, buffer + hsize, buffer_size );
at_stream_end = ( size < buffer_size ); at_stream_end = ( size < buffer_size );
if( size != buffer_size && errno ) if( size != buffer_size && errno )
{ pp(); show_error( "Read error", errno ); fatal(); } { pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
} }
delete[] base_buffer; delete[] base_buffer;
courier.finish(); // no more packets to send courier.finish(); // no more packets to send
@ -339,7 +356,7 @@ extern "C" void * dworker_s( void * arg )
uint8_t * new_data = new( std::nothrow ) uint8_t[max_packet_size]; uint8_t * new_data = new( std::nothrow ) uint8_t[max_packet_size];
LZ_Decoder * const decoder = LZ_decompress_open(); LZ_Decoder * const decoder = LZ_decompress_open();
if( !new_data || !decoder || LZ_decompress_errno( decoder ) != LZ_ok ) if( !new_data || !decoder || LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
int new_pos = 0; int new_pos = 0;
bool trailing_garbage_found = false; bool trailing_garbage_found = false;
@ -370,7 +387,7 @@ extern "C" void * dworker_s( void * arg )
if( LZ_decompress_errno( decoder ) == LZ_header_error ) if( LZ_decompress_errno( decoder ) == LZ_header_error )
trailing_garbage_found = true; trailing_garbage_found = true;
else else
fatal( decompress_read_error( decoder, pp, worker_id ) ); cleanup_and_fail( decompress_read_error( decoder, pp, worker_id ) );
} }
else new_pos += rd; else new_pos += rd;
if( new_pos > max_packet_size ) if( new_pos > max_packet_size )
@ -386,7 +403,7 @@ extern "C" void * dworker_s( void * arg )
courier.collect_packet( opacket, worker_id ); courier.collect_packet( opacket, worker_id );
new_pos = 0; new_pos = 0;
new_data = new( std::nothrow ) uint8_t[max_packet_size]; new_data = new( std::nothrow ) uint8_t[max_packet_size];
if( !new_data ) { pp( "Not enough memory" ); fatal(); } if( !new_data ) { pp( "Not enough memory" ); cleanup_and_fail(); }
} }
if( trailing_garbage_found || if( trailing_garbage_found ||
LZ_decompress_finished( decoder ) == 1 ) LZ_decompress_finished( decoder ) == 1 )
@ -409,9 +426,9 @@ extern "C" void * dworker_s( void * arg )
delete[] new_data; delete[] new_data;
if( LZ_decompress_member_position( decoder ) != 0 ) if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); fatal(); } { pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 ) if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); fatal(); } { pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
return 0; return 0;
} }
@ -431,7 +448,7 @@ void muxer( Packet_courier & courier, const Pretty_print & pp, const int outfd )
{ {
const int wr = writeblock( outfd, opacket->data, opacket->size ); const int wr = writeblock( outfd, opacket->data, opacket->size );
if( wr != opacket->size ) if( wr != opacket->size )
{ pp(); show_error( "Write error", errno ); fatal(); } { pp(); show_error( "Write error", errno ); cleanup_and_fail(); }
} }
delete[] opacket->data; delete[] opacket->data;
delete opacket; delete opacket;
@ -462,12 +479,12 @@ int dec_stream( const int num_workers, const int infd, const int outfd,
pthread_t splitter_thread; pthread_t splitter_thread;
int errcode = pthread_create( &splitter_thread, 0, dsplitter_s, &splitter_arg ); int errcode = pthread_create( &splitter_thread, 0, dsplitter_s, &splitter_arg );
if( errcode ) if( errcode )
{ show_error( "Can't create splitter thread", errcode ); fatal(); } { show_error( "Can't create splitter thread", errcode ); cleanup_and_fail(); }
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers]; Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers]; pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads ) if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i ) for( int i = 0; i < num_workers; ++i )
{ {
worker_args[i].courier = &courier; worker_args[i].courier = &courier;
@ -475,7 +492,7 @@ int dec_stream( const int num_workers, const int infd, const int outfd,
worker_args[i].worker_id = i; worker_args[i].worker_id = i;
errcode = pthread_create( &worker_threads[i], 0, dworker_s, &worker_args[i] ); errcode = pthread_create( &worker_threads[i], 0, dworker_s, &worker_args[i] );
if( errcode ) if( errcode )
{ show_error( "Can't create worker threads", errcode ); fatal(); } { show_error( "Can't create worker threads", errcode ); cleanup_and_fail(); }
} }
muxer( courier, pp, outfd ); muxer( courier, pp, outfd );
@ -484,14 +501,14 @@ int dec_stream( const int num_workers, const int infd, const int outfd,
{ {
errcode = pthread_join( worker_threads[i], 0 ); errcode = pthread_join( worker_threads[i], 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join worker threads", errcode ); fatal(); } { show_error( "Can't join worker threads", errcode ); cleanup_and_fail(); }
} }
delete[] worker_threads; delete[] worker_threads;
delete[] worker_args; delete[] worker_args;
errcode = pthread_join( splitter_thread, 0 ); errcode = pthread_join( splitter_thread, 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join splitter thread", errcode ); fatal(); } { show_error( "Can't join splitter thread", errcode ); cleanup_and_fail(); }
if( verbosity >= 2 && out_size > 0 && in_size > 0 ) if( verbosity >= 2 && out_size > 0 && in_size > 0 )
std::fprintf( stderr, "%6.3f:1, %6.3f bits/byte, %5.2f%% saved. ", std::fprintf( stderr, "%6.3f:1, %6.3f bits/byte, %5.2f%% saved. ",

22
decompress.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009 Laszlo Ersek. Copyright (C) 2009 Laszlo Ersek.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -122,7 +122,7 @@ extern "C" void * dworker( void * arg )
LZ_Decoder * const decoder = LZ_decompress_open(); LZ_Decoder * const decoder = LZ_decompress_open();
if( !ibuffer || !obuffer || !decoder || if( !ibuffer || !obuffer || !decoder ||
LZ_decompress_errno( decoder ) != LZ_ok ) LZ_decompress_errno( decoder ) != LZ_ok )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
for( int i = worker_id; i < file_index.members(); i += num_workers ) for( int i = worker_id; i < file_index.members(); i += num_workers )
{ {
@ -140,7 +140,7 @@ extern "C" void * dworker( void * arg )
if( size > 0 ) if( size > 0 )
{ {
if( preadblock( infd, ibuffer, size, member_pos ) != size ) if( preadblock( infd, ibuffer, size, member_pos ) != size )
{ pp(); show_error( "Read error", errno ); fatal(); } { pp(); show_error( "Read error", errno ); cleanup_and_fail(); }
member_pos += size; member_pos += size;
member_rest -= size; member_rest -= size;
if( LZ_decompress_write( decoder, ibuffer, size ) != size ) if( LZ_decompress_write( decoder, ibuffer, size ) != size )
@ -152,7 +152,7 @@ extern "C" void * dworker( void * arg )
{ {
const int rd = LZ_decompress_read( decoder, obuffer, buffer_size ); const int rd = LZ_decompress_read( decoder, obuffer, buffer_size );
if( rd < 0 ) if( rd < 0 )
fatal( decompress_read_error( decoder, pp, worker_id ) ); cleanup_and_fail( decompress_read_error( decoder, pp, worker_id ) );
if( rd > 0 && outfd >= 0 ) if( rd > 0 && outfd >= 0 )
{ {
const int wr = pwriteblock( outfd, obuffer, rd, data_pos ); const int wr = pwriteblock( outfd, obuffer, rd, data_pos );
@ -162,7 +162,7 @@ extern "C" void * dworker( void * arg )
if( verbosity >= 0 ) if( verbosity >= 0 )
std::fprintf( stderr, "Write error in worker %d: %s\n", std::fprintf( stderr, "Write error in worker %d: %s\n",
worker_id, std::strerror( errno ) ); worker_id, std::strerror( errno ) );
fatal(); cleanup_and_fail();
} }
} }
if( rd > 0 ) if( rd > 0 )
@ -184,9 +184,9 @@ extern "C" void * dworker( void * arg )
delete[] obuffer; delete[] ibuffer; delete[] obuffer; delete[] ibuffer;
if( LZ_decompress_member_position( decoder ) != 0 ) if( LZ_decompress_member_position( decoder ) != 0 )
{ pp( "Error, some data remains in decoder" ); fatal(); } { pp( "Error, some data remains in decoder" ); cleanup_and_fail(); }
if( LZ_decompress_close( decoder ) < 0 ) if( LZ_decompress_close( decoder ) < 0 )
{ pp( "LZ_decompress_close failed" ); fatal(); } { pp( "LZ_decompress_close failed" ); cleanup_and_fail(); }
return 0; return 0;
} }
@ -208,7 +208,7 @@ int decompress( int num_workers, const int infd, const int outfd,
return dec_stream( num_workers, infd, outfd, pp, debug_level, testing ); return dec_stream( num_workers, infd, outfd, pp, debug_level, testing );
} }
if( file_index.retval() != 0 ) if( file_index.retval() != 0 )
{ show_error( file_index.error().c_str() ); return file_index.retval(); } { pp( file_index.error().c_str() ); return file_index.retval(); }
if( num_workers > file_index.members() ) if( num_workers > file_index.members() )
num_workers = file_index.members(); num_workers = file_index.members();
@ -224,7 +224,7 @@ int decompress( int num_workers, const int infd, const int outfd,
Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers]; Worker_arg * worker_args = new( std::nothrow ) Worker_arg[num_workers];
pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers]; pthread_t * worker_threads = new( std::nothrow ) pthread_t[num_workers];
if( !worker_args || !worker_threads ) if( !worker_args || !worker_threads )
{ pp( "Not enough memory" ); fatal(); } { pp( "Not enough memory" ); cleanup_and_fail(); }
for( int i = 0; i < num_workers; ++i ) for( int i = 0; i < num_workers; ++i )
{ {
worker_args[i].file_index = &file_index; worker_args[i].file_index = &file_index;
@ -236,14 +236,14 @@ int decompress( int num_workers, const int infd, const int outfd,
const int errcode = const int errcode =
pthread_create( &worker_threads[i], 0, dworker, &worker_args[i] ); pthread_create( &worker_threads[i], 0, dworker, &worker_args[i] );
if( errcode ) if( errcode )
{ show_error( "Can't create worker threads", errcode ); fatal(); } { show_error( "Can't create worker threads", errcode ); cleanup_and_fail(); }
} }
for( int i = num_workers - 1; i >= 0; --i ) for( int i = num_workers - 1; i >= 0; --i )
{ {
const int errcode = pthread_join( worker_threads[i], 0 ); const int errcode = pthread_join( worker_threads[i], 0 );
if( errcode ) if( errcode )
{ show_error( "Can't join worker threads", errcode ); fatal(); } { show_error( "Can't join worker threads", errcode ); cleanup_and_fail(); }
} }
delete[] worker_threads; delete[] worker_threads;
delete[] worker_args; delete[] worker_args;

4
doc/plzip.1
View file

@ -1,12 +1,12 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1. .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1.
.TH PLZIP "1" "May 2013" "Plzip 1.0" "User Commands" .TH PLZIP "1" "July 2013" "Plzip 1.1-pre1" "User Commands"
.SH NAME .SH NAME
Plzip \- reduces the size of files Plzip \- reduces the size of files
.SH SYNOPSIS .SH SYNOPSIS
.B plzip .B plzip
[\fIoptions\fR] [\fIfiles\fR] [\fIoptions\fR] [\fIfiles\fR]
.SH DESCRIPTION .SH DESCRIPTION
Plzip \- A parallel compressor compatible with lzip. Plzip \- Parallel compressor compatible with lzip.
.SH OPTIONS .SH OPTIONS
.TP .TP
\fB\-h\fR, \fB\-\-help\fR \fB\-h\fR, \fB\-\-help\fR

105
doc/plzip.info
View file

@ -12,16 +12,16 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir)
Plzip Manual Plzip Manual
************ ************
This manual is for Plzip (version 1.0, 29 May 2013). This manual is for Plzip (version 1.1-pre1, 20 July 2013).
* Menu: * Menu:
* Introduction:: Purpose and features of plzip * Introduction:: Purpose and features of plzip
* Program Design:: Internal structure of plzip * Program design:: Internal structure of plzip
* Invoking Plzip:: Command line interface * Invoking plzip:: Command line interface
* File Format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
* Problems:: Reporting bugs * Problems:: Reporting bugs
* Concept Index:: Index of concepts * Concept index:: Index of concepts
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -30,27 +30,46 @@ This manual is for Plzip (version 1.0, 29 May 2013).
copy, distribute and modify it. copy, distribute and modify it.
 
File: plzip.info, Node: Introduction, Next: Program Design, Prev: Top, Up: Top File: plzip.info, Node: Introduction, Next: Program design, Prev: Top, Up: Top
1 Introduction 1 Introduction
************** **************
Plzip is a massively parallel (multi-threaded), lossless data compressor Plzip is a massively parallel (multi-threaded), lossless data compressor
based on the lzlib compression library, with very safe integrity based on the lzlib compression library, with a user interface similar to
checking and a user interface similar to the one of bzip2, gzip or lzip. the one of lzip, bzip2 or gzip.
Plzip is intended for faster compression/decompression of big files Plzip can compress/decompress large files on multiprocessor machines
on multiprocessor machines, which makes it specially well suited for much faster than lzip, at the cost of a slightly reduced compression
distribution of big software files and large scale data archiving. On ratio. On files large enough (several GB), plzip can use hundreds of
files big enough (several GB), plzip can use hundreds of processors. processors. On files of only a few MB it is better to use lzip.
Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer when used in pipes or scripts than
compressors returning ambiguous warning values, like gzip.
Plzip uses the lzip file format; the files produced by plzip are Plzip uses the lzip file format; the files produced by plzip are
fully compatible with lzip-1.4 or newer, and can be rescued with fully compatible with lzip-1.4 or newer, and can be rescued with
lziprecover. lziprecover.
Plzip uses the same well-defined exit status values used by lzip and The lzip file format is designed for long-term data archiving and
bzip2, which makes it safer when used in pipes or scripts than provides very safe integrity checking. The member trailer stores the
compressors returning ambiguous warning values, like gzip. 32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
Plzip replaces every file given in the command line with a compressed Plzip replaces every file given in the command line with a compressed
version of itself, with the name "original_name.lz". Each compressed version of itself, with the name "original_name.lz". Each compressed
@ -76,18 +95,6 @@ filename.lz becomes filename
filename.tlz becomes filename.tar filename.tlz becomes filename.tar
anyothername becomes anyothername.out anyothername becomes anyothername.out
As a self-check for your protection, plzip stores in the member
trailer the 32-bit CRC of the original data, the size of the original
data and the size of the member. These values, together with the value
remaining in the range decoder and the end-of-stream marker, provide a
very safe 4 factor integrity checking which guarantees that the
decompressed version of the data is identical to the original. This
guards against corruption of the compressed data, and against
undetected bugs in plzip (hopefully very unlikely). The chances of data
corruption going undetected are microscopic. Be aware, though, that the
check occurs upon decompression, so it can only tell you that something
is wrong. It can't help you recover the original uncompressed data.
WARNING! Even if plzip is bug-free, other causes may result in a WARNING! Even if plzip is bug-free, other causes may result in a
corrupt compressed file (bugs in the system libraries, memory errors, corrupt compressed file (bugs in the system libraries, memory errors,
etc). Therefore, if the data you are going to compress is important, etc). Therefore, if the data you are going to compress is important,
@ -96,9 +103,9 @@ until you verify the compressed file with a command like
`plzip -cd file.lz | cmp file -'. `plzip -cd file.lz | cmp file -'.
 
File: plzip.info, Node: Program Design, Next: Invoking Plzip, Prev: Introduction, Up: Top File: plzip.info, Node: Program design, Next: Invoking plzip, Prev: Introduction, Up: Top
2 Program Design 2 Program design
**************** ****************
For each input file, a splitter thread and several worker threads are For each input file, a splitter thread and several worker threads are
@ -119,9 +126,9 @@ speed of large files with many members is only limited by the number of
processors available and by I/O speed. processors available and by I/O speed.
 
File: plzip.info, Node: Invoking Plzip, Next: File Format, Prev: Program Design, Up: Top File: plzip.info, Node: Invoking plzip, Next: File format, Prev: Program design, Up: Top
3 Invoking Plzip 3 Invoking plzip
**************** ****************
The format for running plzip is: The format for running plzip is:
@ -220,7 +227,7 @@ The format for running plzip is:
`--verbose' `--verbose'
Verbose mode. Verbose mode.
When compressing, show the compression ratio for each file When compressing, show the compression ratio for each file
processed. processed. A second -v shows the progress of compression.
When decompressing or testing, further -v's (up to 4) increase the When decompressing or testing, further -v's (up to 4) increase the
verbosity level, showing status, compression ratio, decompressed verbosity level, showing status, compression ratio, decompressed
size, and compressed size. size, and compressed size.
@ -275,9 +282,9 @@ invalid input file, 3 for an internal consistency error (eg, bug) which
caused plzip to panic. caused plzip to panic.
 
File: plzip.info, Node: File Format, Next: Problems, Prev: Invoking Plzip, Up: Top File: plzip.info, Node: File format, Next: Problems, Prev: Invoking plzip, Up: Top
4 File Format 4 File format
************* *************
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -348,7 +355,7 @@ additional information before, between, or after them.
 
File: plzip.info, Node: Problems, Next: Concept Index, Prev: File Format, Up: Top File: plzip.info, Node: Problems, Next: Concept index, Prev: File format, Up: Top
5 Reporting Bugs 5 Reporting Bugs
**************** ****************
@ -363,34 +370,34 @@ for all eternity, if not longer.
by running `plzip --version'. by running `plzip --version'.
 
File: plzip.info, Node: Concept Index, Prev: Problems, Up: Top File: plzip.info, Node: Concept index, Prev: Problems, Up: Top
Concept Index Concept index
************* *************
[index] [index]
* Menu: * Menu:
* bugs: Problems. (line 6) * bugs: Problems. (line 6)
* file format: File Format. (line 6) * file format: File format. (line 6)
* getting help: Problems. (line 6) * getting help: Problems. (line 6)
* introduction: Introduction. (line 6) * introduction: Introduction. (line 6)
* invoking: Invoking Plzip. (line 6) * invoking: Invoking plzip. (line 6)
* options: Invoking Plzip. (line 6) * options: Invoking plzip. (line 6)
* program design: Program Design. (line 6) * program design: Program design. (line 6)
* usage: Invoking Plzip. (line 6) * usage: Invoking plzip. (line 6)
* version: Invoking Plzip. (line 6) * version: Invoking plzip. (line 6)
 
Tag Table: Tag Table:
Node: Top223 Node: Top223
Node: Introduction865 Node: Introduction871
Node: Program Design4113 Node: Program design4426
Node: Invoking Plzip5167 Node: Invoking plzip5480
Node: File Format10416 Node: File format10776
Node: Problems12895 Node: Problems13255
Node: Concept Index13424 Node: Concept index13784
 
End Tag Table End Tag Table

80
doc/plzip.texinfo
View file

@ -6,8 +6,8 @@
@finalout @finalout
@c %**end of header @c %**end of header
@set UPDATED 29 May 2013 @set UPDATED 20 July 2013
@set VERSION 1.0 @set VERSION 1.1-pre1
@dircategory Data Compression @dircategory Data Compression
@direntry @direntry
@ -36,11 +36,11 @@ This manual is for Plzip (version @value{VERSION}, @value{UPDATED}).
@menu @menu
* Introduction:: Purpose and features of plzip * Introduction:: Purpose and features of plzip
* Program Design:: Internal structure of plzip * Program design:: Internal structure of plzip
* Invoking Plzip:: Command line interface * Invoking plzip:: Command line interface
* File Format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
* Problems:: Reporting bugs * Problems:: Reporting bugs
* Concept Index:: Index of concepts * Concept index:: Index of concepts
@end menu @end menu
@sp 1 @sp 1
@ -55,21 +55,40 @@ to copy, distribute and modify it.
@cindex introduction @cindex introduction
Plzip is a massively parallel (multi-threaded), lossless data compressor Plzip is a massively parallel (multi-threaded), lossless data compressor
based on the lzlib compression library, with very safe integrity based on the lzlib compression library, with a user interface similar to
checking and a user interface similar to the one of bzip2, gzip or lzip. the one of lzip, bzip2 or gzip.
Plzip is intended for faster compression/decompression of big files on Plzip can compress/decompress large files on multiprocessor machines
multiprocessor machines, which makes it specially well suited for much faster than lzip, at the cost of a slightly reduced compression
distribution of big software files and large scale data archiving. On ratio. On files large enough (several GB), plzip can use hundreds of
files big enough (several GB), plzip can use hundreds of processors. processors. On files of only a few MB it is better to use lzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
Plzip uses the same well-defined exit status values used by lzip and Plzip uses the same well-defined exit status values used by lzip and
bzip2, which makes it safer when used in pipes or scripts than bzip2, which makes it safer when used in pipes or scripts than
compressors returning ambiguous warning values, like gzip. compressors returning ambiguous warning values, like gzip.
Plzip uses the lzip file format; the files produced by plzip are fully
compatible with lzip-1.4 or newer, and can be rescued with lziprecover.
The lzip file format is designed for long-term data archiving and
provides very safe integrity checking. The member trailer stores the
32-bit CRC of the original data, the size of the original data and the
size of the member. These values, together with the value remaining in
the range decoder and the end-of-stream marker, provide a 4 factor
integrity checking which guarantees that the decompressed version of the
data is identical to the original. This guards against corruption of the
compressed data, and against undetected bugs in plzip (hopefully very
unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
If you ever need to recover data from a damaged lzip file, try the
lziprecover program. Lziprecover makes lzip files resistant to bit-flip
(one of the most common forms of data corruption), and provides data
recovery capabilities, including error-checked merging of damaged copies
of a file.
Plzip replaces every file given in the command line with a compressed Plzip replaces every file given in the command line with a compressed
version of itself, with the name "original_name.lz". Each compressed version of itself, with the name "original_name.lz". Each compressed
file has the same modification date, permissions, and, when possible, file has the same modification date, permissions, and, when possible,
@ -96,18 +115,6 @@ file from that of the compressed file as follows:
@item anyothername @tab becomes @tab anyothername.out @item anyothername @tab becomes @tab anyothername.out
@end multitable @end multitable
As a self-check for your protection, plzip stores in the member trailer
the 32-bit CRC of the original data, the size of the original data and
the size of the member. These values, together with the value remaining
in the range decoder and the end-of-stream marker, provide a very safe 4
factor integrity checking which guarantees that the decompressed version
of the data is identical to the original. This guards against corruption
of the compressed data, and against undetected bugs in plzip (hopefully
very unlikely). The chances of data corruption going undetected are
microscopic. Be aware, though, that the check occurs upon decompression,
so it can only tell you that something is wrong. It can't help you
recover the original uncompressed data.
WARNING! Even if plzip is bug-free, other causes may result in a corrupt WARNING! Even if plzip is bug-free, other causes may result in a corrupt
compressed file (bugs in the system libraries, memory errors, etc). compressed file (bugs in the system libraries, memory errors, etc).
Therefore, if the data you are going to compress is important, give the Therefore, if the data you are going to compress is important, give the
@ -116,8 +123,8 @@ you verify the compressed file with a command like
@w{@samp{plzip -cd file.lz | cmp file -}}. @w{@samp{plzip -cd file.lz | cmp file -}}.
@node Program Design @node Program design
@chapter Program Design @chapter Program design
@cindex program design @cindex program design
For each input file, a splitter thread and several worker threads are For each input file, a splitter thread and several worker threads are
@ -138,8 +145,8 @@ large files with many members is only limited by the number of
processors available and by I/O speed. processors available and by I/O speed.
@node Invoking Plzip @node Invoking plzip
@chapter Invoking Plzip @chapter Invoking plzip
@cindex invoking @cindex invoking
@cindex options @cindex options
@cindex usage @cindex usage
@ -237,7 +244,8 @@ Use it together with @samp{-v} to see information about the file.
@item -v @item -v
@itemx --verbose @itemx --verbose
Verbose mode.@* Verbose mode.@*
When compressing, show the compression ratio for each file processed.@* When compressing, show the compression ratio for each file processed. A
second -v shows the progress of compression.@*
When decompressing or testing, further -v's (up to 4) increase the When decompressing or testing, further -v's (up to 4) increase the
verbosity level, showing status, compression ratio, decompressed size, verbosity level, showing status, compression ratio, decompressed size,
and compressed size. and compressed size.
@ -297,8 +305,8 @@ invalid input file, 3 for an internal consistency error (eg, bug) which
caused plzip to panic. caused plzip to panic.
@node File Format @node File format
@chapter File Format @chapter File format
@cindex file format @cindex file format
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -387,8 +395,8 @@ If you find a bug in plzip, please send electronic mail to
find by running @w{@samp{plzip --version}}. find by running @w{@samp{plzip --version}}.
@node Concept Index @node Concept index
@unnumbered Concept Index @unnumbered Concept index
@printindex cp @printindex cp

20
file_index.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
@ -68,23 +68,23 @@ File_index::File_index( const int infd ) : retval_( 0 )
{ error_ = "Input file is not seekable :"; { error_ = "Input file is not seekable :";
error_ += std::strerror( errno ); retval_ = 1; return; } error_ += std::strerror( errno ); retval_ = 1; return; }
if( isize > INT64_MAX ) if( isize > INT64_MAX )
{ error_ = "Input file is too long (2^63 bytes or more)."; { error_ = "Input file is too long (2^63 bytes or more)";
retval_ = 2; return; } retval_ = 2; return; }
long long pos = isize; // always points to a header or EOF long long pos = isize; // always points to a header or EOF
File_header header; File_header header;
File_trailer trailer; File_trailer trailer;
if( isize < min_member_size ) if( isize < min_member_size )
{ error_ = "Input file is too short."; retval_ = 2; return; } { error_ = "Input file is too short"; retval_ = 2; return; }
if( seek_read( infd, header.data, File_header::size, 0 ) != File_header::size ) if( seek_read( infd, header.data, File_header::size, 0 ) != File_header::size )
{ error_ = "Error reading member header :"; { error_ = "Error reading member header :";
error_ += std::strerror( errno ); retval_ = 1; return; } error_ += std::strerror( errno ); retval_ = 1; return; }
if( !header.verify_magic() ) if( !header.verify_magic() )
{ error_ = "Bad magic number (file not in lzip format)."; { error_ = "Bad magic number (file not in lzip format)";
retval_ = 2; return; } retval_ = 2; return; }
if( !header.verify_version() ) if( !header.verify_version() )
{ error_ = "Version "; error_ += format_num( header.version() ); { error_ = "Version "; error_ += format_num( header.version() );
error_ += "member format not supported."; retval_ = 2; return; } error_ += "member format not supported"; retval_ = 2; return; }
while( pos >= min_member_size ) while( pos >= min_member_size )
{ {
@ -114,9 +114,9 @@ File_index::File_index( const int infd ) : retval_( 0 )
if( member_vector.size() == 0 && isize - pos > File_header::size && if( member_vector.size() == 0 && isize - pos > File_header::size &&
seek_read( infd, header.data, File_header::size, pos ) == File_header::size && seek_read( infd, header.data, File_header::size, pos ) == File_header::size &&
header.verify_magic() && header.verify_version() ) header.verify_magic() && header.verify_version() )
{ // last trailer is corrupt {
error_ = "Member size in trailer is corrupt at pos "; error_ = "Last member in input file is truncated or corrupt";
error_ += format_num( isize - 8 ); retval_ = 2; break; retval_ = 2; break;
} }
pos -= member_size; pos -= member_size;
member_vector.push_back( Member( 0, trailer.data_size(), member_vector.push_back( Member( 0, trailer.data_size(),
@ -125,7 +125,7 @@ File_index::File_index( const int infd ) : retval_( 0 )
if( pos != 0 || member_vector.size() == 0 ) if( pos != 0 || member_vector.size() == 0 )
{ {
member_vector.clear(); member_vector.clear();
if( retval_ == 0 ) { error_ = "Can't create file index."; retval_ = 2; } if( retval_ == 0 ) { error_ = "Can't create file index"; retval_ = 2; }
return; return;
} }
std::reverse( member_vector.begin(), member_vector.end() ); std::reverse( member_vector.begin(), member_vector.end() );
@ -135,7 +135,7 @@ File_index::File_index( const int infd ) : retval_( 0 )
if( end < 0 || end > INT64_MAX ) if( end < 0 || end > INT64_MAX )
{ {
member_vector.clear(); member_vector.clear();
error_ = "Data in input file is too long (2^63 bytes or more)."; error_ = "Data in input file is too long (2^63 bytes or more)";
retval_ = 2; return; retval_ = 2; return;
} }
member_vector[i+1].dblock.pos( end ); member_vector[i+1].dblock.pos( end );

2
file_index.h
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify

7
lzip.h
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
@ -196,10 +196,13 @@ int decompress( int num_workers, const int infd, const int outfd,
// defined in main.cc // defined in main.cc
extern int verbosity; extern int verbosity;
void fatal( const int retval = 1 ); // terminate the program void cleanup_and_fail( const int retval = 1 ); // terminate the program
void show_error( const char * const msg, const int errcode = 0, void show_error( const char * const msg, const int errcode = 0,
const bool help = false ); const bool help = false );
void internal_error( const char * const msg ); void internal_error( const char * const msg );
void show_progress( const int packet_size,
const Pretty_print * const p = 0,
const struct stat * const in_statsp = 0 );
class Slot_tally class Slot_tally

106
main.cc
View file

@ -1,4 +1,4 @@
/* Plzip - A parallel compressor compatible with lzip /* Plzip - Parallel compressor compatible with lzip
Copyright (C) 2009 Laszlo Ersek. Copyright (C) 2009 Laszlo Ersek.
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -96,13 +96,11 @@ const mode_t usr_rw = S_IRUSR | S_IWUSR;
const mode_t all_rw = usr_rw | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH; const mode_t all_rw = usr_rw | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH;
mode_t outfd_mode = usr_rw; mode_t outfd_mode = usr_rw;
bool delete_output_on_interrupt = false; bool delete_output_on_interrupt = false;
pthread_t main_thread;
pid_t main_thread_pid;
void show_help( const long num_online ) void show_help( const long num_online )
{ {
std::printf( "%s - A parallel compressor compatible with lzip.\n", Program_name ); std::printf( "%s - Parallel compressor compatible with lzip.\n", Program_name );
std::printf( "\nUsage: %s [options] [files]\n", invocation_name ); std::printf( "\nUsage: %s [options] [files]\n", invocation_name );
std::printf( "\nOptions:\n" std::printf( "\nOptions:\n"
" -h, --help display this help and exit\n" " -h, --help display this help and exit\n"
@ -262,12 +260,13 @@ int open_instream( const char * const name, struct stat * const in_statsp,
const bool can_read = ( i == 0 && const bool can_read = ( i == 0 &&
( S_ISBLK( mode ) || S_ISCHR( mode ) || ( S_ISBLK( mode ) || S_ISCHR( mode ) ||
S_ISFIFO( mode ) || S_ISSOCK( mode ) ) ); S_ISFIFO( mode ) || S_ISSOCK( mode ) ) );
if( i != 0 || ( !S_ISREG( mode ) && ( !to_stdout || !can_read ) ) ) const bool no_ofile = to_stdout || ( program_mode == m_test );
if( i != 0 || ( !S_ISREG( mode ) && ( !can_read || !no_ofile ) ) )
{ {
if( verbosity >= 0 ) if( verbosity >= 0 )
std::fprintf( stderr, "%s: Input file '%s' is not a regular file%s.\n", std::fprintf( stderr, "%s: Input file '%s' is not a regular file%s.\n",
program_name, name, program_name, name,
( can_read && !to_stdout ) ? ( can_read && !no_ofile ) ?
" and '--stdout' was not specified" : "" ); " and '--stdout' was not specified" : "" );
close( infd ); close( infd );
infd = -1; infd = -1;
@ -340,22 +339,6 @@ bool check_tty( const int infd, const Mode program_mode )
} }
void cleanup_and_fail( const int retval )
{
if( delete_output_on_interrupt )
{
delete_output_on_interrupt = false;
if( verbosity >= 0 )
std::fprintf( stderr, "%s: Deleting output file '%s', if it exists.\n",
program_name, output_filename.c_str() );
if( outfd >= 0 ) { close( outfd ); outfd = -1; }
if( std::remove( output_filename.c_str() ) != 0 && errno != ENOENT )
show_error( "WARNING: deletion of output file (apparently) failed." );
}
std::exit( retval );
}
// Set permissions, owner and times. // Set permissions, owner and times.
void close_and_set_permissions( const struct stat * const in_statsp ) void close_and_set_permissions( const struct stat * const in_statsp )
{ {
@ -382,13 +365,10 @@ void close_and_set_permissions( const struct stat * const in_statsp )
} }
extern "C" void signal_handler( int sig ) extern "C" void signal_handler( int )
{ {
if( !pthread_equal( pthread_self(), main_thread ) ) show_error( "Control-C or similar caught, quitting." );
kill( main_thread_pid, sig ); cleanup_and_fail( 1 );
if( sig != SIGUSR1 && sig != SIGUSR2 )
show_error( "Control-C or similar caught, quitting." );
cleanup_and_fail( ( sig != SIGUSR2 ) ? 1 : 2 );
} }
@ -405,14 +385,6 @@ void set_signals()
int verbosity = 0; int verbosity = 0;
// This can be called from any thread, main thread or sub-threads alike,
// since they all call common helper functions that call fatal() in case
// of an error.
//
void fatal( const int retval )
{ signal_handler( ( retval != 2 ) ? SIGUSR1 : SIGUSR2 ); }
void Pretty_print::operator()( const char * const msg ) const void Pretty_print::operator()( const char * const msg ) const
{ {
if( verbosity >= 0 ) if( verbosity >= 0 )
@ -456,6 +428,60 @@ void internal_error( const char * const msg )
} }
// This can be called from any thread, main thread or sub-threads alike,
// since they all call common helper functions that call cleanup_and_fail()
// in case of an error.
//
void cleanup_and_fail( const int retval )
{
// only one thread can delete and exit
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_lock( &mutex ); // ignore errors to avoid loop
if( delete_output_on_interrupt )
{
delete_output_on_interrupt = false;
if( verbosity >= 0 )
std::fprintf( stderr, "%s: Deleting output file '%s', if it exists.\n",
program_name, output_filename.c_str() );
if( outfd >= 0 ) { close( outfd ); outfd = -1; }
if( std::remove( output_filename.c_str() ) != 0 && errno != ENOENT )
show_error( "WARNING: deletion of output file (apparently) failed." );
}
std::exit( retval );
}
void show_progress( const int packet_size,
const Pretty_print * const p,
const struct stat * const in_statsp )
{
static unsigned long long cfile_size = 0; // file_size / 100
static unsigned long long pos = 0;
static const Pretty_print * pp = 0;
static pthread_mutex_t mutex;
if( p ) // initialize static vars
{
if( !pp ) xinit( &mutex ); // init mutex only once
pos = 0; pp = p;
cfile_size = ( in_statsp && S_ISREG( in_statsp->st_mode ) ) ?
in_statsp->st_size / 100 : 0;
return;
}
if( pp )
{
xlock( &mutex );
pos += packet_size;
if( cfile_size > 0 )
std::fprintf( stderr, "%4llu%%", pos / cfile_size );
std::fprintf( stderr, " %.1f MB\r", pos / 1000000.0 );
pp->reset(); (*pp)(); // restore cursor position
xunlock( &mutex );
}
}
int main( const int argc, const char * const argv[] ) int main( const int argc, const char * const argv[] )
{ {
// Mapping from gzip/bzip2 style 1..9 compression modes // Mapping from gzip/bzip2 style 1..9 compression modes
@ -486,8 +512,6 @@ int main( const int argc, const char * const argv[] )
bool recompress = false; bool recompress = false;
bool to_stdout = false; bool to_stdout = false;
invocation_name = argv[0]; invocation_name = argv[0];
main_thread = pthread_self();
main_thread_pid = getpid();
if( LZ_version()[0] != LZ_version_string[0] ) if( LZ_version()[0] != LZ_version_string[0] )
internal_error( "bad library version" ); internal_error( "bad library version" );
@ -598,8 +622,6 @@ int main( const int argc, const char * const argv[] )
if( !to_stdout && program_mode != m_test && if( !to_stdout && program_mode != m_test &&
( filenames_given || default_output_filename.size() ) ) ( filenames_given || default_output_filename.size() ) )
set_signals(); set_signals();
std::signal( SIGUSR1, signal_handler );
std::signal( SIGUSR2, signal_handler );
Pretty_print pp( filenames ); Pretty_print pp( filenames );
@ -668,9 +690,13 @@ int main( const int argc, const char * const argv[] )
if( verbosity >= 1 ) pp(); if( verbosity >= 1 ) pp();
int tmp; int tmp;
if( program_mode == m_compress ) if( program_mode == m_compress )
{
show_progress( 0, &pp, in_statsp ); // initialize static vars
if( verbosity >= 2 ) show_progress( 0 ); // show initial zero size
tmp = compress( data_size, encoder_options.dictionary_size, tmp = compress( data_size, encoder_options.dictionary_size,
encoder_options.match_len_limit, encoder_options.match_len_limit,
num_workers, infd, outfd, pp, debug_level ); num_workers, infd, outfd, pp, debug_level );
}
else else
tmp = decompress( num_workers, infd, outfd, pp, debug_level, tmp = decompress( num_workers, infd, outfd, pp, debug_level,
program_mode == m_test, infd_isreg ); program_mode == m_test, infd_isreg );

45
testsuite/check.sh
View file

@ -1,5 +1,5 @@
#! /bin/sh #! /bin/sh
# check script for Plzip - A parallel compressor compatible with lzip # check script for Plzip - Parallel compressor compatible with lzip
# Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. # Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
# #
# This script is free software: you have unlimited permission # This script is free software: you have unlimited permission
@ -28,13 +28,21 @@ fail=0
printf "testing plzip-%s..." "$2" printf "testing plzip-%s..." "$2"
"${LZIP}" -cqs-1 in > /dev/null "${LZIP}" -cqs-1 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs0 in > /dev/null "${LZIP}" -cqs0 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs4095 in > /dev/null "${LZIP}" -cqs4095 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqm274 in > /dev/null "${LZIP}" -cqm274 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -t "${in_lz}" || fail=1 "${LZIP}" -t "${in_lz}" || fail=1
"${LZIP}" -cd "${in_lz}" > copy || fail=1 "${LZIP}" -cd "${in_lz}" > copy || fail=1
@ -42,7 +50,7 @@ cmp in copy || fail=1
printf . printf .
"${LZIP}" -cfq "${in_lz}" > out "${LZIP}" -cfq "${in_lz}" > out
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cF "${in_lz}" > out || fail=1 "${LZIP}" -cF "${in_lz}" > out || fail=1
"${LZIP}" -cd out | "${LZIP}" -d > copy || fail=1 "${LZIP}" -cd out | "${LZIP}" -d > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
@ -54,32 +62,32 @@ for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
printf "garbage" >> copy.lz || fail=1 printf "garbage" >> copy.lz || fail=1
"${LZIP}" -df copy.lz || fail=1 "${LZIP}" -df copy.lz || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
"${LZIP}" -c -$i in > out || fail=1 "${LZIP}" -c -$i in > out || fail=1
printf "g" >> out || fail=1 printf "g" >> out || fail=1
"${LZIP}" -cd out > copy || fail=1 "${LZIP}" -cd out > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
"${LZIP}" -$i < in > out || fail=1 "${LZIP}" -$i < in > out || fail=1
printf "garbage" >> out || fail=1 printf "garbage" >> out || fail=1
"${LZIP}" -d < out > copy || fail=1 "${LZIP}" -d < out > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
"${LZIP}" -f -$i -o out < in || fail=1 "${LZIP}" -f -$i -o out < in || fail=1
printf "g" >> out.lz || fail=1 printf "g" >> out.lz || fail=1
"${LZIP}" -df -o copy < out.lz || fail=1 "${LZIP}" -df -o copy < out.lz || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
"${LZIP}" < in > anyothername || fail=1 "${LZIP}" < in > anyothername || fail=1
"${LZIP}" -d anyothername || fail=1 "${LZIP}" -d anyothername || fail=1
@ -95,39 +103,38 @@ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ; do
"${LZIP}" -d -n$i out4.lz || fail=1 "${LZIP}" -d -n$i out4.lz || fail=1
cmp in4 out4 || fail=1 cmp in4 out4 || fail=1
rm -f out4 rm -f out4
printf .
done done
printf .
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ; do for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ; do
"${LZIP}" -s4Ki -B8Ki -n$i < in4 > out4 || fail=1 "${LZIP}" -s4Ki -B8Ki -n$i < in4 > out4 || fail=1
printf "g" >> out4 || fail=1 printf "g" >> out4 || fail=1
"${LZIP}" -d -n$i < out4 > copy4 || fail=1 "${LZIP}" -d -n$i < out4 > copy4 || fail=1
cmp in4 copy4 || fail=1 cmp in4 copy4 || fail=1
printf .
done done
printf .
cat "${in_lz}" > ingin.lz || framework_failure cat "${in_lz}" > ingin.lz || framework_failure
printf "g" >> ingin.lz || framework_failure printf "g" >> ingin.lz || framework_failure
cat "${in_lz}" >> ingin.lz || framework_failure cat "${in_lz}" >> ingin.lz || framework_failure
"${LZIP}" -tq ingin.lz "${LZIP}" -tq ingin.lz
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq ingin.lz > out "${LZIP}" -cdq ingin.lz > out
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -t < ingin.lz || fail=1 "${LZIP}" -t < ingin.lz || fail=1
printf .
"${LZIP}" -d < ingin.lz > copy || fail=1 "${LZIP}" -d < ingin.lz > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf . printf .
dd if="${in_lz}" bs=1024 count=10 > trunc.lz 2> /dev/null || framework_failure dd if="${in_lz}" bs=1024 count=10 > trunc.lz 2> /dev/null || framework_failure
"${LZIP}" -tq trunc.lz "${LZIP}" -tq trunc.lz
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq trunc.lz > out "${LZIP}" -cdq trunc.lz > out
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq < trunc.lz "${LZIP}" -tq < trunc.lz
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -dq < trunc.lz > out "${LZIP}" -dq < trunc.lz > out
if [ $? != 2 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
echo echo
if [ ${fail} = 0 ] ; then if [ ${fail} = 0 ] ; then