lzip/lziprecover
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.25~rc1.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-21 11:33:25 +01:00
parent 1d67e88e3c
commit b8e73cb85f
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
39 changed files with 978 additions and 742 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

51
ChangeLog
View file

@ -1,3 +1,15 @@
2024-11-18 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.25-rc1 released.
* byte_repair.cc: Repair a nonzero first LZMA byte.
* Integrate options '--ignore-empty' and '--ignore-nonzero' into
'-i, --ignore-errors'.
* merge.cc (copy_file): Add name arguments, use 'show_file_error'.
* lziprecover.texi: New chapter 'Syntax of command-line arguments'.
* check.sh: Use 'cp' instead of 'cat'.
* testsuite: Add fox_nz.lz, fox6_b1nz.lz.
Remove fox6.lz, fox6_nz.lz, test_em.txt.lz, test_3m.txt.lz.md5.
2024-10-01 Antonio Diaz Diaz <antonio@gnu.org> 2024-10-01 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.25-pre1 released. * Version 1.25-pre1 released.
@ -7,7 +19,7 @@
* New options '--ignore-empty' and '--ignore-nonzero'. * New options '--ignore-empty' and '--ignore-nonzero'.
* Rename option '--clear-marking' to '--nonzero-repair'. * Rename option '--clear-marking' to '--nonzero-repair'.
* Remove options '--empty-error' and '--marking-error'. * Remove options '--empty-error' and '--marking-error'.
* Remove decompression support for Sync Flush marker. * decoder.cc (decode_member): Remove support for Sync Flush marker.
* testsuite: Require lzip/clzip. Add fox6_nz.lz. Remove fox6_mark.lz. * testsuite: Require lzip/clzip. Add fox6_nz.lz. Remove fox6_mark.lz.
2024-01-20 Antonio Diaz Diaz <antonio@gnu.org> 2024-01-20 Antonio Diaz Diaz <antonio@gnu.org>
@ -134,11 +146,10 @@
* repair.cc: Repair a damaged dictionary size in the header. * repair.cc: Repair a damaged dictionary size in the header.
* repair.cc: Try bytes at offsets 7 to 11 first. * repair.cc: Try bytes at offsets 7 to 11 first.
* Decompression time has been reduced by 2%. * Decompression time has been reduced by 2%.
* main.cc (decompress): Print up to 6 bytes of trailing data when
'-tvvvv' is specified.
* decoder.cc (verify_trailer): Remove test of final code.
* main.cc (main): Delete '--output' file if infd is a terminal. * main.cc (main): Delete '--output' file if infd is a terminal.
* main.cc (main): Don't use stdin more than once. (main): Don't use stdin more than once.
(decompress): Print 6 bytes of trailing data at verbosity level 4.
* decoder.cc (verify_trailer): Remove test of final code.
* Use 'close_and_set_permissions' and 'set_signals' in all modes. * Use 'close_and_set_permissions' and 'set_signals' in all modes.
* range_dec.cc (list_file): Show dictionary size and size of * range_dec.cc (list_file): Show dictionary size and size of
trailing data (if any) with '-lv'. trailing data (if any) with '-lv'.
@ -154,8 +165,7 @@
* lziprecover.texi: New chapter 'Trailing data'. * lziprecover.texi: New chapter 'Trailing data'.
* configure: Avoid warning on some shells when testing for g++. * configure: Avoid warning on some shells when testing for g++.
* Makefile.in: Detect the existence of install-info. * Makefile.in: Detect the existence of install-info.
* check.sh: Don't check error messages. * check.sh: Require a POSIX shell. Don't check error messages.
* check.sh: A POSIX shell is required to run the tests.
2015-05-28 Antonio Diaz Diaz <antonio@gnu.org> 2015-05-28 Antonio Diaz Diaz <antonio@gnu.org>
@ -192,11 +202,11 @@
* Option '-l, --list' now accepts more than one file. * Option '-l, --list' now accepts more than one file.
* Decompression time has been reduced by 12%. * Decompression time has been reduced by 12%.
* split.cc: Use as few digits as possible in file names. * split.cc: Use as few digits as possible in file names.
* split.cc: In verbose mode show names of files being created. In verbose mode show names of files being created.
* main.cc (show_header): Show header version if verbosity >= 4. * main.cc (show_header): Show header version if verbosity >= 4.
(main): Use 'setmode' instead of '_setmode' on Windows and OS/2.
* configure: Options now accept a separate argument. * configure: Options now accept a separate argument.
* Makefile.in: New targets 'install-as-lzip' and 'install-bin'. * Makefile.in: New targets 'install-as-lzip' and 'install-bin'.
* main.cc: Use 'setmode' instead of '_setmode' on Windows and OS/2.
2012-02-24 Antonio Diaz Diaz <ant_diaz@teleline.es> 2012-02-24 Antonio Diaz Diaz <ant_diaz@teleline.es>
@ -207,8 +217,7 @@
* lziprecover.cc: Rename to main.cc. * lziprecover.cc: Rename to main.cc.
* New files merge.cc, repair.cc, split.cc, and range_dec.cc. * New files merge.cc, repair.cc, split.cc, and range_dec.cc.
* main.cc: Add decompressor options (-c, -d, -k, -t) so that an * main.cc: Add decompressor options (-c, -d, -k, -t) so that an
external decompressor is not needed for recovery nor for external decompressor is not needed for recovery and 'make check'.
"make check".
* New option '-D, --range-decompress', which extracts a range of * New option '-D, --range-decompress', which extracts a range of
bytes decompressing only the members containing the desired data. bytes decompressing only the members containing the desired data.
* New option '-l, --list', which prints correct total file sizes * New option '-l, --list', which prints correct total file sizes
@ -223,25 +232,23 @@
* Version 1.12 released. * Version 1.12 released.
* lziprecover.cc: If '-v' is not specified show errors only. * lziprecover.cc: If '-v' is not specified show errors only.
* unzcrash.cc: Use Arg_parser. * unzcrash.cc: Use Arg_parser.
* unzcrash.cc: New options '-b, --bits', '-p, --position', and New options '-b, --bits', '-p, --position', and '-s, --size'.
'-s, --size'.
2010-09-16 Antonio Diaz Diaz <ant_diaz@teleline.es> 2010-09-16 Antonio Diaz Diaz <ant_diaz@teleline.es>
* Version 1.11 released. * Version 1.11 released.
* lziprecover.cc: New option '-m, --merge', which tries to produce a
correct file by merging the good parts of two or more damaged copies.
* lziprecover.cc: New option '-R, --repair' for repairing a
1-byte error in single-member files.
* decoder.cc (decode_member): Detect file errors earlier to improve * decoder.cc (decode_member): Detect file errors earlier to improve
efficiency of lziprecover's new repair capability. efficiency of lziprecover's new repair capability.
This change also prevents (harmless) access to uninitialized This change also prevents (harmless) access to uninitialized
memory when decompressing a corrupt file. memory when decompressing a corrupt file.
* lziprecover.cc: New options '-f, --force' and '-o, --output'. * lziprecover.cc: New option '-m, --merge', which tries to produce a
* lziprecover.cc: New option '-s, --split' to select the until now correct file by merging the good parts of two or more damaged copies.
only operation of splitting multimember files. New option '-R, --repair' for repairing a 1-byte error in
* lziprecover.cc: If no operation is specified, warn the user and do single-member files.
nothing. New options '-f, --force' and '-o, --output'.
New option '-s, --split' to select the until now only operation of
splitting multimember files.
If no operation is specified, warn the user and do nothing.
2009-06-22 Antonio Diaz Diaz <ant_diaz@teleline.es> 2009-06-22 Antonio Diaz Diaz <ant_diaz@teleline.es>

7
Makefile.in
View file

@ -2,8 +2,8 @@
DISTNAME = $(pkgname)-$(pkgversion) DISTNAME = $(pkgname)-$(pkgversion)
INSTALL = install INSTALL = install
INSTALL_PROGRAM = $(INSTALL) -m 755 INSTALL_PROGRAM = $(INSTALL) -m 755
INSTALL_DATA = $(INSTALL) -m 644
INSTALL_DIR = $(INSTALL) -d -m 755 INSTALL_DIR = $(INSTALL) -d -m 755
INSTALL_DATA = $(INSTALL) -m 644
SHELL = /bin/sh SHELL = /bin/sh
CAN_RUN_INSTALLINFO = $(SHELL) -c "install-info --version" > /dev/null 2>&1 CAN_RUN_INSTALLINFO = $(SHELL) -c "install-info --version" > /dev/null 2>&1
@ -150,11 +150,9 @@ dist : doc
$(DISTNAME)/testsuite/test.txt \ $(DISTNAME)/testsuite/test.txt \
$(DISTNAME)/testsuite/test21636.txt \ $(DISTNAME)/testsuite/test21636.txt \
$(DISTNAME)/testsuite/test_bad[6-9].txt \ $(DISTNAME)/testsuite/test_bad[6-9].txt \
$(DISTNAME)/testsuite/test_3m.txt.lz.md5 \
$(DISTNAME)/testsuite/fox.lz \ $(DISTNAME)/testsuite/fox.lz \
$(DISTNAME)/testsuite/fox_*.lz \ $(DISTNAME)/testsuite/fox_*.lz \
$(DISTNAME)/testsuite/fox6.lz \ $(DISTNAME)/testsuite/fox6_b1nz.lz \
$(DISTNAME)/testsuite/fox6_nz.lz \
$(DISTNAME)/testsuite/fox6_sc[1-6].lz \ $(DISTNAME)/testsuite/fox6_sc[1-6].lz \
$(DISTNAME)/testsuite/fox6_bad[1-6].lz \ $(DISTNAME)/testsuite/fox6_bad[1-6].lz \
$(DISTNAME)/testsuite/numbers.lz \ $(DISTNAME)/testsuite/numbers.lz \
@ -162,7 +160,6 @@ dist : doc
$(DISTNAME)/testsuite/test.txt.lz \ $(DISTNAME)/testsuite/test.txt.lz \
$(DISTNAME)/testsuite/test.txt.lzma \ $(DISTNAME)/testsuite/test.txt.lzma \
$(DISTNAME)/testsuite/test_bad[1-9].lz \ $(DISTNAME)/testsuite/test_bad[1-9].lz \
$(DISTNAME)/testsuite/test_em.txt.lz \
$(DISTNAME)/testsuite/test.txt.lz.fec \ $(DISTNAME)/testsuite/test.txt.lz.fec \
$(DISTNAME)/testsuite/test.txt.lz.fec16 $(DISTNAME)/testsuite/test.txt.lz.fec16
rm -f $(DISTNAME) rm -f $(DISTNAME)

19
NEWS
View file

@ -12,20 +12,21 @@ The option '--fec-file', which sets the fec file to be used, has been added.
The options '-r, --recursive' and '-R, --dereference-recursive' have been The options '-r, --recursive' and '-R, --dereference-recursive' have been
added for recursive creation and reading of fec files. added for recursive creation and reading of fec files.
The short name of option '--byte-repair' has been changed to "-B". The short name of option '--byte-repair' has been changed to '-B'.
The option '--ignore-empty', which makes lziprecover ignore empty members in The option '--byte-repair' now repairs a nonzero first LZMA byte.
multimember files when decompressing, testing, or listing, has been added.
By default lziprecover now exits with error status 2 if any empty member is
found in a multimember file.
The option '--ignore-nonzero', which makes lziprecover ignore a nonzero When decompressing, testing, or listing, lziprecover now exits with error
first byte in the LZMA stream when decompressing or testing, has been added. status 2 if any empty member is found in a regular multimember file unless
By default lziprecover now exits with error status 2 if the first LZMA byte '-i' is given.
is nonzero in any member of the input files.
When decompressing or testing, lziprecover now exits with error status 2 if
the first byte of the LZMA stream is not 0 unless '-i' is given.
The option '--clear-marking' has been renamed to '--nonzero-repair'. The option '--clear-marking' has been renamed to '--nonzero-repair'.
Options '--empty-error' and '--marking-error' have been removed. Options '--empty-error' and '--marking-error' have been removed.
The chapter 'Syntax of command-line arguments' has been added to the manual.
Lzip 1.16 (or clzip 1.6) or newer is required to run the tests. Lzip 1.16 (or clzip 1.6) or newer is required to run the tests.

6
alone_to_lz.cc
View file

@ -52,7 +52,7 @@ uint8_t * read_file( const int infd, long * const file_sizep,
if( buffer_size >= LONG_MAX ) if( buffer_size >= LONG_MAX )
{ show_file_error( filename, large_file_msg ); { show_file_error( filename, large_file_msg );
std::free( buffer ); return 0; } std::free( buffer ); return 0; }
buffer_size = ( buffer_size <= LONG_MAX / 2 ) ? 2 * buffer_size : LONG_MAX; buffer_size = (buffer_size <= LONG_MAX / 2) ? 2 * buffer_size : LONG_MAX;
uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size ); uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size );
if( !tmp ) { std::free( buffer ); throw std::bad_alloc(); } if( !tmp ) { std::free( buffer ); throw std::bad_alloc(); }
buffer = tmp; buffer = tmp;
@ -143,8 +143,8 @@ int alone_to_lz( const int infd, const Pretty_print & pp )
{ pp( "conversion failed" ); std::free( buffer ); return 2; } { pp( "conversion failed" ); std::free( buffer ); return 2; }
if( writeblock( outfd, buffer + offset, lzip_size ) != lzip_size ) if( writeblock( outfd, buffer + offset, lzip_size ) != lzip_size )
{ {
show_error( "Error writing output file", errno ); show_file_error( printable_name( output_filename, false ), write_error_msg,
std::free( buffer ); return 1; errno ); std::free( buffer ); return 1;
} }
std::free( buffer ); std::free( buffer );
if( verbosity >= 1 ) std::fputs( "done\n", stderr ); if( verbosity >= 1 ) std::fputs( "done\n", stderr );

89
byte_repair.cc
View file

@ -58,6 +58,19 @@ bool gross_damage( const uint8_t * const mbuffer, const long msize )
} }
// Return value: 0 = errors remain, 6 = repaired pos
int repair_nonzero( uint8_t * const mbuffer, const long msize )
{
mbuffer[6] = 0;
const Lzip_header & header = *(Lzip_header *)mbuffer;
const unsigned dictionary_size = header.dictionary_size();
if( !isvalid_ds( dictionary_size ) ) return 0;
LZ_mtester mtester( mbuffer, msize, dictionary_size );
if( mtester.test_member() == 0 ) return 6;
return 0;
}
// Return value: 0 = no change, 5 = repaired pos // Return value: 0 = no change, 5 = repaired pos
int repair_dictionary_size( uint8_t * const mbuffer, const long msize ) int repair_dictionary_size( uint8_t * const mbuffer, const long msize )
{ {
@ -155,6 +168,15 @@ long repair_member( uint8_t * const mbuffer, const long long mpos,
} // end namespace } // end namespace
bool safe_seek( const int fd, const long long pos,
const std::string & filename )
{
if( lseek( fd, pos, SEEK_SET ) == pos ) return true;
show_file_error( filename.c_str(), "Seek error", errno );
return false;
}
long seek_write( const int fd, const uint8_t * const buf, const long size, long seek_write( const int fd, const uint8_t * const buf, const long size,
const long long pos ) const long long pos )
{ {
@ -165,16 +187,16 @@ long seek_write( const int fd, const uint8_t * const buf, const long size,
uint8_t * read_member( const int infd, const long long mpos, uint8_t * read_member( const int infd, const long long mpos,
const long long msize, const char * const filename ) const long long msize, const std::string & filename )
{ {
if( msize <= 0 || msize > LONG_MAX ) if( msize <= 0 || msize > LONG_MAX )
{ show_file_error( filename, { show_file_error( filename.c_str(),
"Input file contains member larger than LONG_MAX." ); return 0; } "Input file contains member larger than LONG_MAX." ); return 0; }
if( !safe_seek( infd, mpos, filename ) ) return 0; if( !safe_seek( infd, mpos, filename ) ) return 0;
uint8_t * const buffer = new uint8_t[msize]; uint8_t * const buffer = new uint8_t[msize];
if( readblock( infd, buffer, msize ) != msize ) if( readblock( infd, buffer, msize ) != msize )
{ show_file_error( filename, read_error_msg, errno ); { show_file_error( filename.c_str(), read_error_msg, errno );
delete[] buffer; return 0; } delete[] buffer; return 0; }
return buffer; return buffer;
} }
@ -206,8 +228,10 @@ int byte_repair( const std::string & input_filename,
const long long msize = lzip_index.mblock( i ).size(); const long long msize = lzip_index.mblock( i ).size();
if( !safe_seek( infd, mpos, filename ) ) cleanup_and_fail( 1 ); if( !safe_seek( infd, mpos, filename ) ) cleanup_and_fail( 1 );
long long failure_pos = 0; long long failure_pos = 0;
if( test_member_from_file( infd, msize, &failure_pos ) == 0 ) continue; bool nonzero = false;
if( failure_pos < Lzip_header::size ) // End Of File const int ret = test_member_from_file( infd, msize, &failure_pos, &nonzero );
if( ret == 0 && !nonzero ) continue;
if( ret != 0 && failure_pos < Lzip_header::size ) // End Of File
{ show_error( "Can't repair error in input file." ); { show_error( "Can't repair error in input file." );
cleanup_and_fail( 2 ); } cleanup_and_fail( 2 ); }
if( failure_pos >= msize - 8 ) failure_pos = msize - 8 - 1; if( failure_pos >= msize - 8 ) failure_pos = msize - 8 - 1;
@ -218,13 +242,16 @@ int byte_repair( const std::string & input_filename,
i + 1, lzip_index.members(), mpos + failure_pos ); i + 1, lzip_index.members(), mpos + failure_pos );
std::fflush( stdout ); std::fflush( stdout );
} }
uint8_t * const mbuffer = read_member( infd, mpos, msize, filename ); uint8_t * const mbuffer = read_member( infd, mpos, msize, input_filename );
if( !mbuffer ) cleanup_and_fail( 1 ); if( !mbuffer ) cleanup_and_fail( 1 );
const Lzip_header & header = *(const Lzip_header *)mbuffer; const Lzip_header & header = *(const Lzip_header *)mbuffer;
const unsigned dictionary_size = header.dictionary_size(); const unsigned dictionary_size = header.dictionary_size();
long pos = 0; long pos = 0;
if( !nonzero && mbuffer[6] != 0 ) nonzero = true; // bad DS
if( !gross_damage( mbuffer, msize ) ) if( !gross_damage( mbuffer, msize ) )
{ {
if( nonzero ) pos = repair_nonzero( mbuffer, msize );
if( pos == 0 )
pos = repair_dictionary_size( mbuffer, msize ); pos = repair_dictionary_size( mbuffer, msize );
if( pos == 0 ) if( pos == 0 )
pos = repair_member( mbuffer, mpos, msize, header.size + 1, pos = repair_member( mbuffer, mpos, msize, header.size + 1,
@ -243,12 +270,14 @@ int byte_repair( const std::string & input_filename,
if( !safe_seek( infd, 0, filename ) ) return 1; if( !safe_seek( infd, 0, filename ) ) return 1;
set_signal_handler(); set_signal_handler();
if( !open_outstream( true, true, false, true, to_file ) ) return 1; if( !open_outstream( true, true, false, true, to_file ) ) return 1;
if( !copy_file( infd, outfd ) ) // copy whole file if( !copy_file( infd, outfd, input_filename, output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 ); // copy whole file
} }
if( seek_write( outfd, mbuffer + pos, 1, mpos + pos ) != 1 ) if( ( nonzero && pos != 6 &&
{ show_error( "Error writing output file", errno ); seek_write( outfd, mbuffer + 6, 1, mpos + 6 ) != 1 ) ||
cleanup_and_fail( 1 ); } seek_write( outfd, mbuffer + pos, 1, mpos + pos ) != 1 )
{ show_file_error( printable_name( output_filename, false ),
write_error_msg, errno ); cleanup_and_fail( 1 ); }
} }
delete[] mbuffer; delete[] mbuffer;
if( pos == 0 ) if( pos == 0 )
@ -272,24 +301,24 @@ int byte_repair( const std::string & input_filename,
} }
int debug_delay( const char * const input_filename, int debug_delay( const std::string & input_filename,
const Cl_options & cl_opts, Block range, const Cl_options & cl_opts, Block range,
const char terminator ) const char terminator )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
if( range.end() > lzip_index.cdata_size() ) if( range.end() > lzip_index.cdata_size() )
range.size( std::max( 0LL, lzip_index.cdata_size() - range.pos() ) ); range.size( std::max( 0LL, lzip_index.cdata_size() - range.pos() ) );
if( range.size() <= 0 ) if( range.size() <= 0 )
{ show_file_error( input_filename, "Nothing to do; range is empty." ); { show_file_error( filename, "Nothing to do; range is empty." ); return 0; }
return 0; }
for( long i = 0; i < lzip_index.members(); ++i ) for( long i = 0; i < lzip_index.members(); ++i )
{ {
@ -355,24 +384,25 @@ int debug_delay( const char * const input_filename,
} }
int debug_byte_repair( const char * const input_filename, int debug_byte_repair( const std::string & input_filename,
const Cl_options & cl_opts, const Bad_byte & bad_byte, const Cl_options & cl_opts, const Bad_byte & bad_byte,
const char terminator ) const char terminator )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
long idx = 0; long idx = 0;
for( ; idx < lzip_index.members(); ++idx ) for( ; idx < lzip_index.members(); ++idx )
if( lzip_index.mblock( idx ).includes( bad_byte.pos ) ) break; if( lzip_index.mblock( idx ).includes( bad_byte.pos ) ) break;
if( idx >= lzip_index.members() ) if( idx >= lzip_index.members() )
{ show_file_error( input_filename, "Nothing to do; byte is beyond EOF." ); { show_file_error( filename, "Nothing to do; byte is beyond EOF." );
return 0; } return 0; }
const long long mpos = lzip_index.mblock( idx ).pos(); const long long mpos = lzip_index.mblock( idx ).pos();
@ -392,11 +422,12 @@ int debug_byte_repair( const char * const input_filename,
if( !mbuffer ) return 1; if( !mbuffer ) return 1;
const Lzip_header & header = *(const Lzip_header *)mbuffer; const Lzip_header & header = *(const Lzip_header *)mbuffer;
const unsigned dictionary_size = header.dictionary_size(); const unsigned dictionary_size = header.dictionary_size();
const uint8_t good_value = mbuffer[bad_byte.pos-mpos]; const long long bad_pos = bad_byte.pos - mpos;
const uint8_t good_value = mbuffer[bad_pos];
const uint8_t bad_value = bad_byte( good_value ); const uint8_t bad_value = bad_byte( good_value );
mbuffer[bad_byte.pos-mpos] = bad_value; mbuffer[bad_pos] = bad_value;
long failure_pos = 0; long failure_pos = 0;
if( bad_byte.pos != 5 || isvalid_ds( header.dictionary_size() ) ) if( bad_pos != 5 || isvalid_ds( header.dictionary_size() ) )
{ {
LZ_mtester mtester( mbuffer, msize, header.dictionary_size() ); LZ_mtester mtester( mbuffer, msize, header.dictionary_size() );
if( mtester.test_member() == 0 && mtester.finished() ) if( mtester.test_member() == 0 && mtester.finished() )
@ -419,6 +450,8 @@ int debug_byte_repair( const char * const input_filename,
} }
if( failure_pos >= msize ) failure_pos = msize - 1; if( failure_pos >= msize ) failure_pos = msize - 1;
long pos = repair_dictionary_size( mbuffer, msize ); long pos = repair_dictionary_size( mbuffer, msize );
if( pos == 0 )
if( mbuffer[6] != 0 ) pos = repair_nonzero( mbuffer, msize );
if( pos == 0 ) if( pos == 0 )
pos = repair_member( mbuffer, mpos, msize, header.size + 1, pos = repair_member( mbuffer, mpos, msize, header.size + 1,
header.size + 6, dictionary_size, terminator ); header.size + 6, dictionary_size, terminator );
@ -441,21 +474,21 @@ int debug_byte_repair( const char * const input_filename,
(Packet sizes are a fractionary number of bytes. The packet and marker (Packet sizes are a fractionary number of bytes. The packet and marker
sizes shown by option -X are the number of extra bytes required to decode sizes shown by option -X are the number of extra bytes required to decode
the packet, not counting the data present in the range decoder before and the packet, not counting the data present in the range decoder before and
after the decoding. The max marker size of a 'Sync Flush marker' does not after the decoding.
include the 5 bytes read by rdec.load).
if bad_byte.pos >= cdata_size, bad_byte is ignored. if bad_byte.pos >= cdata_size, bad_byte is ignored.
*/ */
int debug_decompress( const char * const input_filename, int debug_decompress( const std::string & input_filename,
const Cl_options & cl_opts, const Bad_byte & bad_byte, const Cl_options & cl_opts, const Bad_byte & bad_byte,
const bool show_packets ) const bool show_packets )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; struct stat in_stats;
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
outfd = show_packets ? -1 : STDOUT_FILENO; outfd = show_packets ? -1 : STDOUT_FILENO;

2
configure vendored
View file

@ -6,7 +6,7 @@
# to copy, distribute, and modify it. # to copy, distribute, and modify it.
pkgname=lziprecover pkgname=lziprecover
pkgversion=1.25-pre1 pkgversion=1.25-rc1
progname=lziprecover progname=lziprecover
srctrigger=doc/${pkgname}.texi srctrigger=doc/${pkgname}.texi

28
decoder.cc
View file

@ -77,7 +77,7 @@ bool Range_decoder::read_block()
{ {
stream_pos = readblock( infd, buffer, buffer_size ); stream_pos = readblock( infd, buffer, buffer_size );
if( stream_pos != buffer_size && errno ) throw Error( read_error_msg ); if( stream_pos != buffer_size && errno ) throw Error( read_error_msg );
at_stream_end = ( stream_pos < buffer_size ); at_stream_end = stream_pos < buffer_size;
partial_member_pos += pos; partial_member_pos += pos;
pos = 0; pos = 0;
show_dprogress(); show_dprogress();
@ -99,7 +99,7 @@ void LZ_decoder::flush_data()
const long long s = const long long s =
std::min( positive_diff( outend, sp ), (unsigned long long)size ) - i; std::min( positive_diff( outend, sp ), (unsigned long long)size ) - i;
if( s > 0 && writeblock( outfd, buffer + stream_pos + i, s ) != s ) if( s > 0 && writeblock( outfd, buffer + stream_pos + i, s ) != s )
throw Error( "Write error" ); throw Error( write_error_msg );
} }
if( pos >= dictionary_size ) if( pos >= dictionary_size )
{ partial_data_pos += pos; pos = 0; pos_wrapped = true; } { partial_data_pos += pos; pos = 0; pos_wrapped = true; }
@ -180,7 +180,8 @@ bool LZ_decoder::check_trailer( const Pretty_print & pp ) const
/* Return value: 0 = OK, 1 = decoder error, 2 = unexpected EOF, /* Return value: 0 = OK, 1 = decoder error, 2 = unexpected EOF,
3 = trailer error, 4 = unknown marker found, 3 = trailer error, 4 = unknown marker found,
5 = nonzero first LZMA byte found. */ 5 = nonzero first LZMA byte found. */
int LZ_decoder::decode_member( const Pretty_print & pp, const bool ignore_nonzero ) int LZ_decoder::decode_member( const Pretty_print & pp,
const bool ignore_nonzero )
{ {
Bit_model bm_literal[1<<literal_context_bits][0x300]; Bit_model bm_literal[1<<literal_context_bits][0x300];
Bit_model bm_match[State::states][pos_states]; Bit_model bm_match[State::states][pos_states];
@ -244,22 +245,22 @@ int LZ_decoder::decode_member( const Pretty_print & pp, const bool ignore_nonzer
} }
else // match else // match
{ {
rep3 = rep2; rep2 = rep1; rep1 = rep0;
len = rdec.decode_len( match_len_model, pos_state ); len = rdec.decode_len( match_len_model, pos_state );
unsigned distance = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] ); rep0 = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] );
if( distance >= start_dis_model ) if( rep0 >= start_dis_model )
{ {
const unsigned dis_slot = distance; const unsigned dis_slot = rep0;
const int direct_bits = ( dis_slot >> 1 ) - 1; const int direct_bits = ( dis_slot >> 1 ) - 1;
distance = ( 2 | ( dis_slot & 1 ) ) << direct_bits; rep0 = ( 2 | ( dis_slot & 1 ) ) << direct_bits;
if( dis_slot < end_dis_model ) if( dis_slot < end_dis_model )
distance += rdec.decode_tree_reversed( rep0 += rdec.decode_tree_reversed( bm_dis + ( rep0 - dis_slot ),
bm_dis + ( distance - dis_slot ), direct_bits ); direct_bits );
else else
{ {
distance += rep0 += rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits;
rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits; rep0 += rdec.decode_tree_reversed4( bm_align );
distance += rdec.decode_tree_reversed4( bm_align ); if( rep0 == 0xFFFFFFFFU ) // marker found
if( distance == 0xFFFFFFFFU ) // marker found
{ {
rdec.normalize(); rdec.normalize();
flush_data(); flush_data();
@ -271,7 +272,6 @@ int LZ_decoder::decode_member( const Pretty_print & pp, const bool ignore_nonzer
} }
} }
} }
rep3 = rep2; rep2 = rep1; rep1 = rep0; rep0 = distance;
state.set_match(); state.set_match();
if( rep0 >= dictionary_size || ( rep0 >= pos && !pos_wrapped ) ) if( rep0 >= dictionary_size || ( rep0 >= pos && !pos_wrapped ) )
{ flush_data(); return 1; } { flush_data(); return 1; }

20
decoder.h
View file

@ -26,6 +26,7 @@ class Range_decoder
uint32_t range; uint32_t range;
const int infd; // input file descriptor const int infd; // input file descriptor
bool at_stream_end; bool at_stream_end;
bool nonzero_;
bool read_block(); bool read_block();
@ -42,11 +43,12 @@ public:
code( 0 ), code( 0 ),
range( 0xFFFFFFFFU ), range( 0xFFFFFFFFU ),
infd( ifd ), infd( ifd ),
at_stream_end( false ) at_stream_end( false ), nonzero_( false )
{} {}
~Range_decoder() { delete[] buffer; } ~Range_decoder() { delete[] buffer; }
bool nonzero() const { return nonzero_; }
unsigned get_code() const { return code; } unsigned get_code() const { return code; }
bool finished() { return pos >= stream_pos && !read_block(); } bool finished() { return pos >= stream_pos && !read_block(); }
@ -110,8 +112,10 @@ public:
{ {
code = 0; code = 0;
range = 0xFFFFFFFFU; range = 0xFFFFFFFFU;
// check first byte of the LZMA stream // check first byte of the LZMA stream without reading it
if( get_byte() != 0 && !ignore_nonzero ) return false; nonzero_ = buffer[pos] != 0;
if( nonzero_ && !ignore_nonzero ) return false;
get_byte(); // discard first byte of the LZMA stream
for( int i = 0; i < 4; ++i ) code = ( code << 8 ) | get_byte(); for( int i = 0; i < 4; ++i ) code = ( code << 8 ) | get_byte();
return true; return true;
} }
@ -131,7 +135,7 @@ public:
range >>= 1; range >>= 1;
// symbol <<= 1; // symbol <<= 1;
// if( code >= range ) { code -= range; symbol |= 1; } // if( code >= range ) { code -= range; symbol |= 1; }
const bool bit = ( code >= range ); const bool bit = code >= range;
symbol <<= 1; symbol += bit; symbol <<= 1; symbol += bit;
code -= range & ( 0U - bit ); code -= range & ( 0U - bit );
} }
@ -329,14 +333,14 @@ class LZ_decoder
bool fast, fast2; bool fast, fast2;
if( lpos > distance ) if( lpos > distance )
{ {
fast = ( len < dictionary_size - lpos ); fast = len < dictionary_size - lpos;
fast2 = ( fast && len <= lpos - i ); fast2 = fast && len <= lpos - i;
} }
else else
{ {
i += dictionary_size; i += dictionary_size;
fast = ( len < dictionary_size - i ); // (i == pos) may happen fast = len < dictionary_size - i; // (i == pos) may happen
fast2 = ( fast && len <= i - lpos ); fast2 = fast && len <= i - lpos;
} }
if( fast ) // no wrap if( fast ) // no wrap
{ {

8
doc/lziprecover.1
View file

@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.2. .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.2.
.TH LZIPRECOVER "1" "October 2024" "lziprecover 1.25-pre1" "User Commands" .TH LZIPRECOVER "1" "November 2024" "lziprecover 1.25-rc1" "User Commands"
.SH NAME .SH NAME
lziprecover \- recovers data from damaged lzip files lziprecover \- recovers data from damaged lzip files
.SH SYNOPSIS .SH SYNOPSIS
@ -119,12 +119,6 @@ remove members, tdata from files in place
\fB\-\-strip=\fR<list>:d:e:t \fB\-\-strip=\fR<list>:d:e:t
copy files to stdout stripping members given copy files to stdout stripping members given
.TP .TP
\fB\-\-ignore\-empty\fR
ignore empty members in multimember files
.TP
\fB\-\-ignore\-nonzero\fR
ignore a nonzero first LZMA byte
.TP
\fB\-\-loose\-trailing\fR \fB\-\-loose\-trailing\fR
allow trailing data seeming corrupt header allow trailing data seeming corrupt header
.TP .TP

381
doc/lziprecover.info
View file

@ -12,12 +12,13 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir)
Lziprecover Manual Lziprecover Manual
****************** ******************
This manual is for Lziprecover (version 1.25-pre1, 1 October 2024). This manual is for Lziprecover (version 1.25-rc1, 18 November 2024).
* Menu: * Menu:
* Introduction:: Purpose and features of lziprecover * Introduction:: Purpose and features of lziprecover
* Invoking lziprecover:: Command-line interface * Invoking lziprecover:: Command-line interface
* Argument syntax:: By convention, options start with a hyphen
* File format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
* Data safety:: Protecting data from accidental loss * Data safety:: Protecting data from accidental loss
* Fec files:: Forward Error Correction * Fec files:: Forward Error Correction
@ -112,8 +113,9 @@ pdlzip.
If the cause of file corruption is a damaged medium, the combination If the cause of file corruption is a damaged medium, the combination
GNU ddrescue + lziprecover is the recommended option for recovering data GNU ddrescue + lziprecover is the recommended option for recovering data
from damaged lzip files. *Note ddrescue-example::, and *note from damaged files. *Note ddrescue-example::, *note ddrescue-example2::, and
ddrescue-example2::, for examples. *note ddrescue-example3::, for examples. *Note GNU ddrescue manual:
(ddrescue)Top, for details about ddrescue.
If a file is too damaged for lziprecover to repair it, all the If a file is too damaged for lziprecover to repair it, all the
recoverable data in all members of the file can be extracted with the recoverable data in all members of the file can be extracted with the
@ -135,7 +137,7 @@ have been compressed. Decompressed is used to refer to data which have
undergone the process of decompression. undergone the process of decompression.
 
File: lziprecover.info, Node: Invoking lziprecover, Next: File format, Prev: Introduction, Up: Top File: lziprecover.info, Node: Invoking lziprecover, Next: Argument syntax, Prev: Introduction, Up: Top
2 Invoking lziprecover 2 Invoking lziprecover
********************** **********************
@ -150,8 +152,7 @@ first time it appears in the command line. If no file names are specified,
lziprecover decompresses from standard input to standard output. Remember lziprecover decompresses from standard input to standard output. Remember
to prepend './' to any file name beginning with a hyphen, or use '--'. to prepend './' to any file name beginning with a hyphen, or use '--'.
lziprecover supports the following options: *Note Argument syntax: lziprecover supports the following options: *Note Argument syntax::.
(arg_parser)Argument syntax.
'-h' '-h'
'--help' '--help'
@ -175,7 +176,7 @@ lziprecover supports the following options: *Note Argument syntax:
dictionary size of the resulting file (and therefore the amount of dictionary size of the resulting file (and therefore the amount of
memory required to decompress it). Only streamed files with default memory required to decompress it). Only streamed files with default
LZMA properties can be converted; non-streamed lzma-alone files lack LZMA properties can be converted; non-streamed lzma-alone files lack
the "End Of Stream" marker required in lzip files. the 'End Of Stream' marker required in lzip files.
The name of the converted lzip file is derived from that of the The name of the converted lzip file is derived from that of the
original lzma-alone file as follows: original lzma-alone file as follows:
@ -215,23 +216,24 @@ lziprecover supports the following options: *Note Argument syntax:
status 1. If a file fails to decompress, or is a terminal, lziprecover status 1. If a file fails to decompress, or is a terminal, lziprecover
exits immediately with error status 2 without decompressing the rest exits immediately with error status 2 without decompressing the rest
of the files. A terminal is considered an uncompressed file, and of the files. A terminal is considered an uncompressed file, and
therefore invalid. therefore invalid. A multimember file with one or more empty members
is accepted if redirected to standard input or if '-i' is given.
'-D RANGE' '-D RANGE'
'--range-decompress=RANGE' '--range-decompress=RANGE'
Decompress only a range of bytes starting at decompressed byte position Decompress only a range of bytes starting at decompressed byte position
BEGIN and up to byte position END - 1. Byte positions start at 0. This BEGIN and up to byte position END - 1. Byte positions start at 0. The
option provides random access to the data in multimember files; it bytes produced are sent to standard output unless the option '-o' is
only decompresses the members containing the desired data. In order to used. This option provides random access to the data in multimember
guarantee the correctness of the data produced, all members containing files; it only decompresses the members containing the desired data.
any part of the desired data are decompressed and their integrity is In order to guarantee the correctness of the data produced, all
checked. members containing any part of the desired data are decompressed and
their integrity is checked.
Four formats of RANGE are recognized, 'BEGIN', 'BEGIN-END', Four formats of RANGE are recognized, 'BEGIN', 'BEGIN-END',
'BEGIN,SIZE', and ',SIZE'. If only BEGIN is specified, END is taken as 'BEGIN,SIZE', and ',SIZE'. If only BEGIN is specified, END is taken as
the end of the file. If only SIZE is specified, BEGIN is taken as the the end of the file. If only SIZE is specified, BEGIN is taken as the
beginning of the file. The bytes produced are sent to standard output beginning of the file.
unless the option '--output' is used.
'-e' '-e'
'--reproduce' '--reproduce'
@ -325,7 +327,8 @@ lziprecover supports the following options: *Note Argument syntax:
'-k' '-k'
'--keep' '--keep'
Keep (don't delete) input files during decompression. Keep (don't delete) input files during decompression or conversion from
lzma-alone.
'-l' '-l'
'--list' '--list'
@ -336,9 +339,11 @@ lziprecover supports the following options: *Note Argument syntax:
'-v', the dictionary size, the number of members in the file, and the '-v', the dictionary size, the number of members in the file, and the
amount of trailing data (if any) are also printed. With '-vv', the amount of trailing data (if any) are also printed. With '-vv', the
positions and sizes of each member in multimember files are also positions and sizes of each member in multimember files are also
printed. With '-i', format errors are ignored, and with '-ivv', gaps printed. A multimember file with one or more empty members is accepted
between members are shown. The member numbers shown coincide with the if redirected to standard input or if '-i' is given. With '-i', format
file numbers produced by '--split'. errors are ignored, and with '-ivv', gaps between members are shown.
The member numbers start at 1 and coincide with the file numbers
produced by '--split'.
If any file is damaged, does not exist, can't be opened, or is not If any file is damaged, does not exist, can't be opened, or is not
regular, the final exit status is > 0. '-lq' can be used to check regular, the final exit status is > 0. '-lq' can be used to check
@ -358,8 +363,8 @@ lziprecover supports the following options: *Note Argument syntax:
'-n N' '-n N'
'--threads=N' '--threads=N'
Set the maximum number of worker threads for '--fec=create', Set the maximum number of worker threads for '--fec=create',
overriding the system's default. Valid values range from 1 to "as many overriding the system's default. Valid values range from 1 to as many
as your system can support". If this option is not used, lziprecover as your system can support. If this option is not used, lziprecover
tries to detect the number of processors in the system and use it as tries to detect the number of processors in the system and use it as
default value. 'lziprecover --help' shows the system's default value. default value. 'lziprecover --help' shows the system's default value.
@ -367,7 +372,7 @@ lziprecover supports the following options: *Note Argument syntax:
'--output=FILE[/]' '--output=FILE[/]'
If repairing, place the repaired output into FILE instead of into If repairing, place the repaired output into FILE instead of into
FILE_fixed.lz. If splitting, the names of the files produced are in FILE_fixed.lz. If splitting, the names of the files produced are in
the form 'rec01FILE', 'rec02FILE', etc. the form 'rec1FILE', 'rec2FILE', etc.
If creating FEC data and '-c' has not been also specified, write the If creating FEC data and '-c' has not been also specified, write the
FEC data to FILE. If FILE ends with a slash, it is interpreted as the FEC data to FILE. If FILE ends with a slash, it is interpreted as the
@ -415,8 +420,8 @@ lziprecover supports the following options: *Note Argument syntax:
headers or trailers, try to split FILE and then work on each member headers or trailers, try to split FILE and then work on each member
individually. individually.
The names of the files produced are in the form 'rec01FILE', The names of the files produced are in the form 'rec1FILE',
'rec02FILE', etc, and are designed so that the use of wildcards in 'rec2FILE', etc, and are designed so that the use of wildcards in
subsequent processing, for example, subsequent processing, for example,
'lziprecover -cd rec*FILE > recovered_data', processes the files in 'lziprecover -cd rec*FILE > recovered_data', processes the files in
the correct order. The number of digits used in the names varies the correct order. The number of digits used in the names varies
@ -430,7 +435,9 @@ lziprecover supports the following options: *Note Argument syntax:
fails the test, does not exist, can't be opened, or is a terminal, fails the test, does not exist, can't be opened, or is a terminal,
lziprecover continues testing the rest of the files. A final lziprecover continues testing the rest of the files. A final
diagnostic is shown at verbosity level 1 or higher if any file fails diagnostic is shown at verbosity level 1 or higher if any file fails
the test when testing multiple files. the test when testing multiple files. A multimember file with one or
more empty members is accepted if redirected to standard input or if
'-i' is given.
'-v' '-v'
'--verbose' '--verbose'
@ -448,14 +455,13 @@ lziprecover supports the following options: *Note Argument syntax:
'--dump=[MEMBER_LIST][:damaged][:empty][:tdata]' '--dump=[MEMBER_LIST][:damaged][:empty][:tdata]'
Dump the members listed, the damaged members (if any), the empty Dump the members listed, the damaged members (if any), the empty
members (if any), or the trailing data (if any) of one or more regular members (if any), or the trailing data (if any) of one or more regular
multimember files to standard output, or to a file if the option multimember files to standard output, or to a file if the option '-o'
'--output' is used. If more than one file is given, the elements is used. If more than one file is given, the elements dumped from all
dumped from all the files are concatenated. If a file does not exist, the files are concatenated. If a file does not exist, can't be opened,
can't be opened, or is not regular, lziprecover continues processing or is not regular, lziprecover continues processing the rest of the
the rest of the files. If the dump fails in one file, lziprecover files. If the dump fails in one file, lziprecover exits immediately
exits immediately without processing the rest of the files. Only without processing the rest of the files. Only '--dump=tdata' can
'--dump=tdata' can write to a terminal. '--dump=damaged' implies write to a terminal. '--dump=damaged' implies '--ignore-errors'.
'--ignore-errors'.
The argument to '--dump' is a colon-separated list of the following The argument to '--dump' is a colon-separated list of the following
element specifiers; a member list (1,3-6), a reverse member list element specifiers; a member list (1,3-6), a reverse member list
@ -509,35 +515,23 @@ lziprecover supports the following options: *Note Argument syntax:
'--strip=[MEMBER_LIST][:damaged][:empty][:tdata]' '--strip=[MEMBER_LIST][:damaged][:empty][:tdata]'
Copy one or more regular multimember files to standard output (or to a Copy one or more regular multimember files to standard output (or to a
file if the option '--output' is used), stripping the members listed, file if the option '-o' is used), stripping the members listed, the
the damaged members (if any), the empty members (if any), or the damaged members (if any), the empty members (if any), or the trailing
trailing data (if any) from each file. If all members in a file are data (if any) from each file. If all members in a file are selected to
selected to be stripped, the trailing data (if any) are also stripped be stripped, the trailing data (if any) are also stripped even if
even if 'tdata' is not specified. If more than one file is given, the 'tdata' is not specified. If more than one file is given, the files are
files are concatenated. In this case the trailing data are also concatenated. In this case the trailing data are also stripped from
stripped from all but the last file even if 'tdata' is not specified. all but the last file even if 'tdata' is not specified. If a file does
If a file does not exist, can't be opened, or is not regular, not exist, can't be opened, or is not regular, lziprecover continues
lziprecover continues processing the rest of the files. If a file processing the rest of the files. If a file fails to copy, lziprecover
fails to copy, lziprecover exits immediately without processing the exits immediately without processing the rest of the files. See
rest of the files. See '--dump' above for a description of the '--dump' above for a description of the argument.
argument.
'--ignore-empty'
When decompressing, testing, or listing, ignore empty members in
multimember files. By default lziprecover exits with error status 2 if
any empty member is found in a multimember file.
'--ignore-nonzero'
When decompressing or testing, ignore a nonzero first byte in the LZMA
stream. By default lziprecover exits with error status 2 if the first
LZMA byte is nonzero in any member of the input files. Use
'lziprecover --nonzero-repair' to repair any such nonzero bytes.
'--loose-trailing' '--loose-trailing'
When decompressing, testing, or listing, allow trailing data whose When decompressing, testing, or listing, allow trailing data whose
first bytes are so similar to the magic bytes of a lzip header that first bytes are so similar to the magic bytes of a lzip header that
they can be confused with a corrupt header. Use this option if a file they can be confused with a corrupt header. Use this option if a file
triggers a "corrupt header" error and the cause is not indeed a triggers a 'corrupt header' error and the cause is not indeed a
corrupt header. corrupt header.
'--nonzero-repair' '--nonzero-repair'
@ -625,14 +619,15 @@ lziprecover also supports the following debug options (for experts):
Load the compressed FILE into memory, set the byte at POSITION to Load the compressed FILE into memory, set the byte at POSITION to
VALUE, and decompress the modified compressed data to standard output. VALUE, and decompress the modified compressed data to standard output.
If the damaged member can be decompressed to the end (just fails with If the damaged member can be decompressed to the end (just fails with
a CRC mismatch), the members following it are also decompressed. a CRC mismatch), the members following it are also decompressed. *Note
--set-byte::, for a description of VALUE.
'-X[POSITION,VALUE]' '-X[POSITION,VALUE]'
'--show-packets[=POSITION,VALUE]' '--show-packets[=POSITION,VALUE]'
Load the compressed FILE into memory, optionally set the byte at Load the compressed FILE into memory, optionally set the byte at
POSITION to VALUE, decompress the modified compressed data (discarding POSITION to VALUE, decompress the modified compressed data (discarding
the output), and print to standard output descriptions of the LZMA the output), and print to standard output descriptions of the LZMA
packets being decoded. packets being decoded. *Note --set-byte::, for a description of VALUE.
'-Y RANGE' '-Y RANGE'
'--debug-delay=RANGE' '--debug-delay=RANGE'
@ -649,6 +644,7 @@ lziprecover also supports the following debug options (for experts):
'--debug-byte-repair=POSITION,VALUE' '--debug-byte-repair=POSITION,VALUE'
Load the compressed FILE into memory, set the byte at POSITION to Load the compressed FILE into memory, set the byte at POSITION to
VALUE, and then try to repair the byte error. *Note --byte-repair::. VALUE, and then try to repair the byte error. *Note --byte-repair::.
*Note --set-byte::, for a description of VALUE.
'--gf16' '--gf16'
Forces the use of GF(2^16) when creating FEC blocks even if the number Forces the use of GF(2^16) when creating FEC blocks even if the number
@ -681,9 +677,57 @@ corrupt or invalid input file, 3 for an internal consistency error (e.g.,
bug) which caused lziprecover to panic. bug) which caused lziprecover to panic.
 
File: lziprecover.info, Node: File format, Next: Data safety, Prev: Invoking lziprecover, Up: Top File: lziprecover.info, Node: Argument syntax, Next: File format, Prev: Invoking lziprecover, Up: Top
3 File format 3 Syntax of command-line arguments
**********************************
POSIX recommends these conventions for command-line arguments.
* A command-line argument is an option if it begins with a hyphen ('-').
* Option names are single alphanumeric characters.
* Certain options require an argument.
* An option and its argument may or may not appear as separate tokens.
(In other words, the whitespace separating them is optional, unless the
argument is the empty string). Thus, '-o foo' and '-ofoo' are
equivalent.
* One or more options without arguments, followed by at most one option
that takes an argument, may follow a hyphen in a single token. Thus,
'-abc' is equivalent to '-a -b -c'.
* Options typically precede other non-option arguments.
* The argument '--' terminates all options; any following arguments are
treated as non-option arguments, even if they begin with a hyphen.
* A token consisting of a single hyphen character is interpreted as an
ordinary non-option argument. By convention, it is used to specify
standard input, standard output, or a file named '-'.
GNU adds "long options" to these conventions:
* A long option consists of two hyphens ('--') followed by a name made
of alphanumeric characters and hyphens. Option names are typically one
to three words long, with hyphens to separate words. Abbreviations can
be used for the long option names as long as the abbreviations are
unique.
* A long option and its argument may or may not appear as separate
tokens. In the latter case they must be separated by an equal sign '='.
Thus, '--foo bar' and '--foo=bar' are equivalent.
The syntax of options with an optional argument is
'-<short_option><argument>' (without whitespace), or
'--<long_option>=<argument>'.

File: lziprecover.info, Node: File format, Next: Data safety, Prev: Argument syntax, Up: Top
4 File format
************* *************
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -737,7 +781,7 @@ not allowed in multimember files.
Valid values for dictionary size range from 4 KiB to 512 MiB. Valid values for dictionary size range from 4 KiB to 512 MiB.
'LZMA stream' 'LZMA stream'
The LZMA stream, finished by an "End Of Stream" marker. Uses default The LZMA stream, terminated by an 'End Of Stream' marker. Uses default
values for encoder properties. *Note Stream format: (lzip)Stream values for encoder properties. *Note Stream format: (lzip)Stream
format, for a complete description. format, for a complete description.
@ -757,7 +801,7 @@ not allowed in multimember files.
 
File: lziprecover.info, Node: Data safety, Next: Fec files, Prev: File format, Up: Top File: lziprecover.info, Node: Data safety, Next: Fec files, Prev: File format, Up: Top
4 Protecting data from accidental loss 5 Protecting data from accidental loss
************************************** **************************************
It is a fact of life that sometimes data becomes corrupt. Software has It is a fact of life that sometimes data becomes corrupt. Software has
@ -803,7 +847,7 @@ with gzip and bzip2 with respect to data safety:
 
File: lziprecover.info, Node: Merging with a backup, Next: Reproducing a mailbox, Up: Data safety File: lziprecover.info, Node: Merging with a backup, Next: Reproducing a mailbox, Up: Data safety
4.1 Recovering a file using a damaged backup 5.1 Recovering a file using a damaged backup
============================================ ============================================
Let's suppose that you made a compressed backup of your valuable scientific Let's suppose that you made a compressed backup of your valuable scientific
@ -830,7 +874,7 @@ possible to recover a file with thousands of errors.
 
File: lziprecover.info, Node: Reproducing a mailbox, Prev: Merging with a backup, Up: Data safety File: lziprecover.info, Node: Reproducing a mailbox, Prev: Merging with a backup, Up: Data safety
4.2 Recovering new messages using an old backup 5.2 Recovering new messages using an old backup
=============================================== ===============================================
Let's suppose that you make periodic backups of your email messages stored Let's suppose that you make periodic backups of your email messages stored
@ -876,15 +920,14 @@ identical backups (*note performance-of-merge::).
 
File: lziprecover.info, Node: Fec files, Next: Repairing one byte, Prev: Data safety, Up: Top File: lziprecover.info, Node: Fec files, Next: Repairing one byte, Prev: Data safety, Up: Top
5 Forward Error Correction 6 Forward Error Correction
************************** **************************
"Forward Error Correction" (FEC) is any way of protecting data from Forward Error Correction (FEC) is any way of protecting data from corruption
corruption by creating redundant data that can be used later to repair by creating redundant data that can be used later to repair errors in the
errors in the protected data. Lziprecover uses a Hilbert-based Reed-Solomon protected data. Lziprecover uses a Hilbert-based Reed-Solomon code to create
code to create one fec file (with extension '.fec') for each file that one fec file (with extension '.fec') for each file that needs to be
needs to be protected. The fec files created by lziprecover are protected. The fec files created by lziprecover are reproducible.
reproducible.
Reed-Solomon is the most space-efficient Error Correcting Code (ECC) for Reed-Solomon is the most space-efficient Error Correcting Code (ECC) for
data stored in block devices. It creates redundant FEC blocks in such a way data stored in block devices. It creates redundant FEC blocks in such a way
@ -892,8 +935,7 @@ that X FEC blocks allow the recuperation of any combination of up to X lost
data blocks. All the blocks (data and FEC) are of the same size, which in data blocks. All the blocks (data and FEC) are of the same size, which in
fec files must be a multiple of 512 bytes. Reed-Solomon is not optimum for fec files must be a multiple of 512 bytes. Reed-Solomon is not optimum for
corruption affecting random single bits in a file because each corrupt bit corruption affecting random single bits in a file because each corrupt bit
invalidates the whole block containing it. But in block devices, scattered invalidates the whole block containing it.
bit flips should not happen.
Usually, a corrupt file does not provide an indication of where the Usually, a corrupt file does not provide an indication of where the
corruption is located. Therefore, each fec file stores one or two arrays of corruption is located. Therefore, each fec file stores one or two arrays of
@ -921,7 +963,7 @@ must be intact to provide 'prodata_size', 'prodata_md5', and 'gf16'.
 
File: lziprecover.info, Node: How Reed-Solomon works, Next: Implementation details, Up: Fec files File: lziprecover.info, Node: How Reed-Solomon works, Next: Implementation details, Up: Fec files
5.1 How Reed-Solomon works 6.1 How Reed-Solomon works
========================== ==========================
To illustrate how Reed-Solomon works on the BEC, we will use an example with To illustrate how Reed-Solomon works on the BEC, we will use an example with
@ -944,8 +986,8 @@ p, q, and r can be computed from the values of x, y, and z:
Now, if the values of x and y are lost because of data corruption, they Now, if the values of x and y are lost because of data corruption, they
can be recomputed by using any two of the three equations above. For can be recomputed by using any two of the three equations above. For
example, if we replace the known values of z, p, q, and r in equations (1) example, if we replace the known values of z, p, and q in equations (1) and
and (2) we get: (2) we get:
x + y + 3 = 6 (1b) x + y + 3 = 6 (1b)
x + 2y + 9 = 14 (2b) x + 2y + 9 = 14 (2b)
@ -982,7 +1024,7 @@ obtain the values of x and y (D = A^-1 * F):
 
File: lziprecover.info, Node: Implementation details, Next: Creating fec files, Prev: How Reed-Solomon works, Up: Fec files File: lziprecover.info, Node: Implementation details, Next: Creating fec files, Prev: How Reed-Solomon works, Up: Fec files
5.2 How lziprecover implements Reed-Solomon 6.2 How lziprecover implements Reed-Solomon
=========================================== ===========================================
Lziprecover's implementation of Reed-Solomon can manage up to 128 data Lziprecover's implementation of Reed-Solomon can manage up to 128 data
@ -1011,17 +1053,17 @@ blocks.
Lziprecover implements GF(2^8) with polynomial 0x11D and GF(2^16) with Lziprecover implements GF(2^8) with polynomial 0x11D and GF(2^16) with
polynomial 0x1100B. polynomial 0x1100B.
A Hilbert matrix is defined as 'A[i][j] = 1 / (i + j + 1)' for i and j A Hilbert matrix is defined as A[i][j] = 1 / (i + j + 1) for i,j >= 0.
>= 0. But as in a Galois Field addition is exclusive or, applying the But, as in a Galois Field the addition is the exclusive or operation,
Hilbert definition produces a singular (non invertible) matrix. To avoid applying the Hilbert definition produces a singular (non invertible)
this problem, lziprecover uses a Hilbert matrix starting at row matrix. To avoid this problem, lziprecover uses a Hilbert matrix starting
'gf_size / 2'. I.e., 'A[i][j] = 1 / (i + gf_size / 2 + j)' for at row r0 = gf_size / 2. I.e., A[i][j] = 1 / (i + j + r0) for
'0 <= i,j < gf_size / 2'. (gf_size is the size of the Galois Field). 0 <= i,j < r0. ('gf_size' is the size of the Galois Field).
 
File: lziprecover.info, Node: Creating fec files, Next: Testing with fec files, Prev: Implementation details, Up: Fec files File: lziprecover.info, Node: Creating fec files, Next: Testing with fec files, Prev: Implementation details, Up: Fec files
5.3 How to create fec files 6.3 How to create fec files
=========================== ===========================
Example 1: Create the fec file 'archive.tar.lz.fec' and store it in the Example 1: Create the fec file 'archive.tar.lz.fec' and store it in the
@ -1039,10 +1081,15 @@ Example 3: Create recursively one fec file for each file in the directory
lziprecover -v -r -Fc -o fec/ datadir lziprecover -v -r -Fc -o fec/ datadir
Example 4: Create fec files for a collection of photos stored in directory
'photos' and store them in the directory 'photos-fec'.
lziprecover -v -Fc -o photos-fec/ photos/*
 
File: lziprecover.info, Node: Testing with fec files, Next: Repairing with fec files, Prev: Creating fec files, Up: Fec files File: lziprecover.info, Node: Testing with fec files, Next: Repairing with fec files, Prev: Creating fec files, Up: Fec files
5.4 How to test files using fec files 6.4 How to test files using fec files
===================================== =====================================
Example 1: Test the integrity of 'archive.tar.lz' using the fec file Example 1: Test the integrity of 'archive.tar.lz' using the fec file
@ -1061,10 +1108,15 @@ directory 'fec'.
lziprecover -v -r -Ft --fec-file=fec/ datadir lziprecover -v -r -Ft --fec-file=fec/ datadir
Example 4: Test the integrity of a collection of photos stored in directory
'photos' using fec files from directory 'photos-fec'.
lziprecover -v -Ft --fec-file=photos-fec/ photos/*
 
File: lziprecover.info, Node: Repairing with fec files, Next: Fec file format, Prev: Testing with fec files, Up: Fec files File: lziprecover.info, Node: Repairing with fec files, Next: Fec file format, Prev: Testing with fec files, Up: Fec files
5.5 How to repair files using fec files 6.5 How to repair files using fec files
======================================= =======================================
Example 1: Repair the file 'archive.tar.lz' using the fec file Example 1: Repair the file 'archive.tar.lz' using the fec file
@ -1084,10 +1136,22 @@ directory 'fec'.
lziprecover -v -r -Fr --fec-file=fec/ datadir lziprecover -v -r -Fr --fec-file=fec/ datadir
Example 4: Recover a collection of photos from a damaged external drive
('/dev/sdc1'). The photos are in directory 'photos', and the fec files are
in directory 'photos-fec'.
ddrescue -b4096 -r10 /dev/sdc1 hdimage mapfile
mount -o loop,ro hdimage /mnt/hdimage
cp -a /mnt/hdimage/photos photos
cp -a /mnt/hdimage/photos-fec photos-fec
umount /mnt/hdimage
lziprecover -v -Fr --fec-file=photos-fec/ photos/*
(Check and rename repaired files. They are named 'photos/*_fixed')
 
File: lziprecover.info, Node: Fec file format, Prev: Repairing with fec files, Up: Fec files File: lziprecover.info, Node: Fec file format, Prev: Repairing with fec files, Up: Fec files
5.6 Fec file format 6.6 Fec file format
=================== ===================
A fec file consists of one chksum packet, one or more fec packets, and one A fec file consists of one chksum packet, one or more fec packets, and one
@ -1127,7 +1191,7 @@ achieved by a careful design, without adding any padding bytes.
The fec file format has an overhead of 8 bytes per protected data block, The fec file format has an overhead of 8 bytes per protected data block,
plus 16 bytes per FEC block, plus 80 bytes. plus 16 bytes per FEC block, plus 80 bytes.
5.6.1 Chksum packet 6.6.1 Chksum packet
------------------- -------------------
A chksum packet contains one CRC for each of the N data blocks in the A chksum packet contains one CRC for each of the N data blocks in the
@ -1179,7 +1243,7 @@ payload_crc 36 + 4N 4
present) contains an array of CRC32-Cs. present) contains an array of CRC32-Cs.
For the expected thousands of bit flips caused by a zeroed sector, a For the expected thousands of bit flips caused by a zeroed sector, a
"symmetric" CRC like CRC32 is probably better than CRC32-C, which symmetric CRC like CRC32 is probably better than CRC32-C, which
detects all the errors with an odd number of bit flips at the expense detects all the errors with an odd number of bit flips at the expense
of a larger number of undetected errors with an even number of bit of a larger number of undetected errors with an even number of bit
flips. flips.
@ -1187,7 +1251,7 @@ payload_crc 36 + 4N 4
'payload_crc' 'payload_crc'
CRC32 of the crc_array. CRC32 of the crc_array.
5.6.2 Fec packet 6.6.2 Fec packet
---------------- ----------------
A fec packet contains one FEC block and is structured as shown in the A fec packet contains one FEC block and is structured as shown in the
@ -1224,7 +1288,7 @@ payload_crc 12 + fbs 4
 
File: lziprecover.info, Node: Repairing one byte, Next: Merging files, Prev: Fec files, Up: Top File: lziprecover.info, Node: Repairing one byte, Next: Merging files, Prev: Fec files, Up: Top
6 Repairing one byte 7 Repairing one byte
******************** ********************
Lziprecover can repair perfectly most files with small errors (up to one Lziprecover can repair perfectly most files with small errors (up to one
@ -1238,11 +1302,11 @@ most common forms of data corruption.
is limited to 2 GiB on 32-bit systems. is limited to 2 GiB on 32-bit systems.
The error may be located anywhere in the file except in the first 5 The error may be located anywhere in the file except in the first 5
bytes of each member header or in the 'Member size' field of the trailer bytes of each member header (magic and version) or in the 'Member size'
(last 8 bytes of each member). If the error is in the header it can be field of the trailer (last 8 bytes of each member). If the error is in the
easily repaired with a text editor like GNU Moe (*note File format::). If header it can be easily repaired with a text editor like GNU Moe (*note
the error is in the member size, it is enough to ignore the message about File format::). If the error is in the member size, it is enough to ignore
'bad member size' when decompressing. the message about 'bad member size' when decompressing.
Bit flip happens when one bit in the file is changed from 0 to 1 or vice Bit flip happens when one bit in the file is changed from 0 to 1 or vice
versa. It may be caused by bad RAM or even by natural radiation. I have versa. It may be caused by bad RAM or even by natural radiation. I have
@ -1252,7 +1316,7 @@ seen a case of bit flip in a file stored on an USB flash drive.
transmission errors or I/O errors just affect one byte, or even one bit, of transmission errors or I/O errors just affect one byte, or even one bit, of
the file. Also, unlike magnetic media, where errors usually affect a whole the file. Also, unlike magnetic media, where errors usually affect a whole
sector, solid-state storage devices tend to produce single-byte errors, sector, solid-state storage devices tend to produce single-byte errors,
making of lzip the perfect format for data stored on such devices. which lziprecover can repair.
Repairing a file can take some time. Small files or files with the error Repairing a file can take some time. Small files or files with the error
located near the beginning can be repaired in a few seconds. But repairing located near the beginning can be repaired in a few seconds. But repairing
@ -1266,7 +1330,7 @@ repairs more efficiently the worst errors.
 
File: lziprecover.info, Node: Merging files, Next: Reproducing one sector, Prev: Repairing one byte, Up: Top File: lziprecover.info, Node: Merging files, Next: Reproducing one sector, Prev: Repairing one byte, Up: Top
7 Merging files 8 Merging files
*************** ***************
If you have several copies of a file but all of them are too damaged to If you have several copies of a file but all of them are too damaged to
@ -1320,10 +1384,8 @@ identical to the original, in just 5 seconds:
than the number of corrupt bytes (3104) because contiguous corrupt bytes than the number of corrupt bytes (3104) because contiguous corrupt bytes
are counted as a single multibyte error. are counted as a single multibyte error.
Example 1: Recover a compressed backup from two copies on CD-ROM with Example 1: Recover a compressed backup from two copies on CD-ROM with
error-checked merging of copies. *Note GNU ddrescue manual: (ddrescue)Top, error-checked merging of copies.
for details about ddrescue.
ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1 ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1
mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage
@ -1339,7 +1401,6 @@ for details about ddrescue.
lziprecover -tv backup.tar.lz lziprecover -tv backup.tar.lz
backup.tar.lz: ok backup.tar.lz: ok
Example 2: Recover the first volume of those created with the command Example 2: Recover the first volume of those created with the command
'lzip -b 32MiB -S 650MB big_db' from two copies, 'big_db1_00001.lz' and 'lzip -b 32MiB -S 650MB big_db' from two copies, 'big_db1_00001.lz' and
'big_db2_00001.lz', with member 07 damaged in the first copy, member 18 'big_db2_00001.lz', with member 07 damaged in the first copy, member 18
@ -1354,7 +1415,7 @@ correct file produced is saved in 'big_db_00001.lz'.
 
File: lziprecover.info, Node: Reproducing one sector, Next: Tarlz, Prev: Merging files, Up: Top File: lziprecover.info, Node: Reproducing one sector, Next: Tarlz, Prev: Merging files, Up: Top
8 Reproducing one sector 9 Reproducing one sector
************************ ************************
Lziprecover can recover a zeroed sector in a lzip file by concatenating the Lziprecover can recover a zeroed sector in a lzip file by concatenating the
@ -1430,7 +1491,7 @@ header, and that the archive can be reproduced. The tarlz format has minimum
overhead. It uses basic ustar headers, and only adds extended pax headers overhead. It uses basic ustar headers, and only adds extended pax headers
when they are required. when they are required.
8.1 Performance of '--reproduce' 9.1 Performance of '--reproduce'
================================ ================================
Reproduce mode is especially useful when recovering a corrupt backup (or a Reproduce mode is especially useful when recovering a corrupt backup (or a
@ -1483,7 +1544,6 @@ for a different version of the software.
Member reproduced successfully. Member reproduced successfully.
Copy of input file reproduced successfully. Copy of input file reproduced successfully.
Example 2: Recover a damaged backup with a zeroed sector of 4096 bytes at Example 2: Recover a damaged backup with a zeroed sector of 4096 bytes at
file position 1019904, using as reference a previous backup. The damaged file position 1019904, using as reference a previous backup. The damaged
backup comes from a damaged partition copied with ddrescue. backup comes from a damaged partition copied with ddrescue.
@ -1505,7 +1565,6 @@ backup comes from a damaged partition copied with ddrescue.
Member reproduced successfully. Member reproduced successfully.
Copy of input file reproduced successfully. Copy of input file reproduced successfully.
Example 3: Recover a damaged backup with a zeroed sector of 4096 bytes at Example 3: Recover a damaged backup with a zeroed sector of 4096 bytes at
file position 1019904, using as reference a file from the filesystem. (If file position 1019904, using as reference a file from the filesystem. (If
the zeroed sector encodes (part of) a tar header, the tarball can't be the zeroed sector encodes (part of) a tar header, the tarball can't be
@ -1541,8 +1600,8 @@ has been renamed.
 
File: lziprecover.info, Node: Tarlz, Next: File names, Prev: Reproducing one sector, Up: Top File: lziprecover.info, Node: Tarlz, Next: File names, Prev: Reproducing one sector, Up: Top
9 Options supporting the tar.lz format 10 Options supporting the tar.lz format
************************************** ***************************************
Tarlz is a massively parallel (multi-threaded) combined implementation of Tarlz is a massively parallel (multi-threaded) combined implementation of
the tar archiver and the lzip compressor. the tar archiver and the lzip compressor.
@ -1562,8 +1621,8 @@ alignment between tar members and lzip members minimizes the amount of data
lost in case of corruption. In this chapter we'll explain the ways in which lost in case of corruption. In this chapter we'll explain the ways in which
lziprecover can recover and process multimember tar.lz archives. lziprecover can recover and process multimember tar.lz archives.
9.1 Recovering damaged multimember tar.lz archives 10.1 Recovering damaged multimember tar.lz archives
================================================== ===================================================
If you have several copies of the damaged archive, try merging them first If you have several copies of the damaged archive, try merging them first
because merging has a high probability of success. *Note Merging files::. If because merging has a high probability of success. *Note Merging files::. If
@ -1604,8 +1663,8 @@ possible from each damaged member in 'bad_members.tar.lz':
cd tmp cd tmp
tarlz --keep-damaged -xvf ../bad_members.tar.lz tarlz --keep-damaged -xvf ../bad_members.tar.lz
9.2 Processing multimember tar.lz archives 10.2 Processing multimember tar.lz archives
========================================== ===========================================
Lziprecover is able to copy a list of members from a file to another. For Lziprecover is able to copy a list of members from a file to another. For
example the command example the command
@ -1618,7 +1677,7 @@ end-of-file blocks.
 
File: lziprecover.info, Node: File names, Next: Trailing data, Prev: Tarlz, Up: Top File: lziprecover.info, Node: File names, Next: Trailing data, Prev: Tarlz, Up: Top
10 Names of the files produced by lziprecover 11 Names of the files produced by lziprecover
********************************************* *********************************************
The name of the fixed file produced by '--byte-repair' and '--merge' is The name of the fixed file produced by '--byte-repair' and '--merge' is
@ -1634,7 +1693,7 @@ string '_fixed' is inserted before the extension.
 
File: lziprecover.info, Node: Trailing data, Next: Examples, Prev: File names, Up: Top File: lziprecover.info, Node: Trailing data, Next: Examples, Prev: File names, Up: Top
11 Extra data appended to the file 12 Extra data appended to the file
********************************** **********************************
Sometimes extra data are found appended to a lzip file after the last Sometimes extra data are found appended to a lzip file after the last
@ -1644,7 +1703,7 @@ member. Such trailing data may be:
example when writing to a tape. It is safe to append any amount of example when writing to a tape. It is safe to append any amount of
padding zero bytes to a lzip file. padding zero bytes to a lzip file.
* Useful data added by the user; an "End Of File" string (to check that * Useful data added by the user; an 'End Of File' string (to check that
the file has not been truncated), a cryptographically secure hash, a the file has not been truncated), a cryptographically secure hash, a
description of file contents, etc. It is safe to append any amount of description of file contents, etc. It is safe to append any amount of
text to a lzip file as long as none of the first four bytes of the text to a lzip file as long as none of the first four bytes of the
@ -1691,7 +1750,6 @@ Example 1: Add a comment or description to a compressed file.
# This command removes the comment from file.lz # This command removes the comment from file.lz
lziprecover --remove=tdata file.lz lziprecover --remove=tdata file.lz
Example 2: Add and check a cryptographically secure hash. (This may be Example 2: Add and check a cryptographically secure hash. (This may be
convenient, but a separate copy of the hash must be kept in a safe place to convenient, but a separate copy of the hash must be kept in a safe place to
guarantee that both file and hash have not been maliciously replaced). guarantee that both file and hash have not been maliciously replaced).
@ -1703,7 +1761,7 @@ guarantee that both file and hash have not been maliciously replaced).
 
File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: Trailing data, Up: Top File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: Trailing data, Up: Top
12 A small tutorial with examples 13 A small tutorial with examples
********************************* *********************************
Example 1: Extract all the files from archive 'foo.tar.lz'. Example 1: Extract all the files from archive 'foo.tar.lz'.
@ -1763,7 +1821,7 @@ integrity of the resulting files.
 
File: lziprecover.info, Node: Unzcrash, Next: Problems, Prev: Examples, Up: Top File: lziprecover.info, Node: Unzcrash, Next: Problems, Prev: Examples, Up: Top
13 Testing the robustness of decompressors 14 Testing the robustness of decompressors
****************************************** ******************************************
*Note --unzcrash::, for a faster way of testing the robustness of lzip. *Note --unzcrash::, for a faster way of testing the robustness of lzip.
@ -1849,10 +1907,11 @@ unzcrash supports the following options:
'-B[SIZE][,VALUE]' '-B[SIZE][,VALUE]'
'--block[=SIZE][,VALUE]' '--block[=SIZE][,VALUE]'
Test block errors of given SIZE, simulating a whole sector I/O error. Test block errors of given SIZE, simulating a whole sector I/O error
SIZE defaults to 512 bytes. VALUE defaults to 0. By default, only by setting all the bytes in the block to VALUE before attempting
contiguous, non-overlapping blocks are tested, but this may be changed decompression. SIZE defaults to 512 bytes. VALUE defaults to 0. By
with the option '--delta'. default, only contiguous, non-overlapping blocks are tested, but this
may be changed with the option '--delta'.
'-d N' '-d N'
'--delta=N' '--delta=N'
@ -1918,7 +1977,7 @@ bug) which caused unzcrash to panic.
 
File: lziprecover.info, Node: Problems, Next: Concept index, Prev: Unzcrash, Up: Top File: lziprecover.info, Node: Problems, Next: Concept index, Prev: Unzcrash, Up: Top
14 Reporting bugs 15 Reporting bugs
***************** *****************
There are probably bugs in lziprecover. There are certainly errors and There are probably bugs in lziprecover. There are certainly errors and
@ -1939,6 +1998,7 @@ Concept index
[index] [index]
* Menu: * Menu:
* argument syntax: Argument syntax. (line 6)
* bugs: Problems. (line 6) * bugs: Problems. (line 6)
* chksum packet: Fec file format. (line 46) * chksum packet: Fec file format. (line 46)
* data safety: Data safety. (line 6) * data safety: Data safety. (line 6)
@ -1973,40 +2033,43 @@ Concept index
 
Tag Table: Tag Table:
Node: Top226 Node: Top226
Node: Introduction1463 Node: Introduction1535
Node: Invoking lziprecover6223 Node: Invoking lziprecover6387
Ref: --trailing-error7167 Ref: --trailing-error7308
Ref: --byte-repair8261 Ref: --byte-repair8402
Ref: range-format10138 Ref: range-format10483
Ref: --reproduce10473 Ref: --reproduce10728
Ref: --unzcrash28457 Ref: --unzcrash28455
Node: File format32896 Node: Argument syntax33048
Node: Data safety35653 Node: File format35005
Node: Merging with a backup37899 Node: Data safety37759
Node: Reproducing a mailbox39162 Node: Merging with a backup40005
Node: Fec files41616 Node: Reproducing a mailbox41268
Node: How Reed-Solomon works43945 Node: Fec files43722
Node: Implementation details46119 Node: How Reed-Solomon works45988
Node: Creating fec files48188 Node: Implementation details48159
Node: Testing with fec files48852 Node: Creating fec files50224
Node: Repairing with fec files49619 Node: Testing with fec files51068
Node: Fec file format50437 Node: Repairing with fec files52023
Ref: fbs53308 Ref: ddrescue-example52841
Node: Repairing one byte55099 Node: Fec file format53351
Node: Merging files57208 Ref: fbs56222
Ref: performance-of-merge58387 Node: Repairing one byte58011
Ref: ddrescue-example59996 Node: Merging files60103
Node: Reproducing one sector61283 Ref: performance-of-merge61282
Ref: performance-of-reproduce65220 Ref: ddrescue-example262890
Ref: ddrescue-example267894 Node: Reproducing one sector64106
Node: Tarlz70314 Ref: performance-of-reproduce68043
Node: File names73981 Ref: ddrescue-example370716
Node: Trailing data74714 Node: Tarlz73135
Node: Examples78028 Node: File names76808
Ref: concat-example78600 Node: Trailing data77541
Node: Unzcrash79999 Node: Examples80854
Node: Problems86385 Ref: concat-example81426
Node: Concept index86937 Node: Unzcrash82825
Ref: --set-byte87437
Node: Problems89295
Node: Concept index89847
 
End Tag Table End Tag Table

259
doc/lziprecover.texi
View file

@ -6,8 +6,8 @@
@finalout @finalout
@c %**end of header @c %**end of header
@set UPDATED 1 October 2024 @set UPDATED 18 November 2024
@set VERSION 1.25-pre1 @set VERSION 1.25-rc1
@dircategory Compression @dircategory Compression
@direntry @direntry
@ -38,6 +38,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}).
@menu @menu
* Introduction:: Purpose and features of lziprecover * Introduction:: Purpose and features of lziprecover
* Invoking lziprecover:: Command-line interface * Invoking lziprecover:: Command-line interface
* Argument syntax:: By convention, options start with a hyphen
* File format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
* Data safety:: Protecting data from accidental loss * Data safety:: Protecting data from accidental loss
* Fec files:: Forward Error Correction * Fec files:: Forward Error Correction
@ -139,8 +140,16 @@ pdlzip.
If the cause of file corruption is a damaged medium, the combination If the cause of file corruption is a damaged medium, the combination
@w{GNU ddrescue + lziprecover} is the recommended option for recovering data @w{GNU ddrescue + lziprecover} is the recommended option for recovering data
from damaged lzip files. @xref{ddrescue-example}, and from damaged files. @xref{ddrescue-example}, @ref{ddrescue-example2}, and
@ref{ddrescue-example2}, for examples. @ref{ddrescue-example3}, for examples.
@ifnothtml
@xref{Top,GNU ddrescue manual,,ddrescue},
@end ifnothtml
@ifhtml
See the
@uref{http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html,,ddrescue manual}
@end ifhtml
for details about ddrescue.
If a file is too damaged for lziprecover to repair it, all the recoverable If a file is too damaged for lziprecover to repair it, all the recoverable
data in all members of the file can be extracted with the following command data in all members of the file can be extracted with the following command
@ -186,11 +195,7 @@ standard output. Remember to prepend @file{./} to any file name beginning
with a hyphen, or use @samp{--}. with a hyphen, or use @samp{--}.
@noindent @noindent
lziprecover supports the following lziprecover supports the following options: @xref{Argument syntax}.
@uref{http://www.nongnu.org/arg-parser/manual/arg_parser_manual.html#Argument-syntax,,options}:
@ifnothtml
@xref{Argument syntax,,,arg_parser}.
@end ifnothtml
@table @code @table @code
@item -h @item -h
@ -211,12 +216,12 @@ garbage that can be safely ignored. @xref{concat-example}.
@item -A @item -A
@itemx --alone-to-lz @itemx --alone-to-lz
Convert lzma-alone files to lzip format without recompressing, just Convert lzma-alone files to lzip format without recompressing, just adding a
adding a lzip header and trailer. The conversion minimizes the lzip header and trailer. The conversion minimizes the dictionary size of the
dictionary size of the resulting file (and therefore the amount of resulting file (and therefore the amount of memory required to decompress
memory required to decompress it). Only streamed files with default LZMA it). Only streamed files with default LZMA properties can be converted;
properties can be converted; non-streamed lzma-alone files lack the "End non-streamed lzma-alone files lack the 'End Of Stream' marker required in
Of Stream" marker required in lzip files. lzip files.
The name of the converted lzip file is derived from that of the original The name of the converted lzip file is derived from that of the original
lzma-alone file as follows: lzma-alone file as follows:
@ -258,24 +263,27 @@ already exists and @option{--force} has not been specified, lziprecover
continues decompressing the rest of the files and exits with error status 1. continues decompressing the rest of the files and exits with error status 1.
If a file fails to decompress, or is a terminal, lziprecover exits If a file fails to decompress, or is a terminal, lziprecover exits
immediately with error status 2 without decompressing the rest of the files. immediately with error status 2 without decompressing the rest of the files.
A terminal is considered an uncompressed file, and therefore invalid. A terminal is considered an uncompressed file, and therefore invalid. A
multimember file with one or more empty members is accepted if redirected to
standard input or if '-i' is given.
@item -D @var{range} @item -D @var{range}
@itemx --range-decompress=@var{range} @itemx --range-decompress=@var{range}
Decompress only a range of bytes starting at decompressed byte position Decompress only a range of bytes starting at decompressed byte position
@var{begin} and up to byte position @w{@var{end} - 1}. Byte positions start @var{begin} and up to byte position @w{@var{end} - 1}. Byte positions start
at 0. This option provides random access to the data in multimember files; at 0. The bytes produced are sent to standard output unless the option
it only decompresses the members containing the desired data. In order to @option{-o} is used. This option provides random access to the data in
guarantee the correctness of the data produced, all members containing any multimember files; it only decompresses the members containing the desired
part of the desired data are decompressed and their integrity is checked. data. In order to guarantee the correctness of the data produced, all
members containing any part of the desired data are decompressed and their
integrity is checked.
@anchor{range-format} @anchor{range-format}
Four formats of @var{range} are recognized, @samp{@var{begin}}, Four formats of @var{range} are recognized, @samp{@var{begin}},
@samp{@var{begin}-@var{end}}, @samp{@var{begin},@var{size}}, and @samp{@var{begin}-@var{end}}, @samp{@var{begin},@var{size}}, and
@samp{,@var{size}}. If only @var{begin} is specified, @var{end} is taken as @samp{,@var{size}}. If only @var{begin} is specified, @var{end} is taken as
the end of the file. If only @var{size} is specified, @var{begin} is taken the end of the file. If only @var{size} is specified, @var{begin} is taken
as the beginning of the file. The bytes produced are sent to standard output as the beginning of the file.
unless the option @option{--output} is used.
@anchor{--reproduce} @anchor{--reproduce}
@item -e @item -e
@ -371,7 +379,8 @@ last) may be wrong.
@item -k @item -k
@itemx --keep @itemx --keep
Keep (don't delete) input files during decompression. Keep (don't delete) input files during decompression or conversion from
lzma-alone.
@item -l @item -l
@itemx --list @itemx --list
@ -381,9 +390,11 @@ even for multimember files. If more than one file is given, a final line
containing the cumulative sizes is printed. With @option{-v}, the dictionary containing the cumulative sizes is printed. With @option{-v}, the dictionary
size, the number of members in the file, and the amount of trailing data (if size, the number of members in the file, and the amount of trailing data (if
any) are also printed. With @option{-vv}, the positions and sizes of each any) are also printed. With @option{-vv}, the positions and sizes of each
member in multimember files are also printed. With @option{-i}, format errors member in multimember files are also printed. A multimember file with one or
are ignored, and with @option{-ivv}, gaps between members are shown. The more empty members is accepted if redirected to standard input or if '-i' is
member numbers shown coincide with the file numbers produced by @option{--split}. given. With @option{-i}, format errors are ignored, and with @option{-ivv},
gaps between members are shown. The member numbers start at 1 and coincide
with the file numbers produced by @option{--split}.
If any file is damaged, does not exist, can't be opened, or is not regular, If any file is damaged, does not exist, can't be opened, or is not regular,
the final exit status is @w{> 0}. @option{-lq} can be used to check quickly the final exit status is @w{> 0}. @option{-lq} can be used to check quickly
@ -402,8 +413,8 @@ the merge mode.
@item -n @var{n} @item -n @var{n}
@itemx --threads=@var{n} @itemx --threads=@var{n}
Set the maximum number of worker threads for @option{--fec=create}, Set the maximum number of worker threads for @option{--fec=create},
overriding the system's default. Valid values range from 1 to "as many as overriding the system's default. Valid values range from 1 to as many as
your system can support". If this option is not used, lziprecover tries to your system can support. If this option is not used, lziprecover tries to
detect the number of processors in the system and use it as default value. detect the number of processors in the system and use it as default value.
@w{@samp{lziprecover --help}} shows the system's default value. @w{@samp{lziprecover --help}} shows the system's default value.
@ -411,7 +422,7 @@ detect the number of processors in the system and use it as default value.
@itemx --output=@var{file}[/] @itemx --output=@var{file}[/]
If repairing, place the repaired output into @var{file} instead of into If repairing, place the repaired output into @var{file} instead of into
@var{file}_fixed.lz. If splitting, the names of the files produced are in @var{file}_fixed.lz. If splitting, the names of the files produced are in
the form @file{rec01@var{file}}, @file{rec02@var{file}}, etc. the form @file{rec1@var{file}}, @file{rec2@var{file}}, etc.
If creating FEC data and @option{-c} has not been also specified, write the If creating FEC data and @option{-c} has not been also specified, write the
FEC data to @var{file}. If @var{file} ends with a slash, it is interpreted FEC data to @var{file}. If @var{file} ends with a slash, it is interpreted
@ -458,8 +469,8 @@ members with corrupt headers or trailers. If other lziprecover functions
fail to work on a multimember @var{file} because of damage in headers or fail to work on a multimember @var{file} because of damage in headers or
trailers, try to split @var{file} and then work on each member individually. trailers, try to split @var{file} and then work on each member individually.
The names of the files produced are in the form @file{rec01@var{file}}, The names of the files produced are in the form @file{rec1@var{file}},
@file{rec02@var{file}}, etc, and are designed so that the use of wildcards @file{rec2@var{file}}, etc, and are designed so that the use of wildcards
in subsequent processing, for example, in subsequent processing, for example,
@w{@samp{lziprecover -cd rec*@var{file} > recovered_data}}, processes the @w{@samp{lziprecover -cd rec*@var{file} > recovered_data}}, processes the
files in the correct order. The number of digits used in the names varies files in the correct order. The number of digits used in the names varies
@ -473,7 +484,8 @@ together with @option{-v} to see information about the files. If a file
fails the test, does not exist, can't be opened, or is a terminal, lziprecover fails the test, does not exist, can't be opened, or is a terminal, lziprecover
continues testing the rest of the files. A final diagnostic is shown at continues testing the rest of the files. A final diagnostic is shown at
verbosity level 1 or higher if any file fails the test when testing multiple verbosity level 1 or higher if any file fails the test when testing multiple
files. files. A multimember file with one or more empty members is accepted if
redirected to standard input or if '-i' is given.
@item -v @item -v
@itemx --verbose @itemx --verbose
@ -489,8 +501,8 @@ operations, and extra information (for example, the failed areas).
@item --dump=[@var{member_list}][:damaged][:empty][:tdata] @item --dump=[@var{member_list}][:damaged][:empty][:tdata]
Dump the members listed, the damaged members (if any), the empty members (if Dump the members listed, the damaged members (if any), the empty members (if
any), or the trailing data (if any) of one or more regular multimember files any), or the trailing data (if any) of one or more regular multimember files
to standard output, or to a file if the option @option{--output} is used. If to standard output, or to a file if the option @option{-o} is used. If more
more than one file is given, the elements dumped from all the files are than one file is given, the elements dumped from all the files are
concatenated. If a file does not exist, can't be opened, or is not regular, concatenated. If a file does not exist, can't be opened, or is not regular,
lziprecover continues processing the rest of the files. If the dump fails in lziprecover continues processing the rest of the files. If the dump fails in
one file, lziprecover exits immediately without processing the rest of the one file, lziprecover exits immediately without processing the rest of the
@ -547,7 +559,7 @@ attempting the removal of trailing data.
@item --strip=[@var{member_list}][:damaged][:empty][:tdata] @item --strip=[@var{member_list}][:damaged][:empty][:tdata]
Copy one or more regular multimember files to standard output (or to a file Copy one or more regular multimember files to standard output (or to a file
if the option @option{--output} is used), stripping the members listed, the if the option @option{-o} is used), stripping the members listed, the
damaged members (if any), the empty members (if any), or the trailing data damaged members (if any), the empty members (if any), or the trailing data
(if any) from each file. If all members in a file are selected to be (if any) from each file. If all members in a file are selected to be
stripped, the trailing data (if any) are also stripped even if @samp{tdata} stripped, the trailing data (if any) are also stripped even if @samp{tdata}
@ -559,22 +571,11 @@ the rest of the files. If a file fails to copy, lziprecover exits
immediately without processing the rest of the files. See @option{--dump} immediately without processing the rest of the files. See @option{--dump}
above for a description of the argument. above for a description of the argument.
@item --ignore-empty
When decompressing, testing, or listing, ignore empty members in multimember
files. By default lziprecover exits with error status 2 if any empty member
is found in a multimember file.
@item --ignore-nonzero
When decompressing or testing, ignore a nonzero first byte in the LZMA
stream. By default lziprecover exits with error status 2 if the first LZMA
byte is nonzero in any member of the input files.
Use @w{@samp{lziprecover --nonzero-repair}} to repair any such nonzero bytes.
@item --loose-trailing @item --loose-trailing
When decompressing, testing, or listing, allow trailing data whose first When decompressing, testing, or listing, allow trailing data whose first
bytes are so similar to the magic bytes of a lzip header that they can bytes are so similar to the magic bytes of a lzip header that they can
be confused with a corrupt header. Use this option if a file triggers a be confused with a corrupt header. Use this option if a file triggers a
"corrupt header" error and the cause is not indeed a corrupt header. 'corrupt header' error and the cause is not indeed a corrupt header.
@item --nonzero-repair @item --nonzero-repair
Repair in place a nonzero first LZMA byte in the files specified. With Repair in place a nonzero first LZMA byte in the files specified. With
@ -666,13 +667,14 @@ Load the compressed @var{file} into memory, set the byte at @var{position}
to @var{value}, and decompress the modified compressed data to standard to @var{value}, and decompress the modified compressed data to standard
output. If the damaged member can be decompressed to the end (just fails output. If the damaged member can be decompressed to the end (just fails
with a CRC mismatch), the members following it are also decompressed. with a CRC mismatch), the members following it are also decompressed.
@xref{--set-byte}, for a description of @var{value}.
@item -X[@var{position},@var{value}] @item -X[@var{position},@var{value}]
@itemx --show-packets[=@var{position},@var{value}] @itemx --show-packets[=@var{position},@var{value}]
Load the compressed @var{file} into memory, optionally set the byte at Load the compressed @var{file} into memory, optionally set the byte at
@var{position} to @var{value}, decompress the modified compressed data @var{position} to @var{value}, decompress the modified compressed data
(discarding the output), and print to standard output descriptions of the (discarding the output), and print to standard output descriptions of the
LZMA packets being decoded. LZMA packets being decoded. @xref{--set-byte}, for a description of @var{value}.
@item -Y @var{range} @item -Y @var{range}
@itemx --debug-delay=@var{range} @itemx --debug-delay=@var{range}
@ -689,6 +691,7 @@ description of @var{range}.
@itemx --debug-byte-repair=@var{position},@var{value} @itemx --debug-byte-repair=@var{position},@var{value}
Load the compressed @var{file} into memory, set the byte at @var{position} Load the compressed @var{file} into memory, set the byte at @var{position}
to @var{value}, and then try to repair the byte error. @xref{--byte-repair}. to @var{value}, and then try to repair the byte error. @xref{--byte-repair}.
@xref{--set-byte}, for a description of @var{value}.
@item --gf16 @item --gf16
Forces the use of GF(2^16) when creating FEC blocks even if the number of Forces the use of GF(2^16) when creating FEC blocks even if the number of
@ -723,6 +726,59 @@ indicate a corrupt or invalid input file, 3 for an internal consistency
error (e.g., bug) which caused lziprecover to panic. error (e.g., bug) which caused lziprecover to panic.
@node Argument syntax
@chapter Syntax of command-line arguments
@cindex argument syntax
POSIX recommends these conventions for command-line arguments.
@itemize @bullet
@item A command-line argument is an option if it begins with a hyphen
(@samp{-}).
@item Option names are single alphanumeric characters.
@item Certain options require an argument.
@item An option and its argument may or may not appear as separate tokens.
(In other words, the whitespace separating them is optional, unless the
argument is the empty string).
Thus, @w{@option{-o foo}} and @option{-ofoo} are equivalent.
@item One or more options without arguments, followed by at most one option
that takes an argument, may follow a hyphen in a single token.
Thus, @option{-abc} is equivalent to @w{@option{-a -b -c}}.
@item Options typically precede other non-option arguments.
@item The argument @samp{--} terminates all options; any following arguments
are treated as non-option arguments, even if they begin with a hyphen.
@item A token consisting of a single hyphen character is interpreted as an
ordinary non-option argument. By convention, it is used to specify standard
input, standard output, or a file named @samp{-}.
@end itemize
@noindent
GNU adds @dfn{long options} to these conventions:
@itemize @bullet
@item A long option consists of two hyphens (@samp{--}) followed by a name
made of alphanumeric characters and hyphens. Option names are typically one
to three words long, with hyphens to separate words. Abbreviations can be
used for the long option names as long as the abbreviations are unique.
@item A long option and its argument may or may not appear as separate
tokens. In the latter case they must be separated by an equal sign @samp{=}.
Thus, @w{@option{--foo bar}} and @option{--foo=bar} are equivalent.
@end itemize
@noindent
The syntax of options with an optional argument is
@option{-<short_option><argument>} (without whitespace), or
@option{--<long_option>=<argument>}.
@node File format @node File format
@chapter File format @chapter File format
@cindex file format @cindex file format
@ -785,7 +841,7 @@ Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
Valid values for dictionary size range from 4 KiB to 512 MiB. Valid values for dictionary size range from 4 KiB to 512 MiB.
@item LZMA stream @item LZMA stream
The LZMA stream, finished by an "End Of Stream" marker. Uses default values The LZMA stream, terminated by an 'End Of Stream' marker. Uses default values
for encoder properties. for encoder properties.
@ifnothtml @ifnothtml
@xref{Stream format,,,lzip}, @xref{Stream format,,,lzip},
@ -932,12 +988,11 @@ identical backups (@pxref{performance-of-merge}).
@chapter Forward Error Correction @chapter Forward Error Correction
@cindex forward error correction @cindex forward error correction
"Forward Error Correction" (FEC) is any way of protecting data from Forward Error Correction (FEC) is any way of protecting data from corruption
corruption by creating redundant data that can be used later to repair by creating redundant data that can be used later to repair errors in the
errors in the protected data. Lziprecover uses a Hilbert-based Reed-Solomon protected data. Lziprecover uses a Hilbert-based Reed-Solomon code to create
code to create one fec file (with extension @file{.fec}) for each file that one fec file (with extension @file{.fec}) for each file that needs to be
needs to be protected. The fec files created by lziprecover are protected. The fec files created by lziprecover are reproducible.
reproducible.
Reed-Solomon is the most space-efficient Error Correcting Code (ECC) for Reed-Solomon is the most space-efficient Error Correcting Code (ECC) for
data stored in block devices. It creates redundant FEC blocks in such a way data stored in block devices. It creates redundant FEC blocks in such a way
@ -945,8 +1000,7 @@ that X FEC blocks allow the recuperation of any combination of up to X lost
data blocks. All the blocks (data and FEC) are of the same size, which in data blocks. All the blocks (data and FEC) are of the same size, which in
fec files must be a multiple of 512 bytes. Reed-Solomon is not optimum for fec files must be a multiple of 512 bytes. Reed-Solomon is not optimum for
corruption affecting random single bits in a file because each corrupt bit corruption affecting random single bits in a file because each corrupt bit
invalidates the whole block containing it. But in block devices, scattered invalidates the whole block containing it.
bit flips should not happen.
Usually, a corrupt file does not provide an indication of where the Usually, a corrupt file does not provide an indication of where the
corruption is located. Therefore, each fec file stores one or two arrays of corruption is located. Therefore, each fec file stores one or two arrays of
@ -1000,8 +1054,7 @@ If we have that x = 1, y = 2, and z = 3, then p = 6, q = 14, and r = 13:
Now, if the values of x and y are lost because of data corruption, they can Now, if the values of x and y are lost because of data corruption, they can
be recomputed by using any two of the three equations above. For example, if be recomputed by using any two of the three equations above. For example, if
we replace the known values of z, p, q, and r in equations (1) and (2) we we replace the known values of z, p, and q in equations (1) and (2) we get:
get:
@example @example
x + y + 3 = 6 (1b) x + y + 3 = 6 (1b)
@ -1076,13 +1129,12 @@ missing data blocks.
Lziprecover implements GF(2^8) with polynomial 0x11D and GF(2^16) with Lziprecover implements GF(2^8) with polynomial 0x11D and GF(2^16) with
polynomial 0x1100B. polynomial 0x1100B.
A Hilbert matrix is defined as @w{@samp{A[i][j] = 1 / (i + j + 1)}} for i A Hilbert matrix is defined as @w{A[i][j] = 1 / (i + j + 1)} for
and j >= 0. But as in a Galois Field addition is exclusive or, applying the @w{i,j >= 0}. But, as in a Galois Field the addition is the exclusive or
Hilbert definition produces a singular (non invertible) matrix. To avoid operation, applying the Hilbert definition produces a singular (non
this problem, lziprecover uses a Hilbert matrix starting at row invertible) matrix. To avoid this problem, lziprecover uses a Hilbert matrix
@w{@samp{gf_size / 2}}. I.e., @w{@samp{A[i][j] = 1 / (i + gf_size / 2 + j)}} starting at row @w{r0 = gf_size / 2}. I.e., @w{A[i][j] = 1 / (i + j + r0)}
for @w{@samp{0 <= i,j < gf_size / 2}}. (gf_size is the size of the Galois for @w{0 <= i,j < r0}. (@samp{gf_size} is the size of the Galois Field).
Field).
@node Creating fec files @node Creating fec files
@ -1113,6 +1165,14 @@ Example 3: Create recursively one fec file for each file in the directory
lziprecover -v -r -Fc -o fec/ datadir lziprecover -v -r -Fc -o fec/ datadir
@end example @end example
@noindent
Example 4: Create fec files for a collection of photos stored in directory
@file{photos} and store them in the directory @file{photos-fec}.
@example
lziprecover -v -Fc -o photos-fec/ photos/*
@end example
@node Testing with fec files @node Testing with fec files
@section How to test files using fec files @section How to test files using fec files
@ -1143,6 +1203,14 @@ directory @file{fec}.
lziprecover -v -r -Ft --fec-file=fec/ datadir lziprecover -v -r -Ft --fec-file=fec/ datadir
@end example @end example
@noindent
Example 4: Test the integrity of a collection of photos stored in directory
@file{photos} using fec files from directory @file{photos-fec}.
@example
lziprecover -v -Ft --fec-file=photos-fec/ photos/*
@end example
@node Repairing with fec files @node Repairing with fec files
@section How to repair files using fec files @section How to repair files using fec files
@ -1174,6 +1242,22 @@ directory @file{fec}.
lziprecover -v -r -Fr --fec-file=fec/ datadir lziprecover -v -r -Fr --fec-file=fec/ datadir
@end example @end example
@anchor{ddrescue-example}
@noindent
Example 4: Recover a collection of photos from a damaged external drive
(@file{/dev/sdc1}). The photos are in directory @file{photos}, and the fec
files are in directory @file{photos-fec}.
@example
ddrescue -b4096 -r10 /dev/sdc1 hdimage mapfile
mount -o loop,ro hdimage /mnt/hdimage
cp -a /mnt/hdimage/photos photos
cp -a /mnt/hdimage/photos-fec photos-fec
umount /mnt/hdimage
lziprecover -v -Fr --fec-file=photos-fec/ photos/*
(Check and rename repaired files. They are named @file{photos/*_fixed})
@end example
@node Fec file format @node Fec file format
@section Fec file format @section Fec file format
@ -1274,9 +1358,9 @@ The first chksum packet contains an array of CRC32s, while the second chksum
packet (if present) contains an array of CRC32-Cs. packet (if present) contains an array of CRC32-Cs.
For the expected thousands of bit flips caused by a zeroed sector, a For the expected thousands of bit flips caused by a zeroed sector, a
"symmetric" CRC like CRC32 is probably better than CRC32-C, which detects symmetric CRC like CRC32 is probably better than CRC32-C, which detects all
all the errors with an odd number of bit flips at the expense of a larger the errors with an odd number of bit flips at the expense of a larger number
number of undetected errors with an even number of bit flips. of undetected errors with an even number of bit flips.
@item payload_crc @item payload_crc
CRC32 of the crc_array. CRC32 of the crc_array.
@ -1334,9 +1418,9 @@ The file is repaired in memory. Therefore, enough virtual memory
@w{(RAM + swap)} to contain the largest damaged member is required. Member @w{(RAM + swap)} to contain the largest damaged member is required. Member
size is limited to @w{2 GiB} on 32-bit systems. size is limited to @w{2 GiB} on 32-bit systems.
The error may be located anywhere in the file except in the first 5 The error may be located anywhere in the file except in the first 5 bytes of
bytes of each member header or in the @samp{Member size} field of the each member header (magic and version) or in the @samp{Member size} field of
trailer (last 8 bytes of each member). If the error is in the header it the trailer (last 8 bytes of each member). If the error is in the header it
can be easily repaired with a text editor like GNU Moe (@pxref{File can be easily repaired with a text editor like GNU Moe (@pxref{File
format}). If the error is in the member size, it is enough to ignore the format}). If the error is in the member size, it is enough to ignore the
message about @samp{bad member size} when decompressing. message about @samp{bad member size} when decompressing.
@ -1349,7 +1433,7 @@ One byte may seem small, but most file corruptions not produced by
transmission errors or I/O errors just affect one byte, or even one bit, transmission errors or I/O errors just affect one byte, or even one bit,
of the file. Also, unlike magnetic media, where errors usually affect a of the file. Also, unlike magnetic media, where errors usually affect a
whole sector, solid-state storage devices tend to produce single-byte whole sector, solid-state storage devices tend to produce single-byte
errors, making of lzip the perfect format for data stored on such devices. errors, which lziprecover can repair.
Repairing a file can take some time. Small files or files with the error Repairing a file can take some time. Small files or files with the error
located near the beginning can be repaired in a few seconds. But located near the beginning can be repaired in a few seconds. But
@ -1421,19 +1505,10 @@ Note that the number of errors reported by lziprecover (2552) is lower
than the number of corrupt bytes (3104) because contiguous corrupt bytes than the number of corrupt bytes (3104) because contiguous corrupt bytes
are counted as a single multibyte error. are counted as a single multibyte error.
@sp 1 @anchor{ddrescue-example2}
@anchor{ddrescue-example}
@noindent @noindent
Example 1: Recover a compressed backup from two copies on CD-ROM with Example 1: Recover a compressed backup from two copies on CD-ROM with
error-checked merging of copies. error-checked merging of copies.
@ifnothtml
@xref{Top,GNU ddrescue manual,,ddrescue},
@end ifnothtml
@ifhtml
See the
@uref{http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html,,ddrescue manual}
@end ifhtml
for details about ddrescue.
@example @example
ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1 ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1
@ -1451,7 +1526,6 @@ lziprecover -tv backup.tar.lz
backup.tar.lz: ok backup.tar.lz: ok
@end example @end example
@sp 1
@noindent @noindent
Example 2: Recover the first volume of those created with the command Example 2: Recover the first volume of those created with the command
@w{@samp{lzip -b 32MiB -S 650MB big_db}} from two copies, @w{@samp{lzip -b 32MiB -S 650MB big_db}} from two copies,
@ -1608,8 +1682,7 @@ Member reproduced successfully.
Copy of input file reproduced successfully. Copy of input file reproduced successfully.
@end example @end example
@sp 1 @anchor{ddrescue-example3}
@anchor{ddrescue-example2}
@noindent @noindent
Example 2: Recover a damaged backup with a zeroed sector of 4096 bytes at Example 2: Recover a damaged backup with a zeroed sector of 4096 bytes at
file position 1019904, using as reference a previous backup. The damaged file position 1019904, using as reference a previous backup. The damaged
@ -1634,7 +1707,6 @@ Member reproduced successfully.
Copy of input file reproduced successfully. Copy of input file reproduced successfully.
@end example @end example
@sp 1
@noindent @noindent
Example 3: Recover a damaged backup with a zeroed sector of 4096 bytes at Example 3: Recover a damaged backup with a zeroed sector of 4096 bytes at
file position 1019904, using as reference a file from the filesystem. (If file position 1019904, using as reference a file from the filesystem. (If
@ -1790,7 +1862,7 @@ example when writing to a tape. It is safe to append any amount of
padding zero bytes to a lzip file. padding zero bytes to a lzip file.
@item @item
Useful data added by the user; an "End Of File" string (to check that the Useful data added by the user; an 'End Of File' string (to check that the
file has not been truncated), a cryptographically secure hash, a description file has not been truncated), a cryptographically secure hash, a description
of file contents, etc. It is safe to append any amount of text to a lzip of file contents, etc. It is safe to append any amount of text to a lzip
file as long as none of the first four bytes of the text matches the file as long as none of the first four bytes of the text matches the
@ -1844,7 +1916,6 @@ lziprecover --strip=tdata file.lz > stripped_file.lz
lziprecover --remove=tdata file.lz lziprecover --remove=tdata file.lz
@end example @end example
@sp 1
@noindent @noindent
Example 2: Add and check a cryptographically secure hash. (This may be Example 2: Add and check a cryptographically secure hash. (This may be
convenient, but a separate copy of the hash must be kept in a safe place convenient, but a separate copy of the hash must be kept in a safe place
@ -2036,10 +2107,11 @@ The number of N-bit errors per byte (N = 1 to 8) is:
@item -B[@var{size}][,@var{value}] @item -B[@var{size}][,@var{value}]
@itemx --block[=@var{size}][,@var{value}] @itemx --block[=@var{size}][,@var{value}]
Test block errors of given @var{size}, simulating a whole sector I/O error. Test block errors of given @var{size}, simulating a whole sector I/O error
@var{size} defaults to 512 bytes. @var{value} defaults to 0. By default, by setting all the bytes in the block to @var{value} before attempting
only contiguous, non-overlapping blocks are tested, but this may be changed decompression. @var{size} defaults to 512 bytes. @var{value} defaults to 0.
with the option @option{--delta}. By default, only contiguous, non-overlapping blocks are tested, but this may
be changed with the option @option{--delta}.
@item -d @var{n} @item -d @var{n}
@itemx --delta=@var{n} @itemx --delta=@var{n}
@ -2049,6 +2121,7 @@ non-overlapping blocks, or truncation sizes. Values of @var{n} smaller than
the block size result in overlapping blocks. (Which is convenient for the block size result in overlapping blocks. (Which is convenient for
testing because there are usually too few non-overlapping blocks in a file). testing because there are usually too few non-overlapping blocks in a file).
@anchor{--set-byte}
@item -e @var{position},@var{value} @item -e @var{position},@var{value}
@itemx --set-byte=@var{position},@var{value} @itemx --set-byte=@var{position},@var{value}
Set byte at @var{position} to @var{value} in the internal buffer after Set byte at @var{position} to @var{value} in the internal buffer after

28
dump_remove.cc
View file

@ -59,7 +59,7 @@ int dump_members( const std::vector< std::string > & filenames,
bool stdin_used = false; bool stdin_used = false;
for( unsigned i = 0; i < filenames.size(); ++i ) for( unsigned i = 0; i < filenames.size(); ++i )
{ {
const bool from_stdin = ( filenames[i] == "-" ); const bool from_stdin = filenames[i] == "-";
if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; } if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; }
const char * const input_filename = const char * const input_filename =
from_stdin ? "(stdin)" : filenames[i].c_str(); from_stdin ? "(stdin)" : filenames[i].c_str();
@ -93,8 +93,8 @@ int dump_members( const std::vector< std::string > & filenames,
if( in == !strip ) if( in == !strip )
{ {
if( !safe_seek( infd, stream_pos, input_filename ) || if( !safe_seek( infd, stream_pos, input_filename ) ||
!copy_file( infd, outfd, mb.pos() - stream_pos ) ) !copy_file( infd, outfd, filenames[i], output_filename,
cleanup_and_fail( 1 ); mb.pos() - stream_pos ) ) cleanup_and_fail( 1 );
copied_size += mb.pos() - stream_pos; ++members; copied_size += mb.pos() - stream_pos; ++members;
} }
else { stripped_size += mb.pos() - stream_pos; ++smembers; } else { stripped_size += mb.pos() - stream_pos; ++smembers; }
@ -106,12 +106,13 @@ int dump_members( const std::vector< std::string > & filenames,
if( !in && member_list.damaged ) if( !in && member_list.damaged )
{ {
if( !safe_seek( infd, mb.pos(), input_filename ) ) cleanup_and_fail( 1 ); if( !safe_seek( infd, mb.pos(), input_filename ) ) cleanup_and_fail( 1 );
in = ( test_member_from_file( infd, mb.size() ) != 0 ); // damaged in = test_member_from_file( infd, mb.size() ) != 0; // damaged
} }
if( in == !strip ) if( in == !strip )
{ {
if( !safe_seek( infd, mb.pos(), input_filename ) || if( !safe_seek( infd, mb.pos(), input_filename ) ||
!copy_file( infd, outfd, mb.size() ) ) cleanup_and_fail( 1 ); !copy_file( infd, outfd, filenames[i], output_filename,
mb.size() ) ) cleanup_and_fail( 1 );
copied_size += mb.size(); ++members; copied_size += mb.size(); ++members;
} }
else { stripped_size += mb.size(); ++smembers; } else { stripped_size += mb.size(); ++smembers; }
@ -131,7 +132,8 @@ int dump_members( const std::vector< std::string > & filenames,
( !strip || i + 1 >= filenames.size() ) ) // strip all but last ( !strip || i + 1 >= filenames.size() ) ) // strip all but last
{ {
if( !safe_seek( infd, cdata_size, input_filename ) || if( !safe_seek( infd, cdata_size, input_filename ) ||
!copy_file( infd, outfd, trailing_size ) ) cleanup_and_fail( 1 ); !copy_file( infd, outfd, filenames[i], output_filename,
trailing_size ) ) cleanup_and_fail( 1 );
copied_tsize += trailing_size; copied_tsize += trailing_size;
} }
else if( trailing_size > 0 ) { stripped_tsize += trailing_size; ++tfiles; } else if( trailing_size > 0 ) { stripped_tsize += trailing_size; ++tfiles; }
@ -210,7 +212,8 @@ int remove_members( const std::vector< std::string > & filenames,
if( stream_pos != prev_end && if( stream_pos != prev_end &&
( !safe_seek( infd, prev_end, filename ) || ( !safe_seek( infd, prev_end, filename ) ||
!safe_seek( fd, stream_pos, filename ) || !safe_seek( fd, stream_pos, filename ) ||
!copy_file( infd, fd, mb.pos() - prev_end ) ) ) !copy_file( infd, fd, filenames[i], filenames[i],
mb.pos() - prev_end ) ) )
{ error = true; set_retval( retval, 1 ); break; } { error = true; set_retval( retval, 1 ); break; }
stream_pos += mb.pos() - prev_end; stream_pos += mb.pos() - prev_end;
} }
@ -224,14 +227,14 @@ int remove_members( const std::vector< std::string > & filenames,
{ {
if( !safe_seek( infd, mb.pos(), filename ) ) if( !safe_seek( infd, mb.pos(), filename ) )
{ error = true; set_retval( retval, 1 ); break; } { error = true; set_retval( retval, 1 ); break; }
in = ( test_member_from_file( infd, mb.size() ) != 0 ); // damaged in = test_member_from_file( infd, mb.size() ) != 0; // damaged
} }
if( !in ) if( !in )
{ {
if( stream_pos != mb.pos() && if( stream_pos != mb.pos() &&
( !safe_seek( infd, mb.pos(), filename ) || ( !safe_seek( infd, mb.pos(), filename ) ||
!safe_seek( fd, stream_pos, filename ) || !safe_seek( fd, stream_pos, filename ) ||
!copy_file( infd, fd, mb.size() ) ) ) !copy_file( infd, fd, filenames[i], filenames[i], mb.size() ) ) )
{ error = true; set_retval( retval, 1 ); break; } { error = true; set_retval( retval, 1 ); break; }
stream_pos += mb.size(); stream_pos += mb.size();
} }
@ -254,7 +257,7 @@ int remove_members( const std::vector< std::string > & filenames,
if( stream_pos != cdata_size && if( stream_pos != cdata_size &&
( !safe_seek( infd, cdata_size, filename ) || ( !safe_seek( infd, cdata_size, filename ) ||
!safe_seek( fd, stream_pos, filename ) || !safe_seek( fd, stream_pos, filename ) ||
!copy_file( infd, fd, trailing_size ) ) ) !copy_file( infd, fd, filenames[i], filenames[i], trailing_size ) ) )
{ close( fd ); close( infd ); set_retval( retval, 1 ); break; } { close( fd ); close( infd ); set_retval( retval, 1 ); break; }
stream_pos += trailing_size; stream_pos += trailing_size;
} }
@ -310,8 +313,7 @@ int nonzero_repair( const std::vector< std::string > & filenames,
const int fd = open_truncable_stream( filename, &in_stats ); const int fd = open_truncable_stream( filename, &in_stats );
if( fd < 0 ) { set_retval( retval, 1 ); continue; } if( fd < 0 ) { set_retval( retval, 1 ); continue; }
const Lzip_index lzip_index( fd, cl_opts, cl_opts.ignore_errors, const Lzip_index lzip_index( fd, cl_opts, true, cl_opts.ignore_errors );
cl_opts.ignore_errors );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ {
show_file_error( filename, lzip_index.error().c_str() ); show_file_error( filename, lzip_index.error().c_str() );
@ -332,7 +334,7 @@ int nonzero_repair( const std::vector< std::string > & filenames,
if( seek_read( fd, header_buf, bufsize, mb.pos() ) != bufsize ) if( seek_read( fd, header_buf, bufsize, mb.pos() ) != bufsize )
{ show_file_error( filename, "Error reading member header", errno ); { show_file_error( filename, "Error reading member header", errno );
set_retval( retval, 1 ); break; } set_retval( retval, 1 ); break; }
if( !header.check( cl_opts.ignore_errors ) ) if( !header.check( true ) )
{ show_file_error( filename, "Member header became corrupt as we read it." ); { show_file_error( filename, "Member header became corrupt as we read it." );
set_retval( retval, 2 ); break; } set_retval( retval, 2 ); break; }
if( *mark == 0 ) continue; if( *mark == 0 ) continue;

1
fec.h
View file

@ -234,7 +234,6 @@ void extract_dirname( const std::string & name, std::string & srcdir );
void replace_dirname( const std::string & name, const std::string & srcdir, void replace_dirname( const std::string & name, const std::string & srcdir,
const std::string & destdir, std::string & outname ); const std::string & destdir, std::string & outname );
bool has_fec_extension( const std::string & name ); bool has_fec_extension( const std::string & name );
const char * printable_name( const std::string & filename, const bool in = true );
int fec_create( const std::vector< std::string > & filenames, int fec_create( const std::vector< std::string > & filenames,
const std::string & default_output_filename, const std::string & default_output_filename,
const unsigned long fb_or_pct, const unsigned cl_block_size, const unsigned long fb_or_pct, const unsigned cl_block_size,

16
fec_create.cc
View file

@ -241,7 +241,7 @@ Coded_fbs compute_fbs( const unsigned long prodata_size,
{ {
const unsigned unit_fbs = isvalid_fbs( cl_block_size ) ? cl_block_size : const unsigned unit_fbs = isvalid_fbs( cl_block_size ) ? cl_block_size :
compute_unit_fbs( prodata_size ); compute_unit_fbs( prodata_size );
const unsigned long max_k = ( fec_level == 0 ) ? max_k8 : max_k16; const unsigned long max_k = (fec_level == 0) ? max_k8 : max_k16;
const unsigned k9 = std::min( ceil_divide( prodata_size, unit_fbs ), max_k ); const unsigned k9 = std::min( ceil_divide( prodata_size, unit_fbs ), max_k );
const unsigned long fbsu9 = divide_fbs( prodata_size, k9, unit_fbs ); const unsigned long fbsu9 = divide_fbs( prodata_size, k9, unit_fbs );
const unsigned long fbsu0 = divide_fbs( prodata_size, max_k8, unit_fbs ); const unsigned long fbsu0 = divide_fbs( prodata_size, max_k8, unit_fbs );
@ -258,9 +258,9 @@ unsigned compute_fec_blocks( const unsigned long prodata_size,
{ {
const unsigned long fbs = coded_fbs.val(); const unsigned long fbs = coded_fbs.val();
const unsigned prodata_blocks = ceil_divide( prodata_size, fbs ); const unsigned prodata_blocks = ceil_divide( prodata_size, fbs );
const unsigned long max_k = ( fec_level == 0 ) ? max_k8 : max_k16; const unsigned long max_k = (fec_level == 0) ? max_k8 : max_k16;
if( !isvalid_fbs( fbs ) || prodata_blocks > max_k ) return 0; if( !isvalid_fbs( fbs ) || prodata_blocks > max_k ) return 0;
const unsigned long max_nk = ( fec_level == 0 ) ? max_k8 : max_nk16; const unsigned long max_nk = (fec_level == 0) ? max_k8 : max_nk16;
unsigned fec_blocks; unsigned fec_blocks;
if( fctype == fc_blocks ) fec_blocks = std::min( max_nk, fb_or_pct ); if( fctype == fc_blocks ) fec_blocks = std::min( max_nk, fb_or_pct );
else else
@ -387,7 +387,8 @@ bool write_fec( const char * const input_filename,
return true; return true;
} }
fail: fail:
show_file_error( input_filename, "Write error", errno ); return false; show_file_error( printable_name( output_filename, false ), write_error_msg,
errno ); return false;
} }
@ -523,13 +524,6 @@ bool has_fec_extension( const std::string & name )
} }
const char * printable_name( const std::string & filename, const bool in )
{
if( filename.empty() || filename == "-" ) return in ? "(stdin)" : "(stdout)";
return filename.c_str();
}
int fec_create( const std::vector< std::string > & filenames, int fec_create( const std::vector< std::string > & filenames,
const std::string & default_output_filename, const std::string & default_output_filename,
const unsigned long fb_or_pct, const unsigned cl_block_size, const unsigned long fb_or_pct, const unsigned cl_block_size,

21
fec_repair.cc
View file

@ -76,7 +76,7 @@ uint8_t * read_file( const std::string & filename, long * const file_sizep )
{ {
struct stat in_stats; // not used struct stat in_stats; // not used
const char * const filenamep = printable_name( filename ); const char * const filenamep = printable_name( filename );
const int infd = ( filename == "-" ) ? const int infd = (filename == "-") ?
STDIN_FILENO : open_instream( filenamep, &in_stats, false ); STDIN_FILENO : open_instream( filenamep, &in_stats, false );
if( infd < 0 ) return 0; if( infd < 0 ) return 0;
long buffer_size = 65536; long buffer_size = 65536;
@ -88,7 +88,7 @@ uint8_t * read_file( const std::string & filename, long * const file_sizep )
if( buffer_size >= LONG_MAX ) if( buffer_size >= LONG_MAX )
{ show_file_error( filenamep, large_file_msg ); { show_file_error( filenamep, large_file_msg );
std::free( buffer ); return 0; } std::free( buffer ); return 0; }
buffer_size = ( buffer_size <= LONG_MAX / 2 ) ? 2 * buffer_size : LONG_MAX; buffer_size = (buffer_size <= LONG_MAX / 2) ? 2 * buffer_size : LONG_MAX;
uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size ); uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size );
if( !tmp ) if( !tmp )
{ show_file_error( filenamep, mem_msg ); std::free( buffer ); return 0; } { show_file_error( filenamep, mem_msg ); std::free( buffer ); return 0; }
@ -794,9 +794,12 @@ int fec_test( const std::vector< std::string > & filenames,
// check tty only once and don't try to delete a tty // check tty only once and don't try to delete a tty
if( !open_outstream( force, false ) || !check_tty_out() ) return 1; if( !open_outstream( force, false ) || !check_tty_out() ) return 1;
} }
if( writeblock( outfd, prodata, prodata_size ) != prodata_size || // write repaired prodata
!close_outstream( &in_stats ) ) // write repaired prodata if( writeblock( outfd, prodata, prodata_size ) != prodata_size )
{ set_retval( retval, 1 ); cleanup_and_fail( retval ); } { show_file_error( printable_name( output_filename, false ),
write_error_msg, errno ); set_retval( retval, 1 ); }
else if( !close_outstream( &in_stats ) ) set_retval( retval, 1 );
if( retval ) cleanup_and_fail( retval );
if( verbosity >= 1 ) if( verbosity >= 1 )
std::fprintf( stderr, "Repaired copy of '%s' written to '%s'\n", std::fprintf( stderr, "Repaired copy of '%s' written to '%s'\n",
input_filenamep, printable_name( output_filename, false ) ); input_filenamep, printable_name( output_filename, false ) );
@ -868,7 +871,7 @@ int fec_dc( const std::string & input_filename,
const unsigned long cblock_size = fec_blocks / cblocks * fbs; const unsigned long cblock_size = fec_blocks / cblocks * fbs;
const unsigned long max_saved_size = cblocks * cblock_size; const unsigned long max_saved_size = cblocks * cblock_size;
uint8_t * const sbuf = new uint8_t[max_saved_size]; // saved data bytes uint8_t * const sbuf = new uint8_t[max_saved_size]; // saved data bytes
const long last_pos = ( prodata_size % cblock_size != 0 ) ? const long last_pos = (prodata_size % cblock_size != 0) ?
prodata_size - prodata_size % cblock_size : prodata_size - cblock_size; prodata_size - prodata_size % cblock_size : prodata_size - cblock_size;
if( verbosity >= 0 ) if( verbosity >= 0 )
{ std::printf( "Testing sets of %u block%s of size %s\n", cblocks, { std::printf( "Testing sets of %u block%s of size %s\n", cblocks,
@ -1034,7 +1037,7 @@ int fec_dZ( const std::string & input_filename,
const int rest = std::min( 2UL, sector_size % fbs ); const int rest = std::min( 2UL, sector_size % fbs );
const long max_saved_size = ( sector_size / fbs + rest ) * fbs; const long max_saved_size = ( sector_size / fbs + rest ) * fbs;
uint8_t * const sbuf = new uint8_t[max_saved_size]; // saved data bytes uint8_t * const sbuf = new uint8_t[max_saved_size]; // saved data bytes
const long last_pos = ( prodata_size % sector_size != 0 ) ? const long last_pos = (prodata_size % sector_size != 0) ?
prodata_size - prodata_size % sector_size : prodata_size - sector_size; prodata_size - prodata_size % sector_size : prodata_size - sector_size;
if( verbosity >= 0 ) if( verbosity >= 0 )
{ std::printf( "Testing blocks of size %s (delta %s)\n", { std::printf( "Testing blocks of size %s (delta %s)\n",
@ -1042,8 +1045,8 @@ int fec_dZ( const std::string & input_filename,
std::fflush( stdout ); } std::fflush( stdout ); }
unsigned long combinations = 0, repair_attempts = 0, successes = 0, unsigned long combinations = 0, repair_attempts = 0, successes = 0,
failed_comparisons = 0; failed_comparisons = 0;
int pct = ( prodata_size >= 1000 && isatty( STDERR_FILENO ) ) ? 0 : 100; int pct = (prodata_size >= 1000 && isatty( STDERR_FILENO )) ? 0 : 100;
long pct_pos = ( pct < 100 ) ? 0 : prodata_size; long pct_pos = (pct < 100) ? 0 : prodata_size;
const int saved_verbosity = verbosity; const int saved_verbosity = verbosity;
verbosity = -1; // suppress all messages verbosity = -1; // suppress all messages
for( long pos = 0; pos <= last_pos; pos += delta ) for( long pos = 0; pos <= last_pos; pos += delta )

2
gf16.cc
View file

@ -108,7 +108,7 @@ bool check_inverse( const uint16_t * const A, const uint16_t * const B,
{ if( print && row ) std::fputc( '\n', stderr ); return false; } { if( print && row ) std::fputc( '\n', stderr ); return false; }
if( print ) std::fprintf( stderr, "\r%5u rows checked \r", row + 1 ); if( print ) std::fprintf( stderr, "\r%5u rows checked \r", row + 1 );
} }
return true; // A * B == I return true;
} }

4
gf8.cc
View file

@ -57,7 +57,7 @@ struct Galois8_table // addition/subtraction is exclusive or
{ {
uint8_t * const mul_row = mul_table + i * size; uint8_t * const mul_row = mul_table + i * size;
for( int j = 1; j < size; ++j ) for( int j = 1; j < size; ++j )
mul_row[j] = ilog[(log[i] + log[j]) % (size - 1)]; mul_row[j] = ilog[(log[i] + log[j]) % (size-1)];
} }
for( int i = 0; i < size; ++i ) for( int i = 0; i < size; ++i )
mul_table[0 * size + i] = mul_table[i * size + 0] = 0; mul_table[0 * size + i] = mul_table[i * size + 0] = 0;
@ -79,7 +79,7 @@ bool check_inverse( const uint8_t * const A, const uint8_t * const B,
uint8_t sum = 0; uint8_t sum = 0;
for( unsigned i = 0; i < k; ++i, ++pa, pb += k ) for( unsigned i = 0; i < k; ++i, ++pa, pb += k )
sum ^= gf.mul_table[*pa * gf.size + *pb]; sum ^= gf.mul_table[*pa * gf.size + *pb];
if( sum != ( row == col ) ) return false; // A * B != I if( sum != ( row == col ) ) return false;
} }
return true; return true;
} }

14
list.cc
View file

@ -17,6 +17,7 @@
#define _FILE_OFFSET_BITS 64 #define _FILE_OFFSET_BITS 64
#include <cerrno>
#include <cstdio> #include <cstdio>
#include <cstring> #include <cstring>
#include <string> #include <string>
@ -57,7 +58,7 @@ int list_files( const std::vector< std::string > & filenames,
for( unsigned i = 0; i < filenames.size(); ++i ) for( unsigned i = 0; i < filenames.size(); ++i )
{ {
const bool from_stdin = ( filenames[i] == "-" ); const bool from_stdin = filenames[i] == "-";
if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; } if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; }
const char * const input_filename = const char * const input_filename =
from_stdin ? "(stdin)" : filenames[i].c_str(); from_stdin ? "(stdin)" : filenames[i].c_str();
@ -75,6 +76,9 @@ int list_files( const std::vector< std::string > & filenames,
set_retval( retval, lzip_index.retval() ); set_retval( retval, lzip_index.retval() );
continue; continue;
} }
const bool multi_empty =
!from_stdin && !cl_opts.ignore_errors && lzip_index.multi_empty();
if( multi_empty ) set_retval( retval, 2 );
if( verbosity < 0 ) continue; if( verbosity < 0 ) continue;
const unsigned long long udata_size = lzip_index.udata_size(); const unsigned long long udata_size = lzip_index.udata_size();
const unsigned long long cdata_size = lzip_index.cdata_size(); const unsigned long long cdata_size = lzip_index.cdata_size();
@ -86,6 +90,8 @@ int list_files( const std::vector< std::string > & filenames,
if( verbosity >= 1 ) std::fputs( " dict memb trail ", stdout ); if( verbosity >= 1 ) std::fputs( " dict memb trail ", stdout );
std::fputs( " uncompressed compressed saved name\n", stdout ); std::fputs( " uncompressed compressed saved name\n", stdout );
} }
if( multi_empty )
{ std::fflush( stdout ); show_file_error( input_filename, empty_msg ); }
if( verbosity >= 1 ) if( verbosity >= 1 )
std::printf( "%s %5ld %6lld ", format_ds( lzip_index.dictionary_size() ), std::printf( "%s %5ld %6lld ", format_ds( lzip_index.dictionary_size() ),
members, lzip_index.file_size() - cdata_size ); members, lzip_index.file_size() - cdata_size );
@ -113,12 +119,16 @@ int list_files( const std::vector< std::string > & filenames,
first_post = true; // reprint heading after list of members first_post = true; // reprint heading after list of members
} }
std::fflush( stdout ); std::fflush( stdout );
if( std::ferror( stdout ) ) break;
} }
if( verbosity >= 0 && files > 1 ) if( verbosity >= 0 && files > 1 && !std::ferror( stdout ) )
{ {
if( verbosity >= 1 ) std::fputs( " ", stdout ); if( verbosity >= 1 ) std::fputs( " ", stdout );
list_line( total_uncomp, total_comp, "(totals)" ); list_line( total_uncomp, total_comp, "(totals)" );
std::fflush( stdout ); std::fflush( stdout );
} }
if( verbosity >= 0 && ( std::ferror( stdout ) || std::fclose( stdout ) != 0 ) )
{ show_file_error( "(stdout)", write_error_msg, errno );
set_retval( retval, 1 ); }
return retval; return retval;
} }

42
lunzcrash.cc
View file

@ -56,7 +56,7 @@ bool compare_member( const uint8_t * const mbuffer, const long msize,
{ {
MD5SUM md5sum; MD5SUM md5sum;
LZ_mtester mtester( mbuffer, msize, dictionary_size, -1, &md5sum ); LZ_mtester mtester( mbuffer, msize, dictionary_size, -1, &md5sum );
bool error = ( mtester.test_member() != 0 || !mtester.finished() ); bool error = mtester.test_member() != 0 || !mtester.finished();
if( !error ) if( !error )
{ {
md5_type new_digest; md5_type new_digest;
@ -103,22 +103,23 @@ long next_pct_pos( const Lzip_index & lzip_index, const long i, const int pct,
/* Test 1-bit errors in LZMA streams in file. /* Test 1-bit errors in LZMA streams in file.
Unless verbosity >= 1, print only the bytes with interesting results. */ Unless verbosity >= 1, print only the bytes with interesting results. */
int lunzcrash_bit( const char * const input_filename, int lunzcrash_bit( const std::string & input_filename,
const Cl_options & cl_opts ) const Cl_options & cl_opts )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
if( verbosity >= 2 ) printf( "Testing file '%s'\n", input_filename ); if( verbosity >= 2 ) printf( "Testing file '%s'\n", filename );
const long long cdata_size = lzip_index.cdata_size(); const long long cdata_size = lzip_index.cdata_size();
long positions = 0, decompressions = 0, successes = 0, failed_comparisons = 0; long positions = 0, decompressions = 0, successes = 0, failed_comparisons = 0;
int pct = ( cdata_size >= 1000 && isatty( STDERR_FILENO ) ) ? 0 : 100; int pct = (cdata_size >= 1000 && isatty( STDERR_FILENO )) ? 0 : 100;
for( long i = 0; i < lzip_index.members(); ++i ) for( long i = 0; i < lzip_index.members(); ++i )
{ {
const long long mpos = lzip_index.mblock( i ).pos(); const long long mpos = lzip_index.mblock( i ).pos();
@ -127,8 +128,8 @@ int lunzcrash_bit( const char * const input_filename,
if( !mbuffer ) return 1; if( !mbuffer ) return 1;
const unsigned dictionary_size = lzip_index.dictionary_size( i ); const unsigned dictionary_size = lzip_index.dictionary_size( i );
md5_type md5_orig; md5_type md5_orig;
if( !check_member( mbuffer, msize, dictionary_size, input_filename, if( !check_member( mbuffer, msize, dictionary_size, filename, md5_orig ) )
md5_orig ) ) return 2; return 2;
long pct_pos = next_pct_pos( lzip_index, i, pct ); long pct_pos = next_pct_pos( lzip_index, i, pct );
long pos = Lzip_header::size + 1, printed = 0; // last pos printed long pos = Lzip_header::size + 1, printed = 0; // last pos printed
const long end = msize - 20; const long end = msize - 20;
@ -224,22 +225,23 @@ int lunzcrash_bit( const char * const input_filename,
/* Test zeroed blocks of given size in LZMA streams in file. /* Test zeroed blocks of given size in LZMA streams in file.
Unless verbosity >= 1, print only the bytes with interesting results. */ Unless verbosity >= 1, print only the bytes with interesting results. */
int lunzcrash_block( const char * const input_filename, int lunzcrash_block( const std::string & input_filename,
const Cl_options & cl_opts, const int sector_size ) const Cl_options & cl_opts, const int sector_size )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
if( verbosity >= 2 ) printf( "Testing file '%s'\n", input_filename ); if( verbosity >= 2 ) printf( "Testing file '%s'\n", filename );
const long long cdata_size = lzip_index.cdata_size(); const long long cdata_size = lzip_index.cdata_size();
long decompressions = 0, successes = 0, failed_comparisons = 0; long decompressions = 0, successes = 0, failed_comparisons = 0;
int pct = ( cdata_size >= 1000 && isatty( STDERR_FILENO ) ) ? 0 : 100; int pct = (cdata_size >= 1000 && isatty( STDERR_FILENO )) ? 0 : 100;
uint8_t * const block = new uint8_t[sector_size]; uint8_t * const block = new uint8_t[sector_size];
for( long i = 0; i < lzip_index.members(); ++i ) for( long i = 0; i < lzip_index.members(); ++i )
{ {
@ -251,8 +253,8 @@ int lunzcrash_block( const char * const input_filename,
if( !mbuffer ) return 1; if( !mbuffer ) return 1;
const unsigned dictionary_size = lzip_index.dictionary_size( i ); const unsigned dictionary_size = lzip_index.dictionary_size( i );
md5_type md5_orig; md5_type md5_orig;
if( !check_member( mbuffer, msize, dictionary_size, input_filename, if( !check_member( mbuffer, msize, dictionary_size, filename, md5_orig ) )
md5_orig ) ) return 2; return 2;
long pct_pos = next_pct_pos( lzip_index, i, pct, sector_size ); long pct_pos = next_pct_pos( lzip_index, i, pct, sector_size );
long pos = Lzip_header::size + 1; long pos = Lzip_header::size + 1;
const long end = msize - sector_size - 20; const long end = msize - sector_size - 20;
@ -344,7 +346,7 @@ int md5sum_files( const std::vector< std::string > & filenames )
for( unsigned i = 0; i < filenames.size(); ++i ) for( unsigned i = 0; i < filenames.size(); ++i )
{ {
const bool from_stdin = ( filenames[i] == "-" ); const bool from_stdin = filenames[i] == "-";
if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; } if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; }
const char * const input_filename = filenames[i].c_str(); const char * const input_filename = filenames[i].c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
@ -359,7 +361,8 @@ int md5sum_files( const std::vector< std::string > & filenames )
while( true ) while( true )
{ {
const int len = readblock( infd, buffer, buffer_size ); const int len = readblock( infd, buffer, buffer_size );
if( len != buffer_size && errno ) throw Error( read_error_msg ); if( len != buffer_size && errno )
{ show_file_error( input_filename, read_error_msg, errno ); return 1; }
if( len > 0 ) md5sum.md5_update( buffer, len ); if( len > 0 ) md5sum.md5_update( buffer, len );
if( len < buffer_size ) break; if( len < buffer_size ) break;
} }
@ -367,10 +370,15 @@ int md5sum_files( const std::vector< std::string > & filenames )
if( close( infd ) != 0 ) if( close( infd ) != 0 )
{ show_file_error( input_filename, "Error closing input file", errno ); { show_file_error( input_filename, "Error closing input file", errno );
return 1; } return 1; }
if( verbosity < 0 ) continue;
for( int i = 0; i < 16; ++i ) std::printf( "%02x", md5_digest[i] ); for( int i = 0; i < 16; ++i ) std::printf( "%02x", md5_digest[i] );
std::printf( " %s\n", input_filename ); std::printf( " %s\n", input_filename );
std::fflush( stdout ); std::fflush( stdout );
if( std::ferror( stdout ) ) break;
} }
if( verbosity >= 0 && ( std::ferror( stdout ) || std::fclose( stdout ) != 0 ) )
{ show_file_error( "(stdout)", write_error_msg, errno );
set_retval( retval, 1 ); }
return retval; return retval;
} }

42
lzip.h
View file

@ -321,14 +321,11 @@ struct Lzip_trailer
struct Cl_options // command-line options struct Cl_options // command-line options
{ {
bool ignore_empty;
bool ignore_errors; bool ignore_errors;
bool ignore_nonzero;
bool ignore_trailing; bool ignore_trailing;
bool loose_trailing; bool loose_trailing;
Cl_options() Cl_options() : ignore_errors( false ),
: ignore_empty( false ), ignore_errors( false ), ignore_nonzero( false ),
ignore_trailing( true ), loose_trailing( false ) {} ignore_trailing( true ), loose_trailing( false ) {}
}; };
@ -420,34 +417,44 @@ inline unsigned long long positive_diff( const unsigned long long x,
inline void set_retval( int & retval, const int new_val ) inline void set_retval( int & retval, const int new_val )
{ if( retval < new_val ) retval = new_val; } { if( retval < new_val ) retval = new_val; }
inline const char * printable_name( const std::string & filename,
const bool in = true )
{
if( filename.empty() || filename == "-" ) return in ? "(stdin)" : "(stdout)";
return filename.c_str();
}
const char * const bad_magic_msg = "Bad magic number (file not in lzip format)."; const char * const bad_magic_msg = "Bad magic number (file not in lzip format).";
const char * const bad_dict_msg = "Invalid dictionary size in member header."; const char * const bad_dict_msg = "Invalid dictionary size in member header.";
const char * const corrupt_mm_msg = "Corrupt header in multimember file."; const char * const corrupt_mm_msg = "Corrupt header in multimember file.";
const char * const empty_msg = "Empty member not allowed."; const char * const empty_msg = "Empty member not allowed.";
const char * const nonzero_msg = "Nonzero first LZMA byte.";
const char * const trailing_msg = "Trailing data not allowed.";
const char * const mmap_msg = "Can't mmap"; const char * const mmap_msg = "Can't mmap";
const char * const nonzero_msg = "Nonzero first LZMA byte.";
const char * const short_file_msg = "Input file is too short."; const char * const short_file_msg = "Input file is too short.";
const char * const trailing_msg = "Trailing data not allowed.";
const char * const write_error_msg = "Write error";
// defined in alone_to_lz.cc // defined in alone_to_lz.cc
int alone_to_lz( const int infd, const Pretty_print & pp ); int alone_to_lz( const int infd, const Pretty_print & pp );
// defined in byte_repair.cc // defined in byte_repair.cc
bool safe_seek( const int fd, const long long pos,
const std::string & filename );
long seek_write( const int fd, const uint8_t * const buf, const long size, long seek_write( const int fd, const uint8_t * const buf, const long size,
const long long pos ); const long long pos );
uint8_t * read_member( const int infd, const long long mpos, uint8_t * read_member( const int infd, const long long mpos,
const long long msize, const char * const filename ); const long long msize, const std::string & filename );
int byte_repair( const std::string & input_filename, int byte_repair( const std::string & input_filename,
const std::string & default_output_filename, const std::string & default_output_filename,
const Cl_options & cl_opts, const Cl_options & cl_opts,
const char terminator, const bool force ); const char terminator, const bool force );
int debug_delay( const char * const input_filename, int debug_delay( const std::string & input_filename,
const Cl_options & cl_opts, Block range, const Cl_options & cl_opts, Block range,
const char terminator ); const char terminator );
int debug_byte_repair( const char * const input_filename, int debug_byte_repair( const std::string & input_filename,
const Cl_options & cl_opts, const Bad_byte & bad_byte, const Cl_options & cl_opts, const Bad_byte & bad_byte,
const char terminator ); const char terminator );
int debug_decompress( const char * const input_filename, int debug_decompress( const std::string & input_filename,
const Cl_options & cl_opts, const Bad_byte & bad_byte, const Cl_options & cl_opts, const Bad_byte & bad_byte,
const bool show_packets ); const bool show_packets );
@ -470,9 +477,9 @@ int list_files( const std::vector< std::string > & filenames,
const Cl_options & cl_opts ); const Cl_options & cl_opts );
// defined in lunzcrash.cc // defined in lunzcrash.cc
int lunzcrash_bit( const char * const input_filename, int lunzcrash_bit( const std::string & input_filename,
const Cl_options & cl_opts ); const Cl_options & cl_opts );
int lunzcrash_block( const char * const input_filename, int lunzcrash_block( const std::string & input_filename,
const Cl_options & cl_opts, const int sector_size ); const Cl_options & cl_opts, const int sector_size );
int md5sum_files( const std::vector< std::string > & filenames ); int md5sum_files( const std::vector< std::string > & filenames );
@ -509,10 +516,11 @@ void show_dprogress( const unsigned long long cfile_size = 0,
const Pretty_print * const p = 0 ); const Pretty_print * const p = 0 );
// defined in merge.cc // defined in merge.cc
bool copy_file( const int infd, const int outfd, bool copy_file( const int infd, const int outfd, const std::string & iname,
const long long max_size = -1 ); const std::string & oname, const long long max_size = -1 );
int test_member_from_file( const int infd, const unsigned long long msize, int test_member_from_file( const int infd, const unsigned long long msize,
long long * const failure_posp = 0 ); long long * const failure_posp = 0,
bool * const nonzerop = 0 );
int merge_files( const std::vector< std::string > & filenames, int merge_files( const std::vector< std::string > & filenames,
const std::string & default_output_filename, const std::string & default_output_filename,
const Cl_options & cl_opts, const char terminator, const Cl_options & cl_opts, const char terminator,
@ -526,8 +534,6 @@ int print_nrep_stats( const std::vector< std::string > & filenames,
const char * format_num( unsigned long long num, const char * format_num( unsigned long long num,
unsigned long long limit = -1ULL, unsigned long long limit = -1ULL,
const int set_prefix = 0 ); const int set_prefix = 0 );
bool safe_seek( const int fd, const long long pos,
const char * const filename );
int range_decompress( const std::string & input_filename, int range_decompress( const std::string & input_filename,
const std::string & default_output_filename, const std::string & default_output_filename,
const Cl_options & cl_opts, Block range, const Cl_options & cl_opts, Block range,
@ -540,7 +546,7 @@ int reproduce_file( const std::string & input_filename,
const char * const reference_filename, const char * const reference_filename,
const Cl_options & cl_opts, const int lzip_level, const Cl_options & cl_opts, const int lzip_level,
const char terminator, const bool force ); const char terminator, const bool force );
int debug_reproduce_file( const char * const input_filename, int debug_reproduce_file( const std::string & input_filename,
const char * const lzip_name, const char * const lzip_name,
const char * const reference_filename, const char * const reference_filename,
const Cl_options & cl_opts, const Block & range, const Cl_options & cl_opts, const Block & range,

16
lzip_index.cc
View file

@ -148,13 +148,12 @@ bool Lzip_index::skip_gap( const int fd, unsigned long long & pos,
if( !cl_opts.ignore_trailing ) if( !cl_opts.ignore_trailing )
{ error_ = trailing_msg; retval_ = 2; return false; } { error_ = trailing_msg; retval_ = 2; return false; }
} }
const unsigned long long data_size = trailer.data_size();
pos = ipos + i - member_size; // good member pos = ipos + i - member_size; // good member
const unsigned dictionary_size = header.dictionary_size(); const unsigned dictionary_size = header.dictionary_size();
if( dictionary_size_ < dictionary_size ) if( dictionary_size_ < dictionary_size )
dictionary_size_ = dictionary_size; dictionary_size_ = dictionary_size;
member_vector.push_back( Member( 0, data_size, pos, member_size, member_vector.push_back( Member( 0, trailer.data_size(), pos,
dictionary_size ) ); member_size, dictionary_size ) );
return true; return true;
} }
if( ipos == 0 ) if( ipos == 0 )
@ -197,7 +196,7 @@ Lzip_index::Lzip_index( const int infd, const Cl_options & cl_opts,
retval_ = 2; return; } retval_ = 2; return; }
// pos always points to a header or to ( EOF || max_pos ) // pos always points to a header or to ( EOF || max_pos )
unsigned long long pos = ( max_pos > 0 ) ? max_pos : insize; unsigned long long pos = (max_pos > 0) ? max_pos : insize;
while( pos >= min_member_size ) while( pos >= min_member_size )
{ {
Lzip_trailer trailer; Lzip_trailer trailer;
@ -221,13 +220,12 @@ Lzip_index::Lzip_index( const int infd, const Cl_options & cl_opts,
continue; else return; } continue; else return; }
set_num_error( "Bad header at pos ", pos - member_size ); break; set_num_error( "Bad header at pos ", pos - member_size ); break;
} }
const unsigned long long data_size = trailer.data_size();
pos -= member_size; // good member pos -= member_size; // good member
const unsigned dictionary_size = header.dictionary_size(); const unsigned dictionary_size = header.dictionary_size();
if( dictionary_size_ < dictionary_size ) if( dictionary_size_ < dictionary_size )
dictionary_size_ = dictionary_size; dictionary_size_ = dictionary_size;
member_vector.push_back( Member( 0, data_size, pos, member_size, member_vector.push_back( Member( 0, trailer.data_size(), pos,
dictionary_size ) ); member_size, dictionary_size ) );
} }
// block at pos == 0 must be a member unless shorter than min_member_size // block at pos == 0 must be a member unless shorter than min_member_size
if( pos >= min_member_size || ( pos != 0 && !ignore_gaps ) || if( pos >= min_member_size || ( pos != 0 && !ignore_gaps ) ||
@ -237,10 +235,6 @@ Lzip_index::Lzip_index( const int infd, const Cl_options & cl_opts,
if( retval_ == 0 ) { error_ = "Can't create file index."; retval_ = 2; } if( retval_ == 0 ) { error_ = "Can't create file index."; retval_ = 2; }
return; return;
} }
if( !cl_opts.ignore_empty && member_vector.size() > 1 )
for( unsigned long i = 0; i < member_vector.size(); ++i )
if( member_vector[i].dblock.size() == 0 )
{ member_vector.clear(); error_ = empty_msg; retval_ = 2; return; }
std::reverse( member_vector.begin(), member_vector.end() ); std::reverse( member_vector.begin(), member_vector.end() );
for( unsigned long i = 0; ; ++i ) for( unsigned long i = 0; ; ++i )
{ {

8
lzip_index.h
View file

@ -64,6 +64,14 @@ public:
int retval() const { return retval_; } int retval() const { return retval_; }
unsigned dictionary_size() const { return dictionary_size_; } unsigned dictionary_size() const { return dictionary_size_; }
bool multi_empty() const // multimember file with empty member(s)
{
if( member_vector.size() > 1 )
for( unsigned long i = 0; i < member_vector.size(); ++i )
if( member_vector[i].dblock.size() == 0 ) return true;
return false;
}
bool operator==( const Lzip_index & li ) const bool operator==( const Lzip_index & li ) const
{ {
if( retval_ || li.retval_ || insize != li.insize || if( retval_ || li.retval_ || insize != li.insize ||

85
main.cc
View file

@ -36,6 +36,7 @@
#include <string> #include <string>
#include <vector> #include <vector>
#include <fcntl.h> #include <fcntl.h>
#include <pthread.h> // pthread_t
#include <stdint.h> // SIZE_MAX #include <stdint.h> // SIZE_MAX
#include <unistd.h> #include <unistd.h>
#include <utime.h> #include <utime.h>
@ -104,8 +105,8 @@ enum Mode { m_none, m_alone_to_lz, m_byte_repair, m_check, m_debug_byte_repair,
m_nrep_stats, m_range_dec, m_remove, m_reproduce, m_show_packets, m_nrep_stats, m_range_dec, m_remove, m_reproduce, m_show_packets,
m_split, m_strip, m_test, m_unzcrash_bit, m_unzcrash_block }; m_split, m_strip, m_test, m_unzcrash_bit, m_unzcrash_block };
/* Variable used in signal handler context. /* Variables used in signal handler context.
It is not declared volatile because the handler never returns. */ They are not declared volatile because the handler never returns. */
bool delete_output_on_interrupt = false; bool delete_output_on_interrupt = false;
@ -156,8 +157,6 @@ void show_help( const long num_online )
" --dump=<list>:d:e:t dump members, damaged/empty, tdata to stdout\n" " --dump=<list>:d:e:t dump members, damaged/empty, tdata to stdout\n"
" --remove=<list>:d:e:t remove members, tdata from files in place\n" " --remove=<list>:d:e:t remove members, tdata from files in place\n"
" --strip=<list>:d:e:t copy files to stdout stripping members given\n" " --strip=<list>:d:e:t copy files to stdout stripping members given\n"
" --ignore-empty ignore empty members in multimember files\n"
" --ignore-nonzero ignore a nonzero first LZMA byte\n"
" --loose-trailing allow trailing data seeming corrupt header\n" " --loose-trailing allow trailing data seeming corrupt header\n"
" --nonzero-repair repair in place a nonzero first LZMA byte\n", " --nonzero-repair repair in place a nonzero first LZMA byte\n",
num_online ); num_online );
@ -228,7 +227,7 @@ const char * format_ds( const unsigned dictionary_size )
const char * p = ""; const char * p = "";
const char * np = " "; const char * np = " ";
unsigned num = dictionary_size; unsigned num = dictionary_size;
bool exact = ( num % factor == 0 ); bool exact = num % factor == 0;
for( int i = 0; i < n && ( num > 9999 || ( exact && num >= factor ) ); ++i ) for( int i = 0; i < n && ( num > 9999 || ( exact && num >= factor ) ); ++i )
{ num /= factor; if( num % factor != 0 ) exact = false; { num /= factor; if( num % factor != 0 ) exact = false;
@ -263,12 +262,12 @@ void Member_list::parse_ml( const char * const arg,
if( len <= 7 && std::strncmp( "damaged", p, len ) == 0 ) if( len <= 7 && std::strncmp( "damaged", p, len ) == 0 )
{ damaged = true; cl_opts.ignore_errors = true; goto next; } { damaged = true; cl_opts.ignore_errors = true; goto next; }
if( len <= 5 && std::strncmp( "empty", p, len ) == 0 ) if( len <= 5 && std::strncmp( "empty", p, len ) == 0 )
{ empty = true; cl_opts.ignore_empty = true; goto next; } { empty = true; goto next; }
if( len <= 5 && std::strncmp( "tdata", p, len ) == 0 ) if( len <= 5 && std::strncmp( "tdata", p, len ) == 0 )
{ tdata = true; cl_opts.ignore_trailing = true; goto next; } { tdata = true; cl_opts.ignore_trailing = true; goto next; }
} }
{ {
const bool reverse = ( *p == 'r' ); const bool reverse = *p == 'r';
if( reverse ) ++p; if( reverse ) ++p;
if( *p == '^' ) { ++p; if( reverse ) rin = false; else in = false; } if( *p == '^' ) { ++p; if( reverse ) rin = false; else in = false; }
std::vector< Block > * rvp = reverse ? &rrange_vector : &range_vector; std::vector< Block > * rvp = reverse ? &rrange_vector : &range_vector;
@ -322,7 +321,7 @@ const char * parse_range( const char * const arg, const char * const pn,
range.pos( value ); range.pos( value );
if( tail[0] == 0 || tail[0] == ':' ) if( tail[0] == 0 || tail[0] == ':' )
{ range.size( INT64_MAX - value ); return tail; } { range.size( INT64_MAX - value ); return tail; }
const bool is_size = ( tail[0] == ',' ); const bool is_size = tail[0] == ',';
if( sector_sizep && tail[1] == ',' ) { value = INT64_MAX - value; ++tail; } if( sector_sizep && tail[1] == ',' ) { value = INT64_MAX - value; ++tail; }
else value = getnum( tail + 1, pn, 0, 1, INT64_MAX, &tail ); // size else value = getnum( tail + 1, pn, 0, 1, INT64_MAX, &tail ); // size
if( !is_size && value <= range.pos() ) if( !is_size && value <= range.pos() )
@ -390,6 +389,12 @@ void parse_range_vector( const char * const arg, const char * const pn,
} }
void no_to_stdout( const bool to_stdout )
{
if( to_stdout )
{ show_error( "'--stdout' not allowed." ); std::exit( 1 ); }
}
void one_file( const int files ) void one_file( const int files )
{ {
if( files != 1 ) if( files != 1 )
@ -562,9 +567,9 @@ int open_instream( const char * const name, struct stat * const in_statsp,
{ {
const int i = fstat( infd, in_statsp ); const int i = fstat( infd, in_statsp );
const mode_t mode = in_statsp->st_mode; const mode_t mode = in_statsp->st_mode;
const bool can_read = ( i == 0 && !reg_only && const bool can_read = i == 0 && !reg_only &&
( S_ISBLK( mode ) || S_ISCHR( mode ) || ( S_ISBLK( mode ) || S_ISCHR( mode ) ||
S_ISFIFO( mode ) || S_ISSOCK( mode ) ) ); S_ISFIFO( mode ) || S_ISSOCK( mode ) );
if( i != 0 || ( !S_ISREG( mode ) && ( !can_read || one_to_one ) ) ) if( i != 0 || ( !S_ISREG( mode ) && ( !can_read || one_to_one ) ) )
{ {
if( verbosity >= 0 ) if( verbosity >= 0 )
@ -677,8 +682,8 @@ void set_signals( void (*action)(int) )
void cleanup_and_fail( const int retval ) void cleanup_and_fail( const int retval )
{ {
cleanup_mutex_lock(); // only one thread can delete and exit
set_signals( SIG_IGN ); // ignore signals set_signals( SIG_IGN ); // ignore signals
cleanup_mutex_lock(); // only one thread can delete and exit
if( delete_output_on_interrupt ) if( delete_output_on_interrupt )
{ {
delete_output_on_interrupt = false; delete_output_on_interrupt = false;
@ -795,12 +800,12 @@ bool show_trailing_data( const uint8_t * const data, const int size,
int decompress( const unsigned long long cfile_size, const int infd, int decompress( const unsigned long long cfile_size, const int infd,
const Cl_options & cl_opts, const Pretty_print & pp, const Cl_options & cl_opts, const Pretty_print & pp,
const bool testing ) const bool from_stdin, const bool testing )
{ {
unsigned long long partial_file_pos = 0; unsigned long long partial_file_pos = 0;
Range_decoder rdec( infd ); Range_decoder rdec( infd );
int retval = 0; int retval = 0;
bool empty = false, nonempty = false; bool empty = false, multi = false;
for( bool first_member = true; ; first_member = false ) for( bool first_member = true; ; first_member = false )
{ {
@ -843,11 +848,10 @@ int decompress( const unsigned long long cfile_size, const int infd,
LZ_decoder decoder( rdec, dictionary_size, outfd ); LZ_decoder decoder( rdec, dictionary_size, outfd );
show_dprogress( cfile_size, partial_file_pos, &rdec, &pp ); // init show_dprogress( cfile_size, partial_file_pos, &rdec, &pp ); // init
const int result = decoder.decode_member( pp, cl_opts.ignore_nonzero ); const int result = decoder.decode_member( pp, cl_opts.ignore_errors );
partial_file_pos += rdec.member_position(); partial_file_pos += rdec.member_position();
if( result != 0 ) if( result != 0 )
{ {
retval = 2;
if( verbosity >= 0 && result <= 2 ) if( verbosity >= 0 && result <= 2 )
{ {
pp(); pp();
@ -855,19 +859,20 @@ int decompress( const unsigned long long cfile_size, const int infd,
"File ends unexpectedly" : "Decoder error", "File ends unexpectedly" : "Decoder error",
partial_file_pos ); partial_file_pos );
} }
else if( result == 5 ) { pp( nonzero_msg ); break; } else if( result == 5 ) pp( nonzero_msg );
retval = 2;
if( cl_opts.ignore_errors ) { pp.reset(); continue; } else break; if( cl_opts.ignore_errors ) { pp.reset(); continue; } else break;
} }
if( !cl_opts.ignore_empty ) if( !from_stdin && !cl_opts.ignore_errors ) { multi = !first_member;
{ if( decoder.data_position() == 0 ) empty = true; else nonempty = true; } if( decoder.data_position() == 0 ) empty = true; }
if( verbosity >= 2 ) if( verbosity >= 2 )
{ std::fputs( testing ? "ok\n" : "done\n", stderr ); pp.reset(); } { std::fputs( testing ? "ok\n" : "done\n", stderr ); pp.reset(); }
} }
if( verbosity == 1 && retval == 0 ) if( verbosity == 1 && retval == 0 )
std::fputs( testing ? "ok\n" : "done\n", stderr ); std::fputs( testing ? "ok\n" : "done\n", stderr );
if( retval == 2 && cl_opts.ignore_errors ) retval = 0; if( empty && multi && retval == 0 )
if( empty && nonempty && retval == 0 )
{ show_file_error( pp.name(), empty_msg ); retval = 2; } { show_file_error( pp.name(), empty_msg ); retval = 2; }
if( retval == 2 && cl_opts.ignore_errors ) retval = 0;
return retval; return retval;
} }
@ -972,8 +977,8 @@ int main( const int argc, const char * const argv[] )
bool to_stdout = false; bool to_stdout = false;
if( argc > 0 ) invocation_name = argv[0]; if( argc > 0 ) invocation_name = argv[0];
enum { opt_chk = 256, opt_dbg, opt_du, opt_ff, opt_g16, opt_ie, opt_inz, enum { opt_chk = 256, opt_dbg, opt_du, opt_ff, opt_g16, opt_lt,
opt_lt, opt_lzl, opt_lzn, opt_nzr, opt_ref, opt_rem, opt_rnd, opt_st }; opt_lzl, opt_lzn, opt_nzr, opt_ref, opt_rem, opt_rnd, opt_st };
const Arg_parser::Option options[] = const Arg_parser::Option options[] =
{ {
{ '0', 0, Arg_parser::no }, { '0', 0, Arg_parser::no },
@ -1024,8 +1029,6 @@ int main( const int argc, const char * const argv[] )
{ opt_du, "dump", Arg_parser::yes }, { opt_du, "dump", Arg_parser::yes },
{ opt_ff, "fec-file", Arg_parser::yes }, { opt_ff, "fec-file", Arg_parser::yes },
{ opt_g16, "gf16", Arg_parser::no }, { opt_g16, "gf16", Arg_parser::no },
{ opt_ie, "ignore-empty", Arg_parser::no },
{ opt_inz, "ignore-nonzero", Arg_parser::no },
{ opt_lt, "loose-trailing", Arg_parser::no }, { opt_lt, "loose-trailing", Arg_parser::no },
{ opt_lzl, "lzip-level", Arg_parser::yes }, { opt_lzl, "lzip-level", Arg_parser::yes },
{ opt_lzn, "lzip-name", Arg_parser::yes }, { opt_lzn, "lzip-name", Arg_parser::yes },
@ -1106,8 +1109,6 @@ int main( const int argc, const char * const argv[] )
member_list.parse_ml( arg, pn, cl_opts ); break; member_list.parse_ml( arg, pn, cl_opts ); break;
case opt_ff: cl_fec_filename = sarg; break; case opt_ff: cl_fec_filename = sarg; break;
case opt_g16: cl_gf16 = true; break; case opt_g16: cl_gf16 = true; break;
case opt_ie: cl_opts.ignore_empty = true; break;
case opt_inz: cl_opts.ignore_nonzero = true; break;
case opt_lt: cl_opts.loose_trailing = true; break; case opt_lt: cl_opts.loose_trailing = true; break;
case opt_lzl: lzip_level = parse_lzip_level( arg, pn ); break; case opt_lzl: lzip_level = parse_lzip_level( arg, pn ); break;
case opt_lzn: lzip_name = arg; break; case opt_lzn: lzip_name = arg; break;
@ -1132,9 +1133,6 @@ int main( const int argc, const char * const argv[] )
show_error( "You must specify the operation to be performed.", 0, true ); show_error( "You must specify the operation to be performed.", 0, true );
return 1; return 1;
} }
if( program_mode != m_decompress && program_mode != m_list &&
program_mode != m_test && program_mode != m_range_dec )
cl_opts.ignore_empty = true;
std::vector< std::string > filenames; std::vector< std::string > filenames;
bool filenames_given = false; bool filenames_given = false;
@ -1151,20 +1149,19 @@ int main( const int argc, const char * const argv[] )
case m_none: internal_error( "invalid operation." ); break; case m_none: internal_error( "invalid operation." ); break;
case m_alone_to_lz: break; case m_alone_to_lz: break;
case m_byte_repair: case m_byte_repair:
one_file( filenames.size() ); one_file( filenames.size() ); no_to_stdout( to_stdout );
return byte_repair( filenames[0], default_output_filename, cl_opts, return byte_repair( filenames[0], default_output_filename, cl_opts,
terminator, force ); terminator, force );
case m_check: return gf_check( cblocks, cl_gf16, fec_random ); case m_check: return gf_check( cblocks, cl_gf16, fec_random );
case m_debug_byte_repair: case m_debug_byte_repair:
one_file( filenames.size() ); one_file( filenames.size() );
return debug_byte_repair( filenames[0].c_str(), cl_opts, bad_byte, return debug_byte_repair( filenames[0], cl_opts, bad_byte, terminator );
terminator );
case m_debug_decompress: case m_debug_decompress:
one_file( filenames.size() ); one_file( filenames.size() );
return debug_decompress( filenames[0].c_str(), cl_opts, bad_byte, false ); return debug_decompress( filenames[0], cl_opts, bad_byte, false );
case m_debug_delay: case m_debug_delay:
one_file( filenames.size() ); one_file( filenames.size() );
return debug_delay( filenames[0].c_str(), cl_opts, range, terminator ); return debug_delay( filenames[0], cl_opts, range, terminator );
case m_decompress: break; case m_decompress: break;
case m_dump: case m_dump:
case m_strip: case m_strip:
@ -1197,7 +1194,7 @@ int main( const int argc, const char * const argv[] )
return fec_dZ( filenames[0], cl_fec_filename, delta, sector_size ); return fec_dZ( filenames[0], cl_fec_filename, delta, sector_size );
case m_list: break; case m_list: break;
case m_md5sum: break; case m_md5sum: break;
case m_merge: case m_merge: no_to_stdout( to_stdout );
if( filenames.size() < 2 ) if( filenames.size() < 2 )
{ show_error( "You must specify at least 2 files.", 0, true ); return 1; } { show_error( "You must specify at least 2 files.", 0, true ); return 1; }
return merge_files( filenames, default_output_filename, cl_opts, return merge_files( filenames, default_output_filename, cl_opts,
@ -1215,28 +1212,28 @@ int main( const int argc, const char * const argv[] )
at_least_one_file( filenames.size() ); at_least_one_file( filenames.size() );
return remove_members( filenames, cl_opts, member_list ); return remove_members( filenames, cl_opts, member_list );
case m_reproduce: case m_reproduce:
one_file( filenames.size() ); one_file( filenames.size() ); no_to_stdout( to_stdout );
if( !reference_filename || !reference_filename[0] ) if( !reference_filename || !reference_filename[0] )
{ show_error( "You must specify a reference file.", 0, true ); return 1; } { show_error( "You must specify a reference file.", 0, true ); return 1; }
if( range.size() > 0 ) if( range.size() > 0 )
return debug_reproduce_file( filenames[0].c_str(), lzip_name, return debug_reproduce_file( filenames[0], lzip_name,
reference_filename, cl_opts, range, sector_size, lzip_level ); reference_filename, cl_opts, range, sector_size, lzip_level );
else else
return reproduce_file( filenames[0], default_output_filename, lzip_name, return reproduce_file( filenames[0], default_output_filename, lzip_name,
reference_filename, cl_opts, lzip_level, terminator, force ); reference_filename, cl_opts, lzip_level, terminator, force );
case m_show_packets: case m_show_packets:
one_file( filenames.size() ); one_file( filenames.size() );
return debug_decompress( filenames[0].c_str(), cl_opts, bad_byte, true ); return debug_decompress( filenames[0], cl_opts, bad_byte, true );
case m_split: case m_split:
one_file( filenames.size() ); one_file( filenames.size() ); no_to_stdout( to_stdout );
return split_file( filenames[0], default_output_filename, cl_opts, force ); return split_file( filenames[0], default_output_filename, cl_opts, force );
case m_test: break; case m_test: break;
case m_unzcrash_bit: case m_unzcrash_bit:
one_file( filenames.size() ); one_file( filenames.size() );
return lunzcrash_bit( filenames[0].c_str(), cl_opts ); return lunzcrash_bit( filenames[0], cl_opts );
case m_unzcrash_block: case m_unzcrash_block:
one_file( filenames.size() ); one_file( filenames.size() );
return lunzcrash_block( filenames[0].c_str(), cl_opts, sector_size ); return lunzcrash_block( filenames[0], cl_opts, sector_size );
} }
} }
catch( std::bad_alloc & ) { show_error( mem_msg ); cleanup_and_fail( 1 ); } catch( std::bad_alloc & ) { show_error( mem_msg ); cleanup_and_fail( 1 ); }
@ -1274,9 +1271,10 @@ int main( const int argc, const char * const argv[] )
{ {
std::string input_filename; std::string input_filename;
int infd; int infd;
const bool from_stdin = filenames[i] == "-";
pp.set_name( filenames[i] ); pp.set_name( filenames[i] );
if( filenames[i] == "-" ) if( from_stdin )
{ {
if( stdin_used ) continue; else stdin_used = true; if( stdin_used ) continue; else stdin_used = true;
infd = STDIN_FILENO; infd = STDIN_FILENO;
@ -1318,7 +1316,8 @@ int main( const int argc, const char * const argv[] )
if( program_mode == m_alone_to_lz ) if( program_mode == m_alone_to_lz )
tmp = alone_to_lz( infd, pp ); tmp = alone_to_lz( infd, pp );
else else
tmp = decompress( cfile_size, infd, cl_opts, pp, program_mode == m_test ); tmp = decompress( cfile_size, infd, cl_opts, pp, from_stdin,
program_mode == m_test );
} }
catch( std::bad_alloc & ) { pp( mem_msg ); tmp = 1; } catch( std::bad_alloc & ) { pp( mem_msg ); tmp = 1; }
catch( Error & e ) { pp(); show_error( e.msg, errno ); tmp = 1; } catch( Error & e ) { pp(); show_error( e.msg, errno ); tmp = 1; }

86
merge.cc
View file

@ -140,25 +140,23 @@ bool diff_member( const long long mpos, const long long msize,
continue; continue;
std::vector< Block > bv; std::vector< Block > bv;
long long partial_pos = 0; long long partial_pos = 0;
const char * const filename1 = filenames[i1].c_str();
const char * const filename2 = filenames[i2].c_str();
const int fd1 = infd_vector[i1], fd2 = infd_vector[i2]; const int fd1 = infd_vector[i1], fd2 = infd_vector[i2];
int begin = -1; // begin of block. -1 means no block int begin = -1; // begin of block. -1 means no block
bool prev_equal = true; bool prev_equal = true;
if( !safe_seek( fd1, mpos, filename1 ) || if( !safe_seek( fd1, mpos, filenames[i1] ) ||
!safe_seek( fd2, mpos, filename2 ) ) { error = true; break; } !safe_seek( fd2, mpos, filenames[i2] ) ) { error = true; break; }
while( partial_pos < msize ) while( partial_pos < msize )
{ {
const int size = std::min( (long long)buffer_size, msize - partial_pos ); const int size = std::min( (long long)buffer_size, msize - partial_pos );
const int rd = readblock( fd1, buffer1, size ); const int rd = readblock( fd1, buffer1, size );
if( rd != size && errno ) if( rd != size && errno )
{ show_file_error( filename1, read_error_msg, errno ); { show_file_error( filenames[i1].c_str(), read_error_msg, errno );
error = true; break; } error = true; break; }
if( rd > 0 ) if( rd > 0 )
{ {
if( readblock( fd2, buffer2, rd ) != rd ) if( readblock( fd2, buffer2, rd ) != rd )
{ show_file_error( filename2, read_error_msg, errno ); { show_file_error( filenames[i2].c_str(), read_error_msg, errno );
error = true; break; } error = true; break; }
for( int i = 0; i < rd; ++i ) for( int i = 0; i < rd; ++i )
{ {
@ -285,21 +283,20 @@ int open_input_files( const std::vector< std::string > & filenames,
for( int i = 0; i < files; ++i ) for( int i = 0; i < files; ++i )
{ {
const char * const filename = filenames[i].c_str();
const int infd = infd_vector[i]; const int infd = infd_vector[i];
bool error = false; bool error = false;
for( long j = 0; j < lzip_index.members(); ++j ) for( long j = 0; j < lzip_index.members(); ++j )
{ {
const long long mpos = lzip_index.mblock( j ).pos(); const long long mpos = lzip_index.mblock( j ).pos();
const long long msize = lzip_index.mblock( j ).size(); const long long msize = lzip_index.mblock( j ).size();
if( !safe_seek( infd, mpos, filename ) ) return 1; if( !safe_seek( infd, mpos, filenames[i] ) ) return 1;
if( test_member_from_file( infd, msize ) != 0 ) { error = true; break; } if( test_member_from_file( infd, msize ) != 0 ) { error = true; break; }
} }
if( !error ) if( !error )
{ {
if( verbosity >= 1 ) if( verbosity >= 1 )
std::printf( "Input file '%s' has no errors. Recovery is not needed.\n", std::printf( "Input file '%s' has no errors. Recovery is not needed.\n",
filename ); filenames[i].c_str() );
return 0; return 0;
} }
} }
@ -365,10 +362,10 @@ bool try_merge_member2( const std::vector< std::string > & filenames,
if( i1 == i2 || color_vector[i1] == color_vector[i2] || if( i1 == i2 || color_vector[i1] == color_vector[i2] ||
color_done( color_vector, i1 ) ) continue; color_done( color_vector, i1 ) ) continue;
for( int bi = 0; bi < blocks; ++bi ) for( int bi = 0; bi < blocks; ++bi )
if( !safe_seek( infd_vector[i2], block_vector[bi].pos(), filenames[i2].c_str() ) || if( !safe_seek( infd_vector[i2], block_vector[bi].pos(), filenames[i2] ) ||
!safe_seek( outfd, block_vector[bi].pos(), output_filename.c_str() ) || !safe_seek( outfd, block_vector[bi].pos(), output_filename ) ||
!copy_file( infd_vector[i2], outfd, block_vector[bi].size() ) ) !copy_file( infd_vector[i2], outfd, filenames[i2], output_filename,
cleanup_and_fail( 1 ); block_vector[bi].size() ) ) cleanup_and_fail( 1 );
const int infd = infd_vector[i1]; const int infd = infd_vector[i1];
const int var = ( i1 * ( files - 1 ) ) + i2 - ( i2 > i1 ) + 1; const int var = ( i1 * ( files - 1 ) ) + i2 - ( i2 > i1 ) + 1;
for( int bi = 0; bi + 1 < blocks; ++bi ) for( int bi = 0; bi + 1 < blocks; ++bi )
@ -379,10 +376,11 @@ bool try_merge_member2( const std::vector< std::string > & filenames,
var, variations, bi + 1, terminator ); var, variations, bi + 1, terminator );
std::fflush( stdout ); pending_newline = true; std::fflush( stdout ); pending_newline = true;
} }
if( !safe_seek( infd, block_vector[bi].pos(), filenames[i1].c_str() ) || if( !safe_seek( infd, block_vector[bi].pos(), filenames[i1] ) ||
!safe_seek( outfd, block_vector[bi].pos(), output_filename.c_str() ) || !safe_seek( outfd, block_vector[bi].pos(), output_filename ) ||
!copy_file( infd, outfd, block_vector[bi].size() ) || !copy_file( infd, outfd, filenames[i1], output_filename,
!safe_seek( outfd, mpos, output_filename.c_str() ) ) block_vector[bi].size() ) ||
!safe_seek( outfd, mpos, output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 );
long long failure_pos = 0; long long failure_pos = 0;
if( test_member_from_file( outfd, msize, &failure_pos ) == 0 ) if( test_member_from_file( outfd, msize, &failure_pos ) == 0 )
@ -421,8 +419,7 @@ bool try_merge_member( const std::vector< std::string > & filenames,
if( verbosity >= 2 ) if( verbosity >= 2 )
{ {
long var = 0; long var = 0;
for( int i = 0; i < blocks; ++i ) for( int i = 0; i < blocks; ++i ) var = var * files + file_idx[i];
var = ( var * files ) + file_idx[i];
std::printf( " Trying variation %ld of %ld %c", std::printf( " Trying variation %ld of %ld %c",
var + 1, variations, terminator ); var + 1, variations, terminator );
std::fflush( stdout ); pending_newline = true; std::fflush( stdout ); pending_newline = true;
@ -430,14 +427,13 @@ bool try_merge_member( const std::vector< std::string > & filenames,
while( bi < blocks ) while( bi < blocks )
{ {
const int infd = infd_vector[file_idx[bi]]; const int infd = infd_vector[file_idx[bi]];
if( !safe_seek( infd, block_vector[bi].pos(), filenames[file_idx[bi]].c_str() ) || if( !safe_seek( infd, block_vector[bi].pos(), filenames[file_idx[bi]] ) ||
!safe_seek( outfd, block_vector[bi].pos(), output_filename.c_str() ) || !safe_seek( outfd, block_vector[bi].pos(), output_filename ) ||
!copy_file( infd, outfd, block_vector[bi].size() ) ) !copy_file( infd, outfd, filenames[file_idx[bi]], output_filename,
cleanup_and_fail( 1 ); block_vector[bi].size() ) ) cleanup_and_fail( 1 );
++bi; ++bi;
} }
if( !safe_seek( outfd, mpos, output_filename.c_str() ) ) if( !safe_seek( outfd, mpos, output_filename ) ) cleanup_and_fail( 1 );
cleanup_and_fail( 1 );
long long failure_pos = 0; long long failure_pos = 0;
if( test_member_from_file( outfd, msize, &failure_pos ) == 0 ) return true; if( test_member_from_file( outfd, msize, &failure_pos ) == 0 ) return true;
while( bi > 0 && mpos + failure_pos < block_vector[bi-1].pos() ) --bi; while( bi > 0 && mpos + failure_pos < block_vector[bi-1].pos() ) --bi;
@ -473,12 +469,12 @@ bool try_merge_member1( const std::vector< std::string > & filenames,
{ {
if( i1 == i2 || color_vector[i1] == color_vector[i2] || if( i1 == i2 || color_vector[i1] == color_vector[i2] ||
color_done( color_vector, i1 ) ) continue; color_done( color_vector, i1 ) ) continue;
if( !safe_seek( infd_vector[i1], pos, filenames[i1] ) ||
!safe_seek( infd_vector[i2], pos, filenames[i2] ) ||
!safe_seek( outfd, pos, output_filename ) ||
!copy_file( infd_vector[i2], outfd, filenames[i2], output_filename,
size ) ) cleanup_and_fail( 1 );
const int infd = infd_vector[i1]; const int infd = infd_vector[i1];
if( !safe_seek( infd, pos, filenames[i1].c_str() ) ||
!safe_seek( infd_vector[i2], pos, filenames[i2].c_str() ) ||
!safe_seek( outfd, pos, output_filename.c_str() ) ||
!copy_file( infd_vector[i2], outfd, size ) )
cleanup_and_fail( 1 );
const int var = ( i1 * ( files - 1 ) ) + i2 - ( i2 > i1 ) + 1; const int var = ( i1 * ( files - 1 ) ) + i2 - ( i2 > i1 ) + 1;
for( long long i = 0; i + 1 < size; ++i ) for( long long i = 0; i + 1 < size; ++i )
{ {
@ -488,10 +484,10 @@ bool try_merge_member1( const std::vector< std::string > & filenames,
var, variations, pos + i, terminator ); var, variations, pos + i, terminator );
std::fflush( stdout ); pending_newline = true; std::fflush( stdout ); pending_newline = true;
} }
if( !safe_seek( outfd, pos + i, output_filename.c_str() ) || if( !safe_seek( outfd, pos + i, output_filename ) ||
readblock( infd, &byte, 1 ) != 1 || readblock( infd, &byte, 1 ) != 1 ||
writeblock( outfd, &byte, 1 ) != 1 || writeblock( outfd, &byte, 1 ) != 1 ||
!safe_seek( outfd, mpos, output_filename.c_str() ) ) !safe_seek( outfd, mpos, output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 );
long long failure_pos = 0; long long failure_pos = 0;
if( test_member_from_file( outfd, msize, &failure_pos ) == 0 ) if( test_member_from_file( outfd, msize, &failure_pos ) == 0 )
@ -508,11 +504,12 @@ bool try_merge_member1( const std::vector< std::string > & filenames,
/* infd and outfd can refer to the same file if copying to a lower file /* infd and outfd can refer to the same file if copying to a lower file
position or if source and destination blocks don't overlap. position or if source and destination blocks don't overlap.
max_size < 0 means no size limit. */ max_size < 0 means no size limit. */
bool copy_file( const int infd, const int outfd, const long long max_size ) bool copy_file( const int infd, const int outfd, const std::string & iname,
const std::string & oname, const long long max_size )
{ {
const int buffer_size = 65536; const int buffer_size = 65536;
// remaining number of bytes to copy // remaining number of bytes to copy
long long rest = ( ( max_size >= 0 ) ? max_size : buffer_size ); long long rest = (max_size >= 0) ? max_size : buffer_size;
long long copied_size = 0; long long copied_size = 0;
uint8_t * const buffer = new uint8_t[buffer_size]; uint8_t * const buffer = new uint8_t[buffer_size];
bool error = false; bool error = false;
@ -523,20 +520,22 @@ bool copy_file( const int infd, const int outfd, const long long max_size )
if( max_size >= 0 ) rest -= size; if( max_size >= 0 ) rest -= size;
const int rd = readblock( infd, buffer, size ); const int rd = readblock( infd, buffer, size );
if( rd != size && errno ) if( rd != size && errno )
{ show_error( read_error_msg, errno ); error = true; break; } { show_file_error( printable_name( iname ), read_error_msg, errno );
error = true; break; }
if( rd > 0 ) if( rd > 0 )
{ {
const int wr = writeblock( outfd, buffer, rd ); const int wr = writeblock( outfd, buffer, rd );
if( wr != rd ) if( wr != rd )
{ show_error( "Error writing output file", errno ); { show_file_error( printable_name( oname, false ), write_error_msg,
error = true; break; } errno ); error = true; break; }
copied_size += rd; copied_size += rd;
} }
if( rd < size ) break; // EOF if( rd < size ) break; // EOF
} }
delete[] buffer; delete[] buffer;
if( !error && max_size >= 0 && copied_size != max_size ) if( !error && max_size >= 0 && copied_size != max_size )
{ show_error( "Input file ends unexpectedly." ); error = true; } { show_file_error( printable_name( iname ), "Input file ends unexpectedly." );
error = true; }
return !error; return !error;
} }
@ -544,7 +543,7 @@ bool copy_file( const int infd, const int outfd, const long long max_size )
/* Return value: 0 = OK, 1 = bad msize, 2 = data error. /* Return value: 0 = OK, 1 = bad msize, 2 = data error.
'failure_pos' is relative to the beginning of the member. */ 'failure_pos' is relative to the beginning of the member. */
int test_member_from_file( const int infd, const unsigned long long msize, int test_member_from_file( const int infd, const unsigned long long msize,
long long * const failure_posp ) long long * const failure_posp, bool * const nonzerop )
{ {
Range_decoder rdec( infd ); Range_decoder rdec( infd );
Lzip_header header; Lzip_header header;
@ -559,6 +558,7 @@ int test_member_from_file( const int infd, const unsigned long long msize,
verbosity = -1; // suppress all messages verbosity = -1; // suppress all messages
done = decoder.decode_member() == 0; done = decoder.decode_member() == 0;
verbosity = saved_verbosity; // restore verbosity level verbosity = saved_verbosity; // restore verbosity level
if( nonzerop ) *nonzerop = rdec.nonzero();
if( done && rdec.member_position() == msize ) return 0; if( done && rdec.member_position() == msize ) return 0;
} }
if( failure_posp ) *failure_posp = rdec.member_position(); if( failure_posp ) *failure_posp = rdec.member_position();
@ -578,15 +578,15 @@ int merge_files( const std::vector< std::string > & filenames,
const int retval = const int retval =
open_input_files( filenames, infd_vector, cl_opts, lzip_index, &in_stats ); open_input_files( filenames, infd_vector, cl_opts, lzip_index, &in_stats );
if( retval >= 0 ) return retval; if( retval >= 0 ) return retval;
if( !safe_seek( infd_vector[0], 0, filenames[0].c_str() ) ) return 1; if( !safe_seek( infd_vector[0], 0, filenames[0] ) ) return 1;
const bool to_file = default_output_filename.size(); const bool to_file = default_output_filename.size();
output_filename = output_filename =
to_file ? default_output_filename : insert_fixed( filenames[0] ); to_file ? default_output_filename : insert_fixed( filenames[0] );
set_signal_handler(); set_signal_handler();
if( !open_outstream( force, true, true, false, to_file ) ) return 1; if( !open_outstream( force, true, true, false, to_file ) ) return 1;
if( !copy_file( infd_vector[0], outfd ) ) // copy whole file if( !copy_file( infd_vector[0], outfd, filenames[0], output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 ); // copy whole file
for( long j = 0; j < lzip_index.members(); ++j ) for( long j = 0; j < lzip_index.members(); ++j )
{ {
@ -597,7 +597,7 @@ int merge_files( const std::vector< std::string > & filenames,
// different color means members are different // different color means members are different
std::vector< int > color_vector( files, 0 ); std::vector< int > color_vector( files, 0 );
if( !diff_member( mpos, msize, filenames, infd_vector, block_vector, if( !diff_member( mpos, msize, filenames, infd_vector, block_vector,
color_vector ) || !safe_seek( outfd, mpos, output_filename.c_str() ) ) color_vector ) || !safe_seek( outfd, mpos, output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 );
if( block_vector.empty() ) if( block_vector.empty() )

48
mtester.cc
View file

@ -83,7 +83,7 @@ void LZ_mtester::flush_data()
crc32.update_buf( crc_, buffer + stream_pos, size ); crc32.update_buf( crc_, buffer + stream_pos, size );
if( md5sum ) md5sum->md5_update( buffer + stream_pos, size ); if( md5sum ) md5sum->md5_update( buffer + stream_pos, size );
if( outfd >= 0 && writeblock( outfd, buffer + stream_pos, size ) != size ) if( outfd >= 0 && writeblock( outfd, buffer + stream_pos, size ) != size )
throw Error( "Write error" ); throw Error( write_error_msg );
if( pos >= dictionary_size ) if( pos >= dictionary_size )
{ partial_data_pos += pos; pos = 0; pos_wrapped = true; } { partial_data_pos += pos; pos = 0; pos_wrapped = true; }
stream_pos = pos; stream_pos = pos;
@ -148,7 +148,7 @@ int LZ_mtester::test_member( const unsigned long mpos_limit,
FILE * const f, const unsigned long long byte_pos ) FILE * const f, const unsigned long long byte_pos )
{ {
if( mpos_limit < Lzip_header::size + 5 ) return -1; if( mpos_limit < Lzip_header::size + 5 ) return -1;
if( member_position() == Lzip_header::size ) rdec.load(); if( member_position() == Lzip_header::size && !rdec.load() ) return 1;
while( !rdec.finished() ) while( !rdec.finished() )
{ {
if( member_position() >= mpos_limit || data_position() >= dpos_limit ) if( member_position() >= mpos_limit || data_position() >= dpos_limit )
@ -194,22 +194,22 @@ int LZ_mtester::test_member( const unsigned long mpos_limit,
} }
else // match else // match
{ {
rep3 = rep2; rep2 = rep1; rep1 = rep0;
len = rdec.decode_len( match_len_model, pos_state ); len = rdec.decode_len( match_len_model, pos_state );
unsigned distance = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] ); rep0 = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] );
if( distance >= start_dis_model ) if( rep0 >= start_dis_model )
{ {
const unsigned dis_slot = distance; const unsigned dis_slot = rep0;
const int direct_bits = ( dis_slot >> 1 ) - 1; const int direct_bits = ( dis_slot >> 1 ) - 1;
distance = ( 2 | ( dis_slot & 1 ) ) << direct_bits; rep0 = ( 2 | ( dis_slot & 1 ) ) << direct_bits;
if( dis_slot < end_dis_model ) if( dis_slot < end_dis_model )
distance += rdec.decode_tree_reversed( rep0 += rdec.decode_tree_reversed( bm_dis + ( rep0 - dis_slot ),
bm_dis + ( distance - dis_slot ), direct_bits ); direct_bits );
else else
{ {
distance += rep0 += rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits;
rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits; rep0 += rdec.decode_tree_reversed4( bm_align );
distance += rdec.decode_tree_reversed4( bm_align ); if( rep0 == 0xFFFFFFFFU ) // marker found
if( distance == 0xFFFFFFFFU ) // marker found
{ {
rdec.normalize(); rdec.normalize();
flush_data(); flush_data();
@ -224,7 +224,6 @@ int LZ_mtester::test_member( const unsigned long mpos_limit,
} }
} }
} }
rep3 = rep2; rep2 = rep1; rep1 = rep0; rep0 = distance;
if( rep0 > max_rep0 ) max_rep0 = rep0; if( rep0 > max_rep0 ) max_rep0 = rep0;
state.set_match(); state.set_match();
if( rep0 >= dictionary_size || ( rep0 >= pos && !pos_wrapped ) ) if( rep0 >= dictionary_size || ( rep0 >= pos && !pos_wrapped ) )
@ -242,7 +241,7 @@ int LZ_mtester::test_member( const unsigned long mpos_limit,
int LZ_mtester::debug_decode_member( const long long dpos, const long long mpos, int LZ_mtester::debug_decode_member( const long long dpos, const long long mpos,
const bool show_packets ) const bool show_packets )
{ {
rdec.load(); if( !rdec.load() ) return 1;
unsigned old_tmpos = member_position(); // truncated member position unsigned old_tmpos = member_position(); // truncated member position
while( !rdec.finished() ) while( !rdec.finished() )
{ {
@ -317,22 +316,22 @@ int LZ_mtester::debug_decode_member( const long long dpos, const long long mpos,
} }
else // match else // match
{ {
rep3 = rep2; rep2 = rep1; rep1 = rep0;
len = rdec.decode_len( match_len_model, pos_state ); len = rdec.decode_len( match_len_model, pos_state );
unsigned distance = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] ); rep0 = rdec.decode_tree6( bm_dis_slot[get_len_state(len)] );
if( distance >= start_dis_model ) if( rep0 >= start_dis_model )
{ {
const unsigned dis_slot = distance; const unsigned dis_slot = rep0;
const int direct_bits = ( dis_slot >> 1 ) - 1; const int direct_bits = ( dis_slot >> 1 ) - 1;
distance = ( 2 | ( dis_slot & 1 ) ) << direct_bits; rep0 = ( 2 | ( dis_slot & 1 ) ) << direct_bits;
if( dis_slot < end_dis_model ) if( dis_slot < end_dis_model )
distance += rdec.decode_tree_reversed( rep0 += rdec.decode_tree_reversed( bm_dis + ( rep0 - dis_slot ),
bm_dis + ( distance - dis_slot ), direct_bits ); direct_bits );
else else
{ {
distance += rep0 += rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits;
rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits; rep0 += rdec.decode_tree_reversed4( bm_align );
distance += rdec.decode_tree_reversed4( bm_align ); if( rep0 == 0xFFFFFFFFU ) // marker found
if( distance == 0xFFFFFFFFU ) // marker found
{ {
rdec.normalize(); rdec.normalize();
flush_data(); flush_data();
@ -353,7 +352,6 @@ int LZ_mtester::debug_decode_member( const long long dpos, const long long mpos,
} }
} }
} }
rep3 = rep2; rep2 = rep1; rep1 = rep0; rep0 = distance;
if( rep0 > max_rep0 ) { max_rep0 = rep0; max_rep0_pos = mp; } if( rep0 > max_rep0 ) { max_rep0 = rep0; max_rep0_pos = mp; }
state.set_match(); state.set_match();
if( show_packets ) if( show_packets )

19
mtester.h
View file

@ -53,12 +53,15 @@ public:
return p; return p;
} }
void load() bool load()
{ {
code = 0; code = 0;
range = 0xFFFFFFFFU; range = 0xFFFFFFFFU;
// check first byte of the LZMA stream without reading it
if( buffer[pos] != 0 ) return false;
get_byte(); // discard first byte of the LZMA stream get_byte(); // discard first byte of the LZMA stream
for( int i = 0; i < 4; ++i ) code = ( code << 8 ) | get_byte(); for( int i = 0; i < 4; ++i ) code = ( code << 8 ) | get_byte();
return true;
} }
void normalize() void normalize()
@ -76,7 +79,7 @@ public:
range >>= 1; range >>= 1;
// symbol <<= 1; // symbol <<= 1;
// if( code >= range ) { code -= range; symbol |= 1; } // if( code >= range ) { code -= range; symbol |= 1; }
const bool bit = ( code >= range ); const bool bit = code >= range;
symbol <<= 1; symbol += bit; symbol <<= 1; symbol += bit;
code -= range & ( 0U - bit ); code -= range & ( 0U - bit );
} }
@ -299,14 +302,14 @@ class LZ_mtester
bool fast, fast2; bool fast, fast2;
if( lpos > distance ) if( lpos > distance )
{ {
fast = ( len < dictionary_size - lpos ); fast = len < dictionary_size - lpos;
fast2 = ( fast && len <= lpos - i ); fast2 = fast && len <= lpos - i;
} }
else else
{ {
i += dictionary_size; i += dictionary_size;
fast = ( len < dictionary_size - i ); // (i == pos) may happen fast = len < dictionary_size - i; // (i == pos) may happen
fast2 = ( fast && len <= i - lpos ); fast2 = fast && len <= i - lpos;
} }
if( fast ) // no wrap if( fast ) // no wrap
{ {
@ -378,8 +381,8 @@ public:
const uint8_t * get_buffers( const uint8_t ** const prev_bufferp, const uint8_t * get_buffers( const uint8_t ** const prev_bufferp,
int * const sizep, int * const prev_sizep ) const int * const sizep, int * const prev_sizep ) const
{ *sizep = ( pos_wrapped && pos == 0 ) ? dictionary_size : pos; { *sizep = (pos_wrapped && pos == 0) ? dictionary_size : pos;
*prev_sizep = ( pos_wrapped && pos > 0 ) ? dictionary_size - pos : 0; *prev_sizep = (pos_wrapped && pos > 0) ? dictionary_size - pos : 0;
*prev_bufferp = buffer + pos; return buffer; } *prev_bufferp = buffer + pos; return buffer; }
void duplicate_buffer( uint8_t * const buffer2 ); void duplicate_buffer( uint8_t * const buffer2 );

4
nrep_stats.cc
View file

@ -44,11 +44,11 @@ int print_nrep_stats( const std::vector< std::string > & filenames,
unsigned long long lzma_size = 0; // total size of LZMA data unsigned long long lzma_size = 0; // total size of LZMA data
unsigned long best_pos = 0; unsigned long best_pos = 0;
int best_name = -1, retval = 0; int best_name = -1, retval = 0;
const bool count_all = ( repeated_byte < 0 || repeated_byte >= 256 ); const bool count_all = repeated_byte < 0 || repeated_byte >= 256;
bool stdin_used = false; bool stdin_used = false;
for( unsigned i = 0; i < filenames.size(); ++i ) for( unsigned i = 0; i < filenames.size(); ++i )
{ {
const bool from_stdin = ( filenames[i] == "-" ); const bool from_stdin = filenames[i] == "-";
if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; } if( from_stdin ) { if( stdin_used ) continue; else stdin_used = true; }
const char * const input_filename = const char * const input_filename =
from_stdin ? "(stdin)" : filenames[i].c_str(); from_stdin ? "(stdin)" : filenames[i].c_str();

15
range_dec.cc
View file

@ -53,7 +53,7 @@ bool decompress_member( const int infd, const Cl_options & cl_opts,
if( verbosity >= 2 ) pp(); if( verbosity >= 2 ) pp();
LZ_decoder decoder( rdec, dictionary_size, outfd, outskip, outend ); LZ_decoder decoder( rdec, dictionary_size, outfd, outskip, outend );
const int result = decoder.decode_member( pp, cl_opts.ignore_nonzero ); const int result = decoder.decode_member( pp, cl_opts.ignore_errors );
if( result != 0 ) if( result != 0 )
{ {
if( verbosity >= 0 && result <= 2 ) if( verbosity >= 0 && result <= 2 )
@ -94,7 +94,7 @@ const char * format_num( unsigned long long num,
static int current = 0; static int current = 0;
static bool si = true; static bool si = true;
if( set_prefix ) si = ( set_prefix > 0 ); if( set_prefix ) si = set_prefix > 0;
unsigned long long den = 1; unsigned long long den = 1;
const unsigned factor = si ? 1000 : 1024; const unsigned factor = si ? 1000 : 1024;
char * const buf = buffer[current++]; current %= buffers; char * const buf = buffer[current++]; current %= buffers;
@ -112,15 +112,6 @@ const char * format_num( unsigned long long num,
} }
bool safe_seek( const int fd, const long long pos,
const char * const filename )
{
if( lseek( fd, pos, SEEK_SET ) == pos ) return true;
show_file_error( filename, "Seek error", errno );
return false;
}
int range_decompress( const std::string & input_filename, int range_decompress( const std::string & input_filename,
const std::string & default_output_filename, const std::string & default_output_filename,
const Cl_options & cl_opts, Block range, const Cl_options & cl_opts, Block range,
@ -171,7 +162,7 @@ int range_decompress( const std::string & input_filename,
const long long outskip = std::max( 0LL, range.pos() - db.pos() ); const long long outskip = std::max( 0LL, range.pos() - db.pos() );
const long long outend = std::min( db.size(), range.end() - db.pos() ); const long long outend = std::min( db.size(), range.end() - db.pos() );
const long long mpos = lzip_index.mblock( i ).pos(); const long long mpos = lzip_index.mblock( i ).pos();
if( !safe_seek( infd, mpos, filename ) ) cleanup_and_fail( 1 ); if( !safe_seek( infd, mpos, input_filename ) ) cleanup_and_fail( 1 );
if( !decompress_member( infd, cl_opts, pp, mpos, outskip, outend ) ) if( !decompress_member( infd, cl_opts, pp, mpos, outskip, outend ) )
{ if( cl_opts.ignore_errors ) error = true; else cleanup_and_fail( 2 ); } { if( cl_opts.ignore_errors ) error = true; else cleanup_and_fail( 2 ); }
pp.reset(); pp.reset();

2
recursive.cc
View file

@ -45,7 +45,7 @@ bool test_full_name( const std::string & full_name, const struct stat * stp,
if( !S_ISDIR( st.st_mode ) ) return false; if( !S_ISDIR( st.st_mode ) ) return false;
std::string prev_dir( full_name ); std::string prev_dir( full_name );
bool loop = ( stp && st.st_ino == stp->st_ino && st.st_dev == stp->st_dev ); bool loop = stp && st.st_ino == stp->st_ino && st.st_dev == stp->st_dev;
if( !loop ) if( !loop )
for( unsigned i = prev_dir.size(); i > 1; ) for( unsigned i = prev_dir.size(); i > 1; )
{ {

29
reproduce.cc
View file

@ -459,7 +459,7 @@ int reproduce_member( uint8_t * const mbuffer, const long msize,
if( offset < 0 ) { delete master; return 2; } // no match if( offset < 0 ) { delete master; return 2; } // no match
/* Reference data from offset must be at least as large as zeroed sector /* Reference data from offset must be at least as large as zeroed sector
minus member trailer if trailer is inside the zeroed sector. */ minus member trailer if trailer is inside the zeroed sector. */
const int t = ( begin + size >= msize ) ? 16 + Lzip_trailer::size : 0; const int t = (begin + size >= msize) ? 16 + Lzip_trailer::size : 0;
if( rsize - offset < size - t ) if( rsize - offset < size - t )
{ show_file_error( reference_filename, "Not enough reference data after match." ); { show_file_error( reference_filename, "Not enough reference data after match." );
delete master; return 2; } delete master; return 2; }
@ -482,7 +482,7 @@ int reproduce_member( uint8_t * const mbuffer, const long msize,
std::fflush( stdout ); pending_newline = true; std::fflush( stdout ); pending_newline = true;
} }
const bool level0 = level == '0'; const bool level0 = level == '0';
const bool auto0 = ( level0 && lzip_level != '0' ); const bool auto0 = level0 && lzip_level != '0';
int ret = try_reproduce( mbuffer, msize, dsize, good_dsize, begin, end, int ret = try_reproduce( mbuffer, msize, dsize, good_dsize, begin, end,
rbuf, rsize, offset, dictionary_size, rbuf, rsize, offset, dictionary_size,
level0 ? lzip0_argv : lzip_argv, md5sump, terminator, auto0 ); level0 ? lzip0_argv : lzip_argv, md5sump, terminator, auto0 );
@ -550,7 +550,7 @@ int reproduce_file( const std::string & input_filename,
i + 1, lzip_index.members(), terminator ); i + 1, lzip_index.members(), terminator );
std::fflush( stdout ); pending_newline = true; std::fflush( stdout ); pending_newline = true;
} }
if( !safe_seek( infd, mpos, filename ) ) return 1; if( !safe_seek( infd, mpos, input_filename ) ) return 1;
long long failure_pos = 0; long long failure_pos = 0;
if( test_member_from_file( infd, msize, &failure_pos ) == 0 ) if( test_member_from_file( infd, msize, &failure_pos ) == 0 )
continue; // member is not damaged continue; // member is not damaged
@ -592,11 +592,11 @@ int reproduce_file( const std::string & input_filename,
{ {
if( outfd < 0 ) // first damaged member reproduced if( outfd < 0 ) // first damaged member reproduced
{ {
if( !safe_seek( infd, 0, filename ) ) return 1; if( !safe_seek( infd, 0, input_filename ) ) return 1;
set_signal_handler(); set_signal_handler();
if( !open_outstream( true, true, false, true, to_file ) ) return 1; if( !open_outstream( true, true, false, true, to_file ) ) return 1;
if( !copy_file( infd, outfd ) ) // copy whole file if( !copy_file( infd, outfd, input_filename, output_filename ) )
cleanup_and_fail( 1 ); cleanup_and_fail( 1 ); // copy whole file
} }
if( seek_write( outfd, mbuffer + begin, size, mpos + begin ) != size ) if( seek_write( outfd, mbuffer + begin, size, mpos + begin ) != size )
{ show_file_error( output_filename.c_str(), "Error writing file", errno ); { show_file_error( output_filename.c_str(), "Error writing file", errno );
@ -636,24 +636,25 @@ int reproduce_file( const std::string & input_filename,
/* Passes a 0 terminator to other functions to prevent intramember feedback. /* Passes a 0 terminator to other functions to prevent intramember feedback.
Exits only in case of fatal error. (reference file too large, etc). */ Exits only in case of fatal error. (reference file too large, etc). */
int debug_reproduce_file( const char * const input_filename, int debug_reproduce_file( const std::string & input_filename,
const char * const lzip_name, const char * const lzip_name,
const char * const reference_filename, const char * const reference_filename,
const Cl_options & cl_opts, const Block & range, const Cl_options & cl_opts, const Block & range,
const int sector_size, const int lzip_level ) const int sector_size, const int lzip_level )
{ {
const char * const filename = input_filename.c_str();
struct stat in_stats; // not used struct stat in_stats; // not used
const int infd = open_instream( input_filename, &in_stats, false, true ); const int infd = open_instream( filename, &in_stats, false, true );
if( infd < 0 ) return 1; if( infd < 0 ) return 1;
const Lzip_index lzip_index( infd, cl_opts ); const Lzip_index lzip_index( infd, cl_opts );
if( lzip_index.retval() != 0 ) if( lzip_index.retval() != 0 )
{ show_file_error( input_filename, lzip_index.error().c_str() ); { show_file_error( filename, lzip_index.error().c_str() );
return lzip_index.retval(); } return lzip_index.retval(); }
const long long cdata_size = lzip_index.cdata_size(); const long long cdata_size = lzip_index.cdata_size();
if( range.pos() >= cdata_size ) if( range.pos() >= cdata_size )
{ show_file_error( input_filename, "Range is beyond end of last member." ); { show_file_error( filename, "Range is beyond end of last member." );
return 1; } return 1; }
const long page_size = std::max( 1L, sysconf( _SC_PAGESIZE ) ); const long page_size = std::max( 1L, sysconf( _SC_PAGESIZE ) );
@ -670,7 +671,7 @@ int debug_reproduce_file( const char * const input_filename,
const long long msize = lzip_index.mblock( i ).size(); const long long msize = lzip_index.mblock( i ).size();
if( !range.overlaps( mpos, msize ) ) continue; if( !range.overlaps( mpos, msize ) ) continue;
if( !fits_in_size_t( msize + page_size ) ) // mmap uses size_t if( !fits_in_size_t( msize + page_size ) ) // mmap uses size_t
{ show_file_error( input_filename, { show_file_error( filename,
"Input file contains member too large for mmap." ); return 1; } "Input file contains member too large for mmap." ); return 1; }
const long long dsize = lzip_index.dblock( i ).size(); const long long dsize = lzip_index.dblock( i ).size();
const unsigned dictionary_size = lzip_index.dictionary_size( i ); const unsigned dictionary_size = lzip_index.dictionary_size( i );
@ -687,14 +688,14 @@ int debug_reproduce_file( const char * const input_filename,
uint8_t * const mbuffer_base = (uint8_t *)mmap( 0, msize + mpos_rem, uint8_t * const mbuffer_base = (uint8_t *)mmap( 0, msize + mpos_rem,
PROT_READ | PROT_WRITE, MAP_PRIVATE, infd, mpos - mpos_rem ); PROT_READ | PROT_WRITE, MAP_PRIVATE, infd, mpos - mpos_rem );
if( mbuffer_base == MAP_FAILED ) if( mbuffer_base == MAP_FAILED )
{ show_file_error( input_filename, mmap_msg, errno ); return 1; } { show_file_error( filename, mmap_msg, errno ); return 1; }
uint8_t * const mbuffer = mbuffer_base + mpos_rem; uint8_t * const mbuffer = mbuffer_base + mpos_rem;
if( !md5_valid ) if( !md5_valid )
{ {
if( verbosity >= 0 ) // give a clue of the range being tested if( verbosity >= 0 ) // give a clue of the range being tested
{ std::printf( "Reproducing: %s\nReference file: %s\nTesting " { std::printf( "Reproducing: %s\nReference file: %s\nTesting "
"sectors of size %llu at file positions %llu to %llu\n", "sectors of size %llu at file positions %llu to %llu\n",
input_filename, reference_filename, filename, reference_filename,
std::min( (long long)sector_size, rm_end - sector_pos ), std::min( (long long)sector_size, rm_end - sector_pos ),
sector_pos, rm_end - 1 ); std::fflush( stdout ); } sector_pos, rm_end - 1 ); std::fflush( stdout ); }
md5_valid = true; compute_md5( mbuffer, msize, md5_digest_c ); md5_valid = true; compute_md5( mbuffer, msize, md5_digest_c );
@ -719,7 +720,7 @@ int debug_reproduce_file( const char * const input_filename,
long size = 0; long size = 0;
uint8_t value = 0; uint8_t value = 0;
const long begin = const long begin =
zeroed_sector_pos( mbuffer, msize, input_filename, &size, &value ); zeroed_sector_pos( mbuffer, msize, filename, &size, &value );
if( begin < 0 ) return 2; if( begin < 0 ) return 2;
MD5SUM md5sum; MD5SUM md5sum;
const int ret = reproduce_member( mbuffer, msize, dsize, lzip_name, const int ret = reproduce_member( mbuffer, msize, dsize, lzip_name,

14
split.cc
View file

@ -88,14 +88,14 @@ int split_file( const std::string & input_filename,
long long mpos = b.pos(); long long mpos = b.pos();
long long msize = b.size(); long long msize = b.size();
long long failure_pos = 0; long long failure_pos = 0;
if( !safe_seek( infd, mpos, filename ) ) return 1; if( !safe_seek( infd, mpos, input_filename ) ) return 1;
if( test_member_from_file( infd, msize, &failure_pos ) == 1 ) if( test_member_from_file( infd, msize, &failure_pos ) == 1 )
{ // corrupt or fake trailer { // corrupt or fake trailer
while( true ) while( true )
{ {
mpos += failure_pos; msize -= failure_pos; mpos += failure_pos; msize -= failure_pos;
if( msize < min_member_size ) break; // trailing data if( msize < min_member_size ) break; // trailing data
if( !safe_seek( infd, mpos, filename ) ) return 1; if( !safe_seek( infd, mpos, input_filename ) ) return 1;
if( test_member_from_file( infd, msize, &failure_pos ) != 1 ) break; if( test_member_from_file( infd, msize, &failure_pos ) != 1 ) break;
} }
lzip_index = Lzip_index( infd, cl_opts, true, true, mpos ); lzip_index = Lzip_index( infd, cl_opts, true, true, mpos );
@ -106,7 +106,7 @@ int split_file( const std::string & input_filename,
} }
} }
if( !safe_seek( infd, 0, filename ) ) return 1; if( !safe_seek( infd, 0, input_filename ) ) return 1;
int max_digits = 1; int max_digits = 1;
for( long i = lzip_index.blocks( true ); i >= 10; i /= 10 ) ++max_digits; for( long i = lzip_index.blocks( true ); i >= 10; i /= 10 ) ++max_digits;
bool to_file = // if true, create intermediate dirs bool to_file = // if true, create intermediate dirs
@ -120,12 +120,13 @@ int split_file( const std::string & input_filename,
if( mb.pos() > stream_pos ) // gap if( mb.pos() > stream_pos ) // gap
{ {
if( !open_outstream( force, true, false, false, to_file ) ) return 1; if( !open_outstream( force, true, false, false, to_file ) ) return 1;
if( !copy_file( infd, outfd, mb.pos() - stream_pos ) || if( !copy_file( infd, outfd, input_filename, output_filename,
mb.pos() - stream_pos ) ||
!close_outstream( &in_stats ) ) cleanup_and_fail( 1 ); !close_outstream( &in_stats ) ) cleanup_and_fail( 1 );
next_filename( max_digits ); to_file = false; next_filename( max_digits ); to_file = false;
} }
if( !open_outstream( force, true, false, false, to_file ) ) return 1; // member if( !open_outstream( force, true, false, false, to_file ) ) return 1; // member
if( !copy_file( infd, outfd, mb.size() ) || if( !copy_file( infd, outfd, input_filename, output_filename, mb.size() ) ||
!close_outstream( &in_stats ) ) cleanup_and_fail( 1 ); !close_outstream( &in_stats ) ) cleanup_and_fail( 1 );
next_filename( max_digits ); to_file = false; next_filename( max_digits ); to_file = false;
stream_pos = mb.end(); stream_pos = mb.end();
@ -133,7 +134,8 @@ int split_file( const std::string & input_filename,
if( lzip_index.file_size() > stream_pos ) // trailing data if( lzip_index.file_size() > stream_pos ) // trailing data
{ {
if( !open_outstream( force, true, false, false, to_file ) ) return 1; if( !open_outstream( force, true, false, false, to_file ) ) return 1;
if( !copy_file( infd, outfd, lzip_index.file_size() - stream_pos ) || if( !copy_file( infd, outfd, input_filename, output_filename,
lzip_index.file_size() - stream_pos ) ||
!close_outstream( &in_stats ) ) cleanup_and_fail( 1 ); !close_outstream( &in_stats ) ) cleanup_and_fail( 1 );
next_filename( max_digits ); to_file = false; next_filename( max_digits ); to_file = false;
} }

339
testsuite/check.sh
View file

@ -32,7 +32,6 @@ cd "${objdir}"/tmp || framework_failure
cp "${testdir}"/test.txt in || framework_failure cp "${testdir}"/test.txt in || framework_failure
in_lz="${testdir}"/test.txt.lz in_lz="${testdir}"/test.txt.lz
in_lzma="${testdir}"/test.txt.lzma in_lzma="${testdir}"/test.txt.lzma
in_em="${testdir}"/test_em.txt.lz
inD="${testdir}"/test21636.txt inD="${testdir}"/test21636.txt
bad1_lz="${testdir}"/test_bad1.lz bad1_lz="${testdir}"/test_bad1.lz
bad2_lz="${testdir}"/test_bad2.lz bad2_lz="${testdir}"/test_bad2.lz
@ -44,8 +43,9 @@ bad7_lz="${testdir}"/test_bad7.lz
bad8_lz="${testdir}"/test_bad8.lz bad8_lz="${testdir}"/test_bad8.lz
bad9_lz="${testdir}"/test_bad9.lz bad9_lz="${testdir}"/test_bad9.lz
fox_lz="${testdir}"/fox.lz fox_lz="${testdir}"/fox.lz
fox6_lz="${testdir}"/fox6.lz fnz_lz="${testdir}"/fox_nz.lz
f6b1="${testdir}"/fox6_bad1.txt f6b1="${testdir}"/fox6_bad1.txt
f6b1nz_lz="${testdir}"/fox6_b1nz.lz
f6b1_lz="${testdir}"/fox6_bad1.lz f6b1_lz="${testdir}"/fox6_bad1.lz
f6b2_lz="${testdir}"/fox6_bad2.lz f6b2_lz="${testdir}"/fox6_bad2.lz
f6b3_lz="${testdir}"/fox6_bad3.lz f6b3_lz="${testdir}"/fox6_bad3.lz
@ -58,7 +58,6 @@ f6s3_lz="${testdir}"/fox6_sc3.lz
f6s4_lz="${testdir}"/fox6_sc4.lz f6s4_lz="${testdir}"/fox6_sc4.lz
f6s5_lz="${testdir}"/fox6_sc5.lz f6s5_lz="${testdir}"/fox6_sc5.lz
f6s6_lz="${testdir}"/fox6_sc6.lz f6s6_lz="${testdir}"/fox6_sc6.lz
f6nz_lz="${testdir}"/fox6_nz.lz
num_lz="${testdir}"/numbers.lz num_lz="${testdir}"/numbers.lz
nbt_lz="${testdir}"/numbersbt.lz nbt_lz="${testdir}"/numbersbt.lz
fail=0 fail=0
@ -76,10 +75,8 @@ test_failed() { fail=1 ; printf " $1" ; [ -z "$2" ] || printf "($2)" ; }
# test_bad8.lz: [ 66- 73] --> zeroed (reference test.txt [ 89- 110]) # test_bad8.lz: [ 66- 73] --> zeroed (reference test.txt [ 89- 110])
# test_bad9.lz: [6491-6498] --> zeroed (reference test.txt [17982-18594]) # test_bad9.lz: [6491-6498] --> zeroed (reference test.txt [17982-18594])
# #
# test_em.txt.lz: test.txt split in 3, with 5 empty members (1,3,5-6,8)
# test_3m.txt.lz.md5: md5sum of test_em.txt.lz after removing empty members
#
# 6-member files; fox6.lz with one or more errors # 6-member files; fox6.lz with one or more errors
# fox6_b1nz.lz: fox6_bad1.lz with first LZMA byte set to 'a'..'f'
# fox6_bad1.lz: byte at offset 5 changed from 0x0C to 0x00 (DS) # fox6_bad1.lz: byte at offset 5 changed from 0x0C to 0x00 (DS)
# byte at offset 142 changed from 0x50 to 0x70 (CRC) # byte at offset 142 changed from 0x50 to 0x70 (CRC)
# byte at offset 224 changed from 0x2D to 0x2E (data_size) # byte at offset 224 changed from 0x2D to 0x2E (data_size)
@ -92,8 +89,6 @@ test_failed() { fail=1 ; printf " $1" ; [ -z "$2" ] || printf "($2)" ; }
# fox6_bad5.lz: [380-479] --> zeroed (members 5,6) # fox6_bad5.lz: [380-479] --> zeroed (members 5,6)
# fox6_bad6.lz: [430-439] --> zeroed (member 6) # fox6_bad6.lz: [430-439] --> zeroed (member 6)
# #
# fox6_nz.lz: first LZMA byte of 4 last members set to 'a', 'b', 'c', 'd'
#
# 6-member files "shortcircuited" by a corrupt or fake trailer # 6-member files "shortcircuited" by a corrupt or fake trailer
# fox6_sc1.lz: (corrupt but consistent last trailer) # fox6_sc1.lz: (corrupt but consistent last trailer)
# last CRC != 0 ; dsize = 4 * msize ; msize = 480 (file size) # last CRC != 0 ; dsize = 4 * msize ; msize = 480 (file size)
@ -114,7 +109,8 @@ printf "testing lziprecover-%s..." "$2"
"${LZIPRECOVER}" -q --nrep-stats=0N "${in_lz}" "${LZIPRECOVER}" -q --nrep-stats=0N "${in_lz}"
[ $? = 1 ] || test_failed $LINENO [ $? = 1 ] || test_failed $LINENO
for i in 0 255 0kB 0KiB 0M 0G 0T 0P 0E 0Z 0Y 0R 0Q ; do for i in 0 255 0kB 0KiB 0M 0G 0T 0P 0E 0Z 0Y 0R 0Q ; do
"${LZIPRECOVER}" -q --nrep-stats=$i "${in_lz}" || test_failed $LINENO $i "${LZIPRECOVER}" --nrep-stats=$i "${in_lz}" > /dev/null ||
test_failed $LINENO $i
done done
"${LZIP}" -lq in "${LZIP}" -lq in
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
@ -249,6 +245,7 @@ cp "${in_lzma}" out.lzma || framework_failure
cmp "${in_lz}" out.lz || test_failed $LINENO cmp "${in_lz}" out.lz || test_failed $LINENO
printf "to be overwritten" > out.lz || framework_failure printf "to be overwritten" > out.lz || framework_failure
"${LZIPRECOVER}" -Af out.lzma || test_failed $LINENO "${LZIPRECOVER}" -Af out.lzma || test_failed $LINENO
[ ! -e out.lzma ] || test_failed $LINENO
cmp "${in_lz}" out.lz || test_failed $LINENO cmp "${in_lz}" out.lz || test_failed $LINENO
rm -f out.lz || framework_failure rm -f out.lz || framework_failure
cp "${in_lzma}" out.tlz || framework_failure cp "${in_lzma}" out.tlz || framework_failure
@ -261,39 +258,26 @@ rm -f out.tar.lz || framework_failure
cat in in > in2 || framework_failure cat in in > in2 || framework_failure
"${LZIPRECOVER}" -A -o out2.lz - "${in_lzma}" - < "${in_lzma}" || "${LZIPRECOVER}" -A -o out2.lz - "${in_lzma}" - < "${in_lzma}" ||
test_failed $LINENO test_failed $LINENO
"${LZIP}" -cd out2.lz > out2 || test_failed $LINENO "${LZIP}" -d out2.lz || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO cmp in2 out2 || test_failed $LINENO
rm -f out2.lz out2 || framework_failure rm -f out2 || framework_failure
"${LZIPRECOVER}" -A "${in_lzma}" -o a/b/c/out.lz || test_failed $LINENO "${LZIPRECOVER}" -A "${in_lzma}" -o a/b/c/out.lz || test_failed $LINENO
cmp "${in_lz}" a/b/c/out.lz || test_failed $LINENO cmp "${in_lz}" a/b/c/out.lz || test_failed $LINENO
rm -rf a || framework_failure rm -rf a || framework_failure
printf "\ntesting decompression..." printf "\ntesting decompression..."
for i in "${in_lz}" "${in_em}" ; do "${LZIP}" -l "${in_lz}" > /dev/null || test_failed $LINENO
"${LZIP}" -lq "$i" --ignore-empty || test_failed $LINENO "$i" "${LZIP}" -t "${in_lz}" || test_failed $LINENO
"${LZIP}" -t "$i" --ignore-empty || test_failed $LINENO "$i" "${LZIP}" -d "${in_lz}" -o out || test_failed $LINENO
"${LZIP}" -d "$i" --ignore-empty -o out || test_failed $LINENO "$i" cmp in out || test_failed $LINENO
cmp in out || test_failed $LINENO "$i" "${LZIP}" -cd "${in_lz}" > out || test_failed $LINENO
"${LZIP}" -cd "$i" --ignore-empty > out || test_failed $LINENO "$i" cmp in out || test_failed $LINENO
cmp in out || test_failed $LINENO "$i" "${LZIP}" -d "${in_lz}" -o - > out || test_failed $LINENO
"${LZIP}" -d "$i" --ignore-empty -o - > out || test_failed $LINENO "$i" cmp in out || test_failed $LINENO
cmp in out || test_failed $LINENO "$i" "${LZIP}" -d < "${in_lz}" > out || test_failed $LINENO
"${LZIP}" -d < "$i" --ignore-empty > out || test_failed $LINENO "$i" cmp in out || test_failed $LINENO
cmp in out || test_failed $LINENO "$i" rm -f out || framework_failure
rm -f out || framework_failure
done
lines=`"${LZIP}" -tvv "${in_em}" --ignore-empty 2>&1 | wc -l` ||
test_failed $LINENO
[ "${lines}" -eq 8 ] || test_failed $LINENO "${lines}"
"${LZIP}" -tq "${in_em}"
[ $? = 2 ] || test_failed $LINENO
lines=`"${LZIP}" -lvv "${in_em}" --ignore-empty | wc -l` || test_failed $LINENO
[ "${lines}" -eq 11 ] || test_failed $LINENO "${lines}"
"${LZIP}" -lq "${in_em}"
[ $? = 2 ] || test_failed $LINENO
cp "${in_lz}" out.lz || framework_failure cp "${in_lz}" out.lz || framework_failure
"${LZIP}" -dk out.lz || test_failed $LINENO "${LZIP}" -dk out.lz || test_failed $LINENO
@ -315,7 +299,6 @@ rm -f copy out || framework_failure
printf "to be overwritten" > out || framework_failure printf "to be overwritten" > out || framework_failure
"${LZIP}" -df -o out < "${in_lz}" || test_failed $LINENO "${LZIP}" -df -o out < "${in_lz}" || test_failed $LINENO
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
rm -f out || framework_failure
"${LZIP}" -d -o ./- "${in_lz}" || test_failed $LINENO "${LZIP}" -d -o ./- "${in_lz}" || test_failed $LINENO
cmp in ./- || test_failed $LINENO cmp in ./- || test_failed $LINENO
rm -f ./- || framework_failure rm -f ./- || framework_failure
@ -328,7 +311,7 @@ cp "${in_lz}" anyothername || framework_failure
test_failed $LINENO test_failed $LINENO
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
cmp in anyothername.out || test_failed $LINENO cmp in anyothername.out || test_failed $LINENO
rm -f out anyothername.out || framework_failure rm -f anyothername.out || framework_failure
"${LZIP}" -lq in "${in_lz}" "${LZIP}" -lq in "${in_lz}"
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
@ -365,7 +348,7 @@ done
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
rm -f out || framework_failure rm -f out || framework_failure
"${LZIP}" -lq "${in_lz}" "${in_lz}" || test_failed $LINENO "${LZIP}" -l "${in_lz}" "${in_lz}" > /dev/null || test_failed $LINENO
"${LZIP}" -t "${in_lz}" "${in_lz}" || test_failed $LINENO "${LZIP}" -t "${in_lz}" "${in_lz}" || test_failed $LINENO
"${LZIP}" -cd "${in_lz}" "${in_lz}" -o out > out2 || test_failed $LINENO "${LZIP}" -cd "${in_lz}" "${in_lz}" -o out > out2 || test_failed $LINENO
[ ! -e out ] || test_failed $LINENO # override -o [ ! -e out ] || test_failed $LINENO # override -o
@ -376,10 +359,13 @@ cmp in2 out2 || test_failed $LINENO
rm -f out2 || framework_failure rm -f out2 || framework_failure
cat "${in_lz}" "${in_lz}" > out2.lz || framework_failure cat "${in_lz}" "${in_lz}" > out2.lz || framework_failure
lines=`"${LZIP}" -tvv out2.lz 2>&1 | wc -l` || test_failed $LINENO
[ "${lines}" -eq 2 ] || test_failed $LINENO "${lines}"
lines=`"${LZIP}" -lvv out2.lz | wc -l` || test_failed $LINENO
[ "${lines}" -eq 5 ] || test_failed $LINENO "${lines}"
printf "\ngarbage" >> out2.lz || framework_failure printf "\ngarbage" >> out2.lz || framework_failure
"${LZIP}" -tvvvv out2.lz 2> /dev/null || test_failed $LINENO "${LZIP}" -tvvvv out2.lz 2> /dev/null || test_failed $LINENO
"${LZIPRECOVER}" -aD0 -q out2.lz
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -alq out2.lz "${LZIP}" -alq out2.lz
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -atq out2.lz "${LZIP}" -atq out2.lz
@ -392,6 +378,8 @@ printf "\ngarbage" >> out2.lz || framework_failure
"${LZIP}" -adkq -o out2 < out2.lz "${LZIP}" -adkq -o out2 < out2.lz
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
[ ! -e out2 ] || test_failed $LINENO [ ! -e out2 ] || test_failed $LINENO
"${LZIPRECOVER}" -aD0 -q out2.lz
[ $? = 2 ] || test_failed $LINENO
printf "to be overwritten" > out2 || framework_failure printf "to be overwritten" > out2 || framework_failure
"${LZIP}" -df out2.lz || test_failed $LINENO "${LZIP}" -df out2.lz || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO cmp in2 out2 || test_failed $LINENO
@ -410,11 +398,21 @@ rm -rf a || framework_failure
if [ -z "${LZIP_NAME}" ] ; then LZIP_NAME=lzip ; fi if [ -z "${LZIP_NAME}" ] ; then LZIP_NAME=lzip ; fi
touch empty em || framework_failure touch empty em || framework_failure
"${LZIP_NAME}" -0 em || test_failed $LINENO "${LZIP_NAME}" -0 em || test_failed $LINENO
"${LZIP}" -lq em.lz || test_failed $LINENO "${LZIP}" -l em.lz > /dev/null || test_failed $LINENO
"${LZIP}" -d em.lz || test_failed $LINENO "${LZIP}" -dk em.lz || test_failed $LINENO
cmp empty em || test_failed $LINENO cmp empty em || test_failed $LINENO
rm -f em || framework_failure cat em.lz em.lz | "${LZIP}" -t || test_failed $LINENO
cat em.lz em.lz | "${LZIP}" -d > em || test_failed $LINENO
cmp empty em || test_failed $LINENO
cat em.lz "${in_lz}" | "${LZIP}" -t || test_failed $LINENO
cat em.lz "${in_lz}" | "${LZIP}" -d > out || test_failed $LINENO
cmp in out || test_failed $LINENO
cat "${in_lz}" em.lz | "${LZIP}" -t || test_failed $LINENO
cat "${in_lz}" em.lz | "${LZIP}" -d > out || test_failed $LINENO
cmp in out || test_failed $LINENO
cat em.lz "${fox_lz}" em.lz "${fox_lz}" em.lz em.lz "${fox_lz}" em.lz > \
f3_em.lz || framework_failure
"${LZIPRECOVER}" -D ,18000 "${in_lz}" > out || test_failed $LINENO "${LZIPRECOVER}" -D ,18000 "${in_lz}" > out || test_failed $LINENO
"${LZIPRECOVER}" -D 18000 "${in_lz}" >> out || test_failed $LINENO "${LZIPRECOVER}" -D 18000 "${in_lz}" >> out || test_failed $LINENO
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
@ -422,22 +420,60 @@ cmp in out || test_failed $LINENO
cmp "${inD}" out || test_failed $LINENO cmp "${inD}" out || test_failed $LINENO
"${LZIPRECOVER}" -D 21636,397 "${in_lz}" > out || test_failed $LINENO "${LZIPRECOVER}" -D 21636,397 "${in_lz}" > out || test_failed $LINENO
cmp "${inD}" out || test_failed $LINENO cmp "${inD}" out || test_failed $LINENO
"${LZIPRECOVER}" -D 21636,397 "${in_em}" --ignore-empty > out || "${LZIPRECOVER}" -D 45,45 f3_em.lz > out || test_failed $LINENO
test_failed $LINENO cmp fox out || test_failed $LINENO
cmp "${inD}" out || test_failed $LINENO
"${LZIPRECOVER}" -q -D 21636,397 "${in_em}"
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -D 0 "${in_lz}" -o a/b/c/out || test_failed $LINENO "${LZIP}" -D 0 "${in_lz}" -o a/b/c/out || test_failed $LINENO
cmp in a/b/c/out || test_failed $LINENO cmp in a/b/c/out || test_failed $LINENO
rm -rf a || framework_failure rm -rf a || framework_failure
printf "\ntesting bad input..." printf "\ntesting bad input..."
cat em.lz em.lz > ee.lz || framework_failure
"${LZIP}" -l < ee.lz > /dev/null || test_failed $LINENO
"${LZIP}" -t < ee.lz || test_failed $LINENO
"${LZIP}" -d < ee.lz > em || test_failed $LINENO
cmp empty em || test_failed $LINENO
"${LZIPRECOVER}" -li ee.lz > /dev/null || test_failed $LINENO
"${LZIPRECOVER}" -ti ee.lz || test_failed $LINENO
"${LZIPRECOVER}" -cdi ee.lz > em || test_failed $LINENO
cmp empty em || test_failed $LINENO
"${LZIP}" -lq ee.lz
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -tq ee.lz
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -dq ee.lz
[ $? = 2 ] || test_failed $LINENO
[ ! -e ee ] || test_failed $LINENO
"${LZIP}" -cdq ee.lz > em
[ $? = 2 ] || test_failed $LINENO
cmp empty em || test_failed $LINENO
rm -f em || framework_failure
cat "${in_lz}" em.lz "${in_lz}" > inein.lz || framework_failure
"${LZIP}" -l < inein.lz > /dev/null || test_failed $LINENO
"${LZIP}" -t < inein.lz || test_failed $LINENO
"${LZIP}" -d < inein.lz > out2 || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO
"${LZIPRECOVER}" -li inein.lz > /dev/null || test_failed $LINENO
"${LZIPRECOVER}" -ti inein.lz || test_failed $LINENO
"${LZIPRECOVER}" -cdi inein.lz > out2 || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO
"${LZIP}" -lq inein.lz
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -tq inein.lz
[ $? = 2 ] || test_failed $LINENO
"${LZIP}" -dq inein.lz
[ $? = 2 ] || test_failed $LINENO
[ ! -e inein ] || test_failed $LINENO
"${LZIP}" -cdq inein.lz > out2
[ $? = 2 ] || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO
rm -f out2 inein.lz em.lz || framework_failure
headers='LZIp LZiP LZip LzIP LzIp LziP lZIP lZIp lZiP lzIP' headers='LZIp LZiP LZip LzIP LzIp LziP lZIP lZIp lZiP lzIP'
body='\001\014\000\000\101\376\367\377\377\340\000\200\000\215\357\002\322\001\000\000\000\000\000\000\000\045\000\000\000\000\000\000\000' body='\001\014\000\000\101\376\367\377\377\340\000\200\000\215\357\002\322\001\000\000\000\000\000\000\000\045\000\000\000\000\000\000\000'
cp "${in_lz}" int.lz || framework_failure cp "${in_lz}" int.lz || framework_failure
printf "LZIP${body}" >> int.lz || framework_failure printf "LZIP${body}" >> int.lz || framework_failure
if "${LZIP}" -tq int.lz ; then if "${LZIP}" -t int.lz ; then
for header in ${headers} ; do for header in ${headers} ; do
printf "${header}${body}" > int.lz || framework_failure printf "${header}${body}" > int.lz || framework_failure
"${LZIP}" -lq int.lz # first member "${LZIP}" -lq int.lz # first member
@ -466,7 +502,7 @@ if "${LZIP}" -tq int.lz ; then
[ $? = 2 ] || test_failed $LINENO ${header} [ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -cdq int.lz > /dev/null "${LZIP}" -cdq int.lz > /dev/null
[ $? = 2 ] || test_failed $LINENO ${header} [ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -lq --loose-trailing int.lz || "${LZIP}" -l --loose-trailing int.lz > /dev/null ||
test_failed $LINENO ${header} test_failed $LINENO ${header}
"${LZIP}" -t --loose-trailing int.lz || "${LZIP}" -t --loose-trailing int.lz ||
test_failed $LINENO ${header} test_failed $LINENO ${header}
@ -497,24 +533,31 @@ if "${LZIP}" -tq int.lz ; then
cmp "${in_lz}" int.lz || test_failed $LINENO ${header} cmp "${in_lz}" int.lz || test_failed $LINENO ${header}
done done
else else
printf "\nwarning: skipping header test: 'printf' does not work on your system." printf "warning: skipping header test: 'printf' does not work on your system."
fi fi
rm -f int.lz out || framework_failure rm -f int.lz || framework_failure
"${LZIP}" -cd "${fox6_lz}" > out || test_failed $LINENO "${LZIP}" -l "${fnz_lz}" > /dev/null || test_failed $LINENO
"${LZIPRECOVER}" -cd "${f6nz_lz}" --ignore-nonzero > copy || test_failed $LINENO "${LZIP}" -tq "${fnz_lz}"
cmp copy out || test_failed $LINENO
rm -f copy out || framework_failure
"${LZIP}" -lq "${f6nz_lz}" || test_failed $LINENO
"${LZIP}" -tq "${f6nz_lz}"
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
cp "${f6nz_lz}" f6nz.lz || framework_failure cat "${fox_lz}" "${fox_lz}" "${fox_lz}" "${fox_lz}" "${fox_lz}" "${fox_lz}" \
cp "${f6nz_lz}" f6nz2.lz || framework_failure > fox6.lz || framework_failure
cmp -s "${fox6_lz}" f6nz.lz && test_failed $LINENO cat "${fox_lz}" "${fnz_lz}" "${fox_lz}" "${fnz_lz}" "${fox_lz}" "${fnz_lz}" \
"${LZIPRECOVER}" --nonzero-repair f6nz.lz f6nz2.lz || test_failed $LINENO > fox6_nz.lz || framework_failure
cmp "${fox6_lz}" f6nz.lz || test_failed $LINENO "${LZIP}" -cd fox6.lz > out || test_failed $LINENO
cmp "${fox6_lz}" f6nz2.lz || test_failed $LINENO "${LZIPRECOVER}" -cd -i fox6_nz.lz > copy || test_failed $LINENO
rm -f f6nz.lz f6nz2.lz || framework_failure cmp copy out || test_failed $LINENO
rm -f copy || framework_failure
cp fox6_nz.lz f6nz.lz || framework_failure
cp fox6_nz.lz f6nz2.lz || framework_failure
cp "${f6b1nz_lz}" f6b1nz.lz || framework_failure
cmp -s fox6.lz f6nz.lz && test_failed $LINENO
"${LZIPRECOVER}" --nonzero-repair f6nz.lz f6nz2.lz f6b1nz.lz ||
test_failed $LINENO
cmp fox6.lz f6nz.lz || test_failed $LINENO
cmp fox6.lz f6nz2.lz || test_failed $LINENO
cmp "${f6b1_lz}" f6b1nz.lz || test_failed $LINENO
rm -f f6nz.lz f6nz2.lz f6b1nz.lz || framework_failure
for i in fox_v2.lz fox_s11.lz fox_de20.lz \ for i in fox_v2.lz fox_s11.lz fox_de20.lz \
fox_bcrc.lz fox_crc0.lz fox_das46.lz fox_mes81.lz ; do fox_bcrc.lz fox_crc0.lz fox_das46.lz fox_mes81.lz ; do
@ -530,12 +573,12 @@ for i in fox_bcrc.lz fox_crc0.lz fox_das46.lz fox_mes81.lz ; do
"${LZIPRECOVER}" -cdq -i "${testdir}"/$i > out || test_failed $LINENO $i "${LZIPRECOVER}" -cdq -i "${testdir}"/$i > out || test_failed $LINENO $i
cmp fox out || test_failed $LINENO $i cmp fox out || test_failed $LINENO $i
done done
rm -f fox out || framework_failure rm -f fox || framework_failure
cat "${in_lz}" "${in_lz}" > in2.lz || framework_failure cat "${in_lz}" "${in_lz}" > in2.lz || framework_failure
cat "${in_lz}" "${in_lz}" "${in_lz}" > in3.lz || framework_failure cat "${in_lz}" "${in_lz}" "${in_lz}" > in3.lz || framework_failure
if dd if=in3.lz of=trunc.lz bs=14682 count=1 2> /dev/null && if dd if=in3.lz of=trunc.lz bs=14682 count=1 2> /dev/null &&
[ -e trunc.lz ] && cmp in2.lz trunc.lz > /dev/null 2>&1 ; then [ -e trunc.lz ] && cmp in2.lz trunc.lz ; then
for i in 6 20 14664 14683 14684 14685 14686 14687 14688 ; do for i in 6 20 14664 14683 14684 14685 14686 14687 14688 ; do
dd if=in3.lz of=trunc.lz bs=$i count=1 2> /dev/null dd if=in3.lz of=trunc.lz bs=$i count=1 2> /dev/null
"${LZIP}" -lq trunc.lz "${LZIP}" -lq trunc.lz
@ -550,7 +593,7 @@ if dd if=in3.lz of=trunc.lz bs=14682 count=1 2> /dev/null &&
[ $? = 2 ] || test_failed $LINENO $i [ $? = 2 ] || test_failed $LINENO $i
done done
else else
printf "\nwarning: skipping truncation test: 'dd' does not work on your system." printf "warning: skipping truncation test: 'dd' does not work on your system."
fi fi
rm -f in3.lz trunc.lz || framework_failure rm -f in3.lz trunc.lz || framework_failure
@ -572,17 +615,22 @@ cat "${in_lz}" >> ingin.lz || framework_failure
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -atq < ingin.lz "${LZIP}" -atq < ingin.lz
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -acdq ingin.lz > /dev/null "${LZIP}" -acdq ingin.lz > out
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -adq < ingin.lz > /dev/null cmp in out || test_failed $LINENO
"${LZIP}" -adq < ingin.lz > out
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIPRECOVER}" -lq -i ingin.lz || test_failed $LINENO cmp in out || test_failed $LINENO
"${LZIP}" -t ingin.lz || test_failed $LINENO "${LZIP}" -t ingin.lz || test_failed $LINENO
"${LZIP}" -t < ingin.lz || test_failed $LINENO "${LZIP}" -t < ingin.lz || test_failed $LINENO
"${LZIP}" -dk ingin.lz || test_failed $LINENO
cmp in ingin || test_failed $LINENO
"${LZIP}" -cd ingin.lz > out || test_failed $LINENO "${LZIP}" -cd ingin.lz > out || test_failed $LINENO
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
"${LZIP}" -d < ingin.lz > out || test_failed $LINENO "${LZIP}" -d < ingin.lz > out || test_failed $LINENO
cmp in out || test_failed $LINENO cmp in out || test_failed $LINENO
rm -f ingin || framework_failure
"${LZIPRECOVER}" -l -i ingin.lz > /dev/null || test_failed $LINENO
"${LZIPRECOVER}" -cd -i ingin.lz > out2 || test_failed $LINENO "${LZIPRECOVER}" -cd -i ingin.lz > out2 || test_failed $LINENO
cmp in2 out2 || test_failed $LINENO cmp in2 out2 || test_failed $LINENO
@ -615,7 +663,7 @@ rm -f in2 in2t.lz out out2 || framework_failure
printf "\ntesting --byte-repair..." printf "\ntesting --byte-repair..."
rm -f out.lz || framework_failure rm -f out.lz || framework_failure
"${LZIPRECOVER}" -B -o out.lz "${fox6_lz}" || test_failed $LINENO "${LZIPRECOVER}" -B -o out.lz fox6.lz || test_failed $LINENO
[ ! -e out.lz ] || test_failed $LINENO [ ! -e out.lz ] || test_failed $LINENO
"${LZIPRECOVER}" -B -o out.lz "${bad2_lz}" -q "${LZIPRECOVER}" -B -o out.lz "${bad2_lz}" -q
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
@ -627,7 +675,11 @@ rm -f out.lz || framework_failure
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
[ ! -e out.lz ] || test_failed $LINENO [ ! -e out.lz ] || test_failed $LINENO
"${LZIPRECOVER}" -Bf -o out.lz "${f6b1_lz}" || test_failed $LINENO "${LZIPRECOVER}" -Bf -o out.lz "${f6b1_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -Bf fox6_nz.lz -o out.lz || test_failed $LINENO
cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -Bf "${f6b1nz_lz}" -o out.lz || test_failed $LINENO
cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -Bf -o out.lz "${bad1_lz}" || test_failed $LINENO "${LZIPRECOVER}" -Bf -o out.lz "${bad1_lz}" || test_failed $LINENO
cmp "${in_lz}" out.lz || test_failed $LINENO cmp "${in_lz}" out.lz || test_failed $LINENO
"${LZIPRECOVER}" -B -o a/b/c/out.lz "${bad1_lz}" || test_failed $LINENO "${LZIPRECOVER}" -B -o a/b/c/out.lz "${bad1_lz}" || test_failed $LINENO
@ -667,21 +719,21 @@ cmp fec/test.txt.lz.fec fecfile.fec || test_failed $LINENO
"${LZIPRECOVER}" -Ft "${in_lz}" --fec-file=fec/ || test_failed $LINENO "${LZIPRECOVER}" -Ft "${in_lz}" --fec-file=fec/ || test_failed $LINENO
rm -rf fec || framework_failure rm -rf fec || framework_failure
mkdir a mkdir a
cp "${in_lz}" "${fox6_lz}" a || framework_failure cp fox6.lz "${in_lz}" a || framework_failure
"${LZIPRECOVER}" -r -Fc a/ -o fec/ || test_failed $LINENO "${LZIPRECOVER}" -r -Fc a/ -o fec/ || test_failed $LINENO
cmp fec/test.txt.lz.fec fecfile.fec || test_failed $LINENO
[ -e fec/fox6.lz.fec ] || test_failed $LINENO [ -e fec/fox6.lz.fec ] || test_failed $LINENO
cmp fec/test.txt.lz.fec fecfile.fec || test_failed $LINENO
"${LZIPRECOVER}" -r -Fc a -o fec/ || test_failed $LINENO "${LZIPRECOVER}" -r -Fc a -o fec/ || test_failed $LINENO
cmp fec/a/test.txt.lz.fec fecfile.fec || test_failed $LINENO
[ -e fec/a/fox6.lz.fec ] || test_failed $LINENO [ -e fec/a/fox6.lz.fec ] || test_failed $LINENO
cmp fec/a/test.txt.lz.fec fecfile.fec || test_failed $LINENO
rm -rf a fec fecfile.fec || framework_failure rm -rf a fec fecfile.fec || framework_failure
printf "\ntesting --merge..." printf "\ntesting --merge..."
rm -f out.lz || framework_failure rm -f out.lz || framework_failure
"${LZIPRECOVER}" -m -o out.lz "${fox6_lz}" "${f6b1_lz}" || test_failed $LINENO "${LZIPRECOVER}" -m -o out.lz fox6.lz "${f6b1_lz}" || test_failed $LINENO
[ ! -e out.lz ] || test_failed $LINENO [ ! -e out.lz ] || test_failed $LINENO
"${LZIPRECOVER}" -m -o out.lz "${f6b1_lz}" "${fox6_lz}" || test_failed $LINENO "${LZIPRECOVER}" -m -o out.lz "${f6b1_lz}" fox6.lz || test_failed $LINENO
[ ! -e out.lz ] || test_failed $LINENO [ ! -e out.lz ] || test_failed $LINENO
"${LZIPRECOVER}" -m -o out.lz "${bad1_lz}" "${bad2_lz}" "${bad1_lz}" -q "${LZIPRECOVER}" -m -o out.lz "${bad1_lz}" "${bad2_lz}" "${bad1_lz}" -q
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
@ -705,56 +757,56 @@ rm -f bad2.lz || framework_failure
[ ! -e out.lz ] || test_failed $LINENO [ ! -e out.lz ] || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b4_lz}" || test_failed $LINENO "${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b4_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${f6b4_lz}" "${f6b1_lz}" || test_failed $LINENO "${LZIPRECOVER}" -mf -o out.lz "${f6b4_lz}" "${f6b1_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -m -o a/b/c/out.lz "${f6b1_lz}" "${f6b4_lz}" || "${LZIPRECOVER}" -m -o a/b/c/out.lz "${f6b1_lz}" "${f6b4_lz}" ||
test_failed $LINENO test_failed $LINENO
cmp "${fox6_lz}" a/b/c/out.lz || test_failed $LINENO cmp fox6.lz a/b/c/out.lz || test_failed $LINENO
rm -rf a || framework_failure rm -rf a || framework_failure
for i in "${f6b1_lz}" "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" "${f6b6_lz}" ; do for i in "${f6b1_lz}" "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" "${f6b6_lz}" ; do
"${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "$i" || "${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "$i" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b2_lz}" || "${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b2_lz}" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
done done
for i in "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" "${f6b6_lz}" ; do for i in "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" "${f6b6_lz}" ; do
"${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b2_lz}" "$i" || "${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b2_lz}" "$i" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "$i" "${f6b2_lz}" || "${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "$i" "${f6b2_lz}" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "${f6b1_lz}" "$i" || "${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "${f6b1_lz}" "$i" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "$i" "${f6b1_lz}" || "${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "$i" "${f6b1_lz}" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b1_lz}" "${f6b2_lz}" || "${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b1_lz}" "${f6b2_lz}" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
"${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b2_lz}" "${f6b1_lz}" || "${LZIPRECOVER}" -mf -o out.lz "$i" "${f6b2_lz}" "${f6b1_lz}" ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "${fox6_lz}" out.lz || test_failed $LINENO "$i" cmp fox6.lz out.lz || test_failed $LINENO "$i"
done done
"${LZIPRECOVER}" -mf -o out.lz "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" || "${LZIPRECOVER}" -mf -o out.lz "${f6b3_lz}" "${f6b4_lz}" "${f6b5_lz}" ||
test_failed $LINENO test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b3_lz}" "${f6b4_lz}" \ "${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b3_lz}" "${f6b4_lz}" \
"${f6b5_lz}" || test_failed $LINENO "${f6b5_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "${f6b3_lz}" "${f6b4_lz}" \ "${LZIPRECOVER}" -mf -o out.lz "${f6b2_lz}" "${f6b3_lz}" "${f6b4_lz}" \
"${f6b5_lz}" || test_failed $LINENO "${f6b5_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b2_lz}" "${f6b3_lz}" \ "${LZIPRECOVER}" -mf -o out.lz "${f6b1_lz}" "${f6b2_lz}" "${f6b3_lz}" \
"${f6b4_lz}" "${f6b5_lz}" || test_failed $LINENO "${f6b4_lz}" "${f6b5_lz}" || test_failed $LINENO
cmp "${fox6_lz}" out.lz || test_failed $LINENO cmp fox6.lz out.lz || test_failed $LINENO
"${LZIPRECOVER}" -mf -o out.lz "${bad1_lz}" "${bad2_lz}" || test_failed $LINENO "${LZIPRECOVER}" -mf -o out.lz "${bad1_lz}" "${bad2_lz}" || test_failed $LINENO
cmp "${in_lz}" out.lz || test_failed $LINENO cmp "${in_lz}" out.lz || test_failed $LINENO
@ -836,7 +888,7 @@ rm -f bad345.lz bad453.lz bad534.lz out4.lz || framework_failure
printf "\ntesting --reproduce..." printf "\ntesting --reproduce..."
if /bin/sh -c "${LZIP_NAME} -s18KiB" < in > out 2> /dev/null && if /bin/sh -c "${LZIP_NAME} -s18KiB" < in > out 2> /dev/null &&
cmp "${in_lz}" out > /dev/null 2>&1 ; then cmp "${in_lz}" out ; then
rm -f out || framework_failure rm -f out || framework_failure
"${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" -o out \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" -o out \
--reference-file=foo "${in_lz}" || test_failed $LINENO "${LZIP_NAME}" --reference-file=foo "${in_lz}" || test_failed $LINENO "${LZIP_NAME}"
@ -850,19 +902,19 @@ if /bin/sh -c "${LZIP_NAME} -s18KiB" < in > out 2> /dev/null &&
for i in 6 7 8 9 ; do for i in 6 7 8 9 ; do
for f in "${testdir}"/test_bad${i}.txt in ; do for f in "${testdir}"/test_bad${i}.txt in ; do
rm -f out || framework_failure rm -f out || framework_failure
"${LZIPRECOVER}" -q --reproduce --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" -o out \
--reference-file="$f" "${testdir}"/test_bad${i}.lz -o out || --reference-file="$f" "${testdir}"/test_bad${i}.lz > /dev/null ||
test_failed $LINENO "${LZIP_NAME} $i $f" test_failed $LINENO "${LZIP_NAME} $i $f"
cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f" cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f"
rm -f out || framework_failure rm -f out || framework_failure
"${LZIPRECOVER}" -q --reproduce --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" -o out \
--reference-file="$f" "${testdir}"/test_bad${i}.lz -o out \ --lzip-level=6 --reference-file="$f" "${testdir}"/test_bad${i}.lz \
--lzip-level=6 || test_failed $LINENO "${LZIP_NAME} $i $f level=6" > /dev/null || test_failed $LINENO "${LZIP_NAME} $i $f level=6"
cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f level=6" cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f level=6"
rm -f out || framework_failure rm -f out || framework_failure
"${LZIPRECOVER}" -q --reproduce --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" -o out \
--reference-file="$f" "${testdir}"/test_bad${i}.lz -o out \ --lzip-level=m36 --reference-file="$f" "${testdir}"/test_bad${i}.lz \
--lzip-level=m36 || test_failed $LINENO "${LZIP_NAME} $i $f level=m36" > /dev/null || test_failed $LINENO "${LZIP_NAME} $i $f level=m36"
cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f level=m36" cmp "${in_lz}" out || test_failed $LINENO "${LZIP_NAME} $i $f level=m36"
done done
done done
@ -872,8 +924,8 @@ if /bin/sh -c "${LZIP_NAME} -s18KiB" < in > out 2> /dev/null &&
framework_failure framework_failure
rm -f out || framework_failure rm -f out || framework_failure
for i in 6 7 8 9 ; do # reproduce one member each time for i in 6 7 8 9 ; do # reproduce one member each time
"${LZIPRECOVER}" -q --reproduce --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" mm_bad.lz \
--reference-file="${testdir}"/test_bad${i}.txt mm_bad.lz -o out || --reference-file="${testdir}"/test_bad${i}.txt -o out > /dev/null ||
test_failed $LINENO "${LZIP_NAME} $i" test_failed $LINENO "${LZIP_NAME} $i"
mv out mm_bad.lz || framework_failure mv out mm_bad.lz || framework_failure
done done
@ -884,22 +936,23 @@ if /bin/sh -c "${LZIP_NAME} -s18KiB" < in > out 2> /dev/null &&
framework_failure framework_failure
rm -f out || framework_failure rm -f out || framework_failure
for i in 6 7 8 9 ; do # reproduce one member each time for i in 6 7 8 9 ; do # reproduce one member each time
"${LZIPRECOVER}" -q --reproduce --lzip-name="${LZIP_NAME}" -o out \ "${LZIPRECOVER}" --reproduce --lzip-name="${LZIP_NAME}" mm_bad.lz -o out \
--reference-file=in mm_bad.lz || test_failed $LINENO "${LZIP_NAME} $i" --reference-file=in > /dev/null || test_failed $LINENO "${LZIP_NAME} $i"
mv out mm_bad.lz || framework_failure mv out mm_bad.lz || framework_failure
done done
cmp in4.lz mm_bad.lz || test_failed $LINENO "${LZIP_NAME}" cmp in4.lz mm_bad.lz || test_failed $LINENO "${LZIP_NAME}"
rm -f mm_bad.lz || framework_failure rm -f mm_bad.lz || framework_failure
"${LZIPRECOVER}" -q --debug-reproduce=13-7356 --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --reference-file=in "${in_lz}" --lzip-name="${LZIP_NAME}" \
--reference-file=in "${in_lz}" || test_failed $LINENO "${LZIP_NAME}" --debug-reproduce=13-7356 > /dev/null || test_failed $LINENO "${LZIP_NAME}"
"${LZIPRECOVER}" --debug-reproduce=512,5120,512 --lzip-name="${LZIP_NAME}" \ "${LZIPRECOVER}" --debug-reproduce=512,5120,512 --lzip-name="${LZIP_NAME}" \
-q --reference-file=in "${in_lz}" || test_failed $LINENO "${LZIP_NAME}" -q --reference-file=in "${in_lz}" || test_failed $LINENO "${LZIP_NAME}"
else else
printf "\nwarning: skipping --reproduce test: ${LZIP_NAME} not found or not the right version.\n" printf "warning: skipping --reproduce test: "
${LZIP_NAME} -V printf "${LZIP_NAME} not found, not the right compressor, not the right version, or this is not a POSIX system.\n"
printf "\nTry 'make LZIP_NAME=<name_of_lzip_executable> check'." if ${LZIP_NAME} -V > /dev/null 2>&1 ; then ${LZIP_NAME} -V | sed -e 1q
else printf "Try 'make LZIP_NAME=<name_of_lzip_executable> check'.\n" ; fi
fi fi
rm -f in4.lz || framework_failure rm -f in4.lz || framework_failure
@ -1222,21 +1275,17 @@ for i in 1 2 3 4 5 6 7 8 9 10 ; do
done done
rm -f nbt.lz || framework_failure rm -f nbt.lz || framework_failure
cp "${in_em}" test_3m.txt.lz || framework_failure cat "${fox_lz}" "${fox_lz}" "${fox_lz}" > fox3.lz || framework_failure
"${LZIPRECOVER}" --remove=empty test_3m.txt.lz || test_failed $LINENO cp f3_em.lz out.lz || framework_failure
"${LZIPRECOVER}" -M test_3m.txt.lz | cmp "${testdir}"/test_3m.txt.lz.md5 - || "${LZIPRECOVER}" --remove=empty out.lz || test_failed $LINENO
cmp fox3.lz out.lz || test_failed $LINENO
rm -f out.lz || framework_failure
"${LZIPRECOVER}" --dump=2,4,7 f3_em.lz | cmp fox3.lz - || test_failed $LINENO
"${LZIPRECOVER}" --strip=e f3_em.lz | cmp fox3.lz - || test_failed $LINENO
"${LZIPRECOVER}" --strip=1,3,5-6,8 f3_em.lz | cmp fox3.lz - ||
test_failed $LINENO test_failed $LINENO
"${LZIPRECOVER}" --dump=2,4,7 "${in_em}" | cmp test_3m.txt.lz - || "${LZIPRECOVER}" --dump=empty f3_em.lz | "${LZIP}" -d | cmp empty - ||
test_failed $LINENO test_failed $LINENO
"${LZIPRECOVER}" --strip=e "${in_em}" | cmp test_3m.txt.lz - ||
test_failed $LINENO
"${LZIPRECOVER}" --strip=e "${in_em}" | cmp test_3m.txt.lz - ||
test_failed $LINENO
"${LZIPRECOVER}" --strip=1,3,5-6,8 "${in_em}" | cmp test_3m.txt.lz - ||
test_failed $LINENO
"${LZIPRECOVER}" --dump=emp "${in_em}" | "${LZIP}" -d --ignore-empty | \
cmp empty - || test_failed $LINENO
rm -f test_3m.txt.lz || framework_failure
printf "\ntesting --dump/remove/strip=damaged..." printf "\ntesting --dump/remove/strip=damaged..."
@ -1295,9 +1344,9 @@ cmp "${f6b1_lz}" out || test_failed $LINENO
cat "${f6b1_lz}" in > f6bt.lz || framework_failure cat "${f6b1_lz}" in > f6bt.lz || framework_failure
"${LZIPRECOVER}" --dump=damaged f6bt.lz > out || test_failed $LINENO "${LZIPRECOVER}" --dump=damaged f6bt.lz > out || test_failed $LINENO
cmp "${f6b1_lz}" out || test_failed $LINENO cmp "${f6b1_lz}" out || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged "${f6b1_lz}" > out || test_failed $LINENO "${LZIPRECOVER}" --strip=damaged "${f6b1_lz}" > out || test_failed $LINENO
cmp empty out || test_failed $LINENO cmp empty out || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged f6bt.lz > out || test_failed $LINENO "${LZIPRECOVER}" --strip=damaged f6bt.lz > out || test_failed $LINENO
cmp empty out || test_failed $LINENO cmp empty out || test_failed $LINENO
cp "${f6b1_lz}" f6b.lz || framework_failure cp "${f6b1_lz}" f6b.lz || framework_failure
"${LZIPRECOVER}" -q --remove=damaged f6b.lz "${LZIPRECOVER}" -q --remove=damaged f6b.lz
@ -1335,7 +1384,6 @@ cat "${f6b3_lz}" in > f6bt.lz || framework_failure
"${LZIPRECOVER}" --dump=damaged f6bt.lz > out || test_failed $LINENO "${LZIPRECOVER}" --dump=damaged f6bt.lz > out || test_failed $LINENO
cat "${fox_lz}" "${fox_lz}" out "${fox_lz}" | cmp "${f6b3_lz}" - || cat "${fox_lz}" "${fox_lz}" out "${fox_lz}" | cmp "${f6b3_lz}" - ||
test_failed $LINENO test_failed $LINENO
cat "${fox_lz}" "${fox_lz}" "${fox_lz}" > fox3.lz || framework_failure
"${LZIPRECOVER}" --strip=damaged "${f6b3_lz}" > out || test_failed $LINENO "${LZIPRECOVER}" --strip=damaged "${f6b3_lz}" > out || test_failed $LINENO
cmp fox3.lz out || test_failed $LINENO cmp fox3.lz out || test_failed $LINENO
"${LZIPRECOVER}" --strip=damaged f6bt.lz > out || test_failed $LINENO "${LZIPRECOVER}" --strip=damaged f6bt.lz > out || test_failed $LINENO
@ -1392,10 +1440,9 @@ for i in "${f6s1_lz}" "${f6s2_lz}" ; do
"${LZIPRECOVER}" --dump=damaged f6bt.lz > out || "${LZIPRECOVER}" --dump=damaged f6bt.lz > out ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp "$i" out || test_failed $LINENO "$i" cmp "$i" out || test_failed $LINENO "$i"
"${LZIPRECOVER}" -q --strip=damaged "$i" > out || "${LZIPRECOVER}" --strip=damaged "$i" > out || test_failed $LINENO "$i"
test_failed $LINENO "$i"
cmp empty out || test_failed $LINENO "$i" cmp empty out || test_failed $LINENO "$i"
"${LZIPRECOVER}" -q --strip=damaged f6bt.lz > out || "${LZIPRECOVER}" --strip=damaged f6bt.lz > out ||
test_failed $LINENO "$i" test_failed $LINENO "$i"
cmp empty out || test_failed $LINENO "$i" cmp empty out || test_failed $LINENO "$i"
cp "$i" f6b.lz || framework_failure cp "$i" f6b.lz || framework_failure
@ -1458,22 +1505,22 @@ cat "${bad2_lz}" out "${bad2_lz}" out | cmp out4 - || test_failed $LINENO
"${bad2_lz}" > out4 || test_failed $LINENO "${bad2_lz}" > out4 || test_failed $LINENO
cat out "${bad2_lz}" out "${bad2_lz}" | cmp out4 - || test_failed $LINENO cat out "${bad2_lz}" out "${bad2_lz}" | cmp out4 - || test_failed $LINENO
# #
"${LZIPRECOVER}" -q --strip=damaged "${bad2_lz}" "${f6b2_lz}" > out || "${LZIPRECOVER}" --strip=damaged "${bad2_lz}" "${f6b2_lz}" > out ||
test_failed $LINENO test_failed $LINENO
cmp fox5.lz out || test_failed $LINENO cmp fox5.lz out || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged bad2t.lz "${f6b2_lz}" > out || "${LZIPRECOVER}" --strip=damaged bad2t.lz "${f6b2_lz}" > out ||
test_failed $LINENO test_failed $LINENO
cmp fox5.lz out || test_failed $LINENO cmp fox5.lz out || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged "${f6b2_lz}" bad2t.lz f6bt.lz > out || "${LZIPRECOVER}" --strip=damaged "${f6b2_lz}" bad2t.lz f6bt.lz > out ||
test_failed $LINENO test_failed $LINENO
cat fox5.lz fox5.lz in | cmp out - || test_failed $LINENO cat fox5.lz fox5.lz in | cmp out - || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged "${f6b2_lz}" f6bt.lz bad2t.lz > out || "${LZIPRECOVER}" --strip=damaged "${f6b2_lz}" f6bt.lz bad2t.lz > out ||
test_failed $LINENO test_failed $LINENO
cat fox5.lz fox5.lz | cmp out - || test_failed $LINENO cat fox5.lz fox5.lz | cmp out - || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged f6bt.lz bad2t.lz > out || "${LZIPRECOVER}" --strip=damaged f6bt.lz bad2t.lz > out ||
test_failed $LINENO test_failed $LINENO
cmp fox5.lz out || test_failed $LINENO cmp fox5.lz out || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=damaged f6bt.lz "${in_lz}" > out || "${LZIPRECOVER}" --strip=damaged f6bt.lz "${in_lz}" > out ||
test_failed $LINENO test_failed $LINENO
cat fox5.lz "${in_lz}" | cmp out - || test_failed $LINENO cat fox5.lz "${in_lz}" | cmp out - || test_failed $LINENO
"${LZIPRECOVER}" --strip=damaged --strip=tdata f6bt.lz "${in_lz}" > out || "${LZIPRECOVER}" --strip=damaged --strip=tdata f6bt.lz "${in_lz}" > out ||
@ -1487,7 +1534,7 @@ cat "${bad2_lz}" in | cmp bad2t.lz - || test_failed $LINENO
cmp fox5.lz f6b.lz || test_failed $LINENO cmp fox5.lz f6b.lz || test_failed $LINENO
cat fox5.lz in | cmp f6bt.lz - || test_failed $LINENO cat fox5.lz in | cmp f6bt.lz - || test_failed $LINENO
cat "${bad2_lz}" in > bad2t.lz || framework_failure cat "${bad2_lz}" in > bad2t.lz || framework_failure
cat "${fox6_lz}" "${inD}" > fox6t.lz || framework_failure cat fox6.lz "${inD}" > fox6t.lz || framework_failure
cat "${f6b1_lz}" in > f6abt.lz || framework_failure cat "${f6b1_lz}" in > f6abt.lz || framework_failure
cp "${f6b2_lz}" f6b.lz || framework_failure cp "${f6b2_lz}" f6b.lz || framework_failure
cat "${f6b2_lz}" in > f6bt.lz || framework_failure cat "${f6b2_lz}" in > f6bt.lz || framework_failure
@ -1495,7 +1542,7 @@ cat "${f6b2_lz}" in > f6bt.lz || framework_failure
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
cat "${bad2_lz}" in | cmp bad2t.lz - || test_failed $LINENO cat "${bad2_lz}" in | cmp bad2t.lz - || test_failed $LINENO
cat "${f6b1_lz}" in | cmp f6abt.lz - || test_failed $LINENO cat "${f6b1_lz}" in | cmp f6abt.lz - || test_failed $LINENO
cmp "${fox6_lz}" fox6t.lz || test_failed $LINENO cmp fox6.lz fox6t.lz || test_failed $LINENO
cmp fox5.lz f6b.lz || test_failed $LINENO cmp fox5.lz f6b.lz || test_failed $LINENO
cmp fox5.lz f6bt.lz || test_failed $LINENO cmp fox5.lz f6bt.lz || test_failed $LINENO
rm -f fox6t.lz f6b.lz f6bt.lz f6abt.lz bad2t.lz fox5.lz out2 out4 || rm -f fox6t.lz f6b.lz f6bt.lz f6abt.lz bad2t.lz fox5.lz out2 out4 ||
@ -1509,7 +1556,7 @@ cmp "${inD}" out || test_failed $LINENO
rm -f out || framework_failure rm -f out || framework_failure
"${LZIPRECOVER}" --dump=tdat int.lz -o out || test_failed $LINENO "${LZIPRECOVER}" --dump=tdat int.lz -o out || test_failed $LINENO
cmp "${inD}" out || test_failed $LINENO cmp "${inD}" out || test_failed $LINENO
cat "${fox6_lz}" "${inD}" > fox6t.lz || framework_failure cat fox6.lz "${inD}" > fox6t.lz || framework_failure
cat "${inD}" "${inD}" > inD2 || framework_failure cat "${inD}" "${inD}" > inD2 || framework_failure
"${LZIPRECOVER}" --dump=tda int.lz fox6t.lz -f -o out || test_failed $LINENO "${LZIPRECOVER}" --dump=tda int.lz fox6t.lz -f -o out || test_failed $LINENO
cmp inD2 out || test_failed $LINENO cmp inD2 out || test_failed $LINENO
@ -1526,12 +1573,12 @@ rm -f out || framework_failure
"${LZIPRECOVER}" --strip=tdata int.lz -o out || test_failed $LINENO "${LZIPRECOVER}" --strip=tdata int.lz -o out || test_failed $LINENO
cmp "${in_lz}" out || test_failed $LINENO cmp "${in_lz}" out || test_failed $LINENO
"${LZIPRECOVER}" --strip=tdata fox6t.lz -f -o out || test_failed $LINENO "${LZIPRECOVER}" --strip=tdata fox6t.lz -f -o out || test_failed $LINENO
cmp "${fox6_lz}" out || test_failed $LINENO cmp fox6.lz out || test_failed $LINENO
"${LZIPRECOVER}" --strip=tdata int.lz int.lz -f -o out || test_failed $LINENO "${LZIPRECOVER}" --strip=tdata int.lz int.lz -f -o out || test_failed $LINENO
cmp in2.lz out || test_failed $LINENO cmp in2.lz out || test_failed $LINENO
rm -f in2.lz || framework_failure rm -f in2.lz || framework_failure
"${LZIPRECOVER}" --strip=tdata int.lz fox6t.lz > out || test_failed $LINENO "${LZIPRECOVER}" --strip=tdata int.lz fox6t.lz > out || test_failed $LINENO
cat "${in_lz}" "${fox6_lz}" | cmp out - || test_failed $LINENO cat "${in_lz}" fox6.lz | cmp out - || test_failed $LINENO
"${LZIPRECOVER}" -q --strip=tdata ingint.lz > out # /dev/null returns 1 on OS/2 "${LZIPRECOVER}" -q --strip=tdata ingint.lz > out # /dev/null returns 1 on OS/2
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIPRECOVER}" -i --strip=tdata ingint.lz > out || test_failed $LINENO "${LZIPRECOVER}" -i --strip=tdata ingint.lz > out || test_failed $LINENO
@ -1539,11 +1586,11 @@ cmp ingin.lz out || test_failed $LINENO
"${LZIPRECOVER}" --remove=tdata int.lz fox6t.lz || test_failed $LINENO "${LZIPRECOVER}" --remove=tdata int.lz fox6t.lz || test_failed $LINENO
cmp "${in_lz}" int.lz || test_failed $LINENO cmp "${in_lz}" int.lz || test_failed $LINENO
cmp "${fox6_lz}" fox6t.lz || test_failed $LINENO cmp fox6.lz fox6t.lz || test_failed $LINENO
"${LZIPRECOVER}" --remove=tdata int.lz || test_failed $LINENO "${LZIPRECOVER}" --remove=tdata int.lz || test_failed $LINENO
cmp "${in_lz}" int.lz || test_failed $LINENO cmp "${in_lz}" int.lz || test_failed $LINENO
"${LZIPRECOVER}" --remove=tdata fox6t.lz || test_failed $LINENO "${LZIPRECOVER}" --remove=tdata fox6t.lz || test_failed $LINENO
cmp "${fox6_lz}" fox6t.lz || test_failed $LINENO cmp fox6.lz fox6t.lz || test_failed $LINENO
"${LZIPRECOVER}" -q --remove=tdata ingint.lz "${LZIPRECOVER}" -q --remove=tdata ingint.lz
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
cmp -s ingin.lz ingint.lz && test_failed $LINENO cmp -s ingin.lz ingint.lz && test_failed $LINENO
@ -1554,15 +1601,15 @@ rm -f int.lz fox6t.lz ingint.lz ingin.lz || framework_failure
for i in "${f6s3_lz}" "${f6s4_lz}" "${f6s5_lz}" "${f6s6_lz}" ; do for i in "${f6s3_lz}" "${f6s4_lz}" "${f6s5_lz}" "${f6s6_lz}" ; do
"${LZIPRECOVER}" --strip=tdata "$i" > out || test_failed $LINENO "$i" "${LZIPRECOVER}" --strip=tdata "$i" > out || test_failed $LINENO "$i"
"${LZIPRECOVER}" --dump=tdata "$i" > tdata || test_failed $LINENO "$i" "${LZIPRECOVER}" --dump=tdata "$i" > tdata || test_failed $LINENO "$i"
cmp "${fox6_lz}" out || test_failed $LINENO "$i" cmp fox6.lz out || test_failed $LINENO "$i"
cat out tdata | cmp "$i" - || test_failed $LINENO "$i" cat out tdata | cmp "$i" - || test_failed $LINENO "$i"
cat "$i" "${inD}" > f6t.lz || framework_failure cat "$i" "${inD}" > f6t.lz || framework_failure
"${LZIPRECOVER}" --strip=tdata f6t.lz > out || test_failed $LINENO "$i" "${LZIPRECOVER}" --strip=tdata f6t.lz > out || test_failed $LINENO "$i"
"${LZIPRECOVER}" --dump=tdata f6t.lz > tdata || test_failed $LINENO "$i" "${LZIPRECOVER}" --dump=tdata f6t.lz > tdata || test_failed $LINENO "$i"
cmp "${fox6_lz}" out || test_failed $LINENO "$i" cmp fox6.lz out || test_failed $LINENO "$i"
cat out tdata | cmp f6t.lz - || test_failed $LINENO "$i" cat out tdata | cmp f6t.lz - || test_failed $LINENO "$i"
"${LZIPRECOVER}" --remove=tdata f6t.lz || test_failed $LINENO "$i" "${LZIPRECOVER}" --remove=tdata f6t.lz || test_failed $LINENO "$i"
cmp "${fox6_lz}" f6t.lz || test_failed $LINENO "$i" cmp fox6.lz f6t.lz || test_failed $LINENO "$i"
rm -f out tdata f6t.lz || framework_failure rm -f out tdata f6t.lz || framework_failure
done done

BIN
testsuite/fox6.lz
View file

Binary file not shown.

BIN
testsuite/fox6_b1nz.lz Normal file
View file

Binary file not shown.

BIN
testsuite/fox6_nz.lz
View file

Binary file not shown.

BIN
testsuite/fox_nz.lz Normal file
View file

Binary file not shown.

1
testsuite/test_3m.txt.lz.md5
View file

@ -1 +0,0 @@
aa8ca65001d627f89e7494fa829e710f test_3m.txt.lz

BIN
testsuite/test_em.txt.lz
View file

Binary file not shown.

8
unzcrash.cc
View file

@ -149,7 +149,7 @@ uint8_t * read_file( const char * const filename, long * const file_sizep )
if( buffer_size >= LONG_MAX ) if( buffer_size >= LONG_MAX )
{ show_file_error( filename, large_file_msg ); { show_file_error( filename, large_file_msg );
std::free( buffer ); return 0; } std::free( buffer ); return 0; }
buffer_size = ( buffer_size <= LONG_MAX / 2 ) ? 2 * buffer_size : LONG_MAX; buffer_size = (buffer_size <= LONG_MAX / 2) ? 2 * buffer_size : LONG_MAX;
uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size ); uint8_t * const tmp = (uint8_t *)std::realloc( buffer, buffer_size );
if( !tmp ) if( !tmp )
{ show_file_error( filename, mem_msg ); std::free( buffer ); return 0; } { show_file_error( filename, mem_msg ); std::free( buffer ); return 0; }
@ -354,7 +354,7 @@ int main( const int argc, const char * const argv[] )
{ {
enum Mode { m_block, m_byte, m_truncate }; enum Mode { m_block, m_byte, m_truncate };
const char * mode_str[3] = { "block", "byte", "size" }; const char * mode_str[3] = { "block", "byte", "size" };
Bitset8 bits; // if Bitset8::parse_bs not called test full byte Bitset8 bits; // if Bitset8::parse_bs not called, test full byte
Bad_byte bad_byte; Bad_byte bad_byte;
const char * zcmp_program = "zcmp"; const char * zcmp_program = "zcmp";
long pos = 0; long pos = 0;
@ -423,7 +423,7 @@ int main( const int argc, const char * const argv[] )
return 1; return 1;
} }
if( delta <= 0 ) delta = ( program_mode == m_block ) ? block_size : 1; if( delta <= 0 ) delta = (program_mode == m_block) ? block_size : 1;
const char * const command = parser.argument( argind ).c_str(); const char * const command = parser.argument( argind ).c_str();
std::vector< std::string > command_args; std::vector< std::string > command_args;
@ -498,7 +498,7 @@ int main( const int argc, const char * const argv[] )
( max_size < 0 && -max_size >= file_size - pos ) ) ( max_size < 0 && -max_size >= file_size - pos ) )
{ show_error( "Nothing to do; domain is empty." ); return 0; } { show_error( "Nothing to do; domain is empty." ); return 0; }
if( max_size < 0 ) max_size += file_size - pos; if( max_size < 0 ) max_size += file_size - pos;
const long end = ( ( max_size < file_size - pos ) ? pos + max_size : file_size ); const long end = (max_size < file_size - pos) ? pos + max_size : file_size;
if( bad_byte.pos >= file_size ) if( bad_byte.pos >= file_size )
{ show_option_error( bad_byte.argument, "Position is beyond end of file in", { show_option_error( bad_byte.argument, "Position is beyond end of file in",
bad_byte.option_name ); return 1; } bad_byte.option_name ); return 1; }