lzip/clzip
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.10.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-17 20:45:25 +01:00
parent 6e0e1539c4
commit 15bdbbe06a
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
24 changed files with 811 additions and 454 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
ChangeLog
View file

@ -1,3 +1,20 @@
2018-02-06 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.10 released.
* main.c: Added new option '--loose-trailing'.
* Improved corrupt header detection to HD=3.
* main.c: Show corrupt or truncated header in multimember file.
* main.c (main): Option '-S, --volume-size' now keeps input files.
* encoder_base.*: Adjust dictionary size for each member.
* Replaced 'bits/byte' with inverse compression ratio in output.
* Show progress of decompression at verbosity level 2 (-vv).
* Show progress of (de)compression only if stderr is a terminal.
* main.c: Show final diagnostic when testing multiple files.
* main.c: Do not add a second .lz extension to the arg of -o.
* decoder.c (LZd_verify_trailer): Show stored sizes also in hex.
Show dictionary size at verbosity level 4 (-vvvv).
* lzip.texi: Added chapter 'Output'.
2017-04-13 Antonio Diaz Diaz <antonio@gnu.org> 2017-04-13 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.9 released. * Version 1.9 released.
@ -24,8 +41,8 @@
* clzip.texi: Added chapter 'Trailing data'. * clzip.texi: Added chapter 'Trailing data'.
* configure: Avoid warning on some shells when testing for gcc. * configure: Avoid warning on some shells when testing for gcc.
* Makefile.in: Detect the existence of install-info. * Makefile.in: Detect the existence of install-info.
* testsuite/check.sh: A POSIX shell is required to run the tests. * check.sh: A POSIX shell is required to run the tests.
* testsuite/check.sh: Don't check error messages. * check.sh: Don't check error messages.
2015-07-07 Antonio Diaz Diaz <antonio@gnu.org> 2015-07-07 Antonio Diaz Diaz <antonio@gnu.org>
@ -108,7 +125,7 @@
* Translated to C from the C++ source of lzip 1.10. * Translated to C from the C++ source of lzip 1.10.
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This file is a collection of facts, and thus it is not copyrightable, This file is a collection of facts, and thus it is not copyrightable,
but just in case, you have unlimited permission to copy, distribute and but just in case, you have unlimited permission to copy, distribute and

2
INSTALL
View file

@ -62,7 +62,7 @@ After running 'configure', you can run 'make' and 'make install' as
explained above. explained above.
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This file is free documentation: you have unlimited permission to copy, This file is free documentation: you have unlimited permission to copy,
distribute and modify it. distribute and modify it.

47
NEWS
View file

@ -1,21 +1,42 @@
Changes in version 1.9: Changes in version 1.10:
The option '-l, --list' has been ported from lziprecover. The option '--loose-trailing', has been added.
It is now an error to specify two or more different operations in the The test used by clzip to discriminate trailing data from a corrupt
command line (--decompress, --list or --test). header in multimember or concatenated files has been improved to a
Hamming distance (HD) of 3, and the 3 bit flips must happen in different
magic bytes for the test to fail. As a consequence some kinds of files
no longer can be appended to a lzip file as trailing data unless the
'--loose-trailing' option is used when decompressing.
Lziprecover can be used to remove conflicting trailing data from a file.
Compression time of option '-0' has been reduced by 6%. The contents of a corrupt or truncated header found in a multimember
file is now shown, after the error message, in the same format as
trailing data.
Compression time of options '-1' to '-9' has been reduced by 1%. Option '-S, --volume-size' now keeps input files unchanged.
Decompression time has been reduced by 7%. When creating multimember files or splitting the output in volumes, the
dictionary size is now adjusted for each member individually.
In test mode, clzip now continues checking the rest of the files if any The 'bits/byte' ratio has been replaced with the inverse compression
input file is a terminal. ratio in the output.
Trailing data are now shown both in hexadecimal and as a string of The progress of decompression is now shown at verbosity level 2 (-vv) or
printable ASCII characters. higher.
Three missing chapters have been added to the manual, which now contains Progress of (de)compression is only shown if stderr is a terminal.
all the chapters of the lzip manual.
A final diagnostic is now shown at verbosity level 1 (-v) or higher if
any file fails the test when testing multiple files.
A second '.lz' extension is no longer added to the argument of '-o' if
it already ends in '.lz' or '.tlz'.
In case of (de)compressed size mismatch, the stored size is now also
shown in hexadecimal to ease visual comparison.
The dictionary size is now shown at verbosity level 4 (-vvvv) when
decompressing or testing.
The new chapter "Meaning of clzip's output" has been added to the manual.

4
README
View file

@ -78,7 +78,7 @@ incomprehensible and therefore pointless.
Clzip will correctly decompress a file which is the concatenation of two Clzip will correctly decompress a file which is the concatenation of two
or more compressed files. The result is the concatenation of the or more compressed files. The result is the concatenation of the
corresponding uncompressed files. Integrity testing of concatenated corresponding decompressed files. Integrity testing of concatenated
compressed files is also supported. compressed files is also supported.
Clzip can produce multimember files, and lziprecover can safely recover Clzip can produce multimember files, and lziprecover can safely recover
@ -116,7 +116,7 @@ range encoding), Igor Pavlov (for putting all the above together in
LZMA), and Julian Seward (for bzip2's CLI). LZMA), and Julian Seward (for bzip2's CLI).
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This file is free documentation: you have unlimited permission to copy, This file is free documentation: you have unlimited permission to copy,
distribute and modify it. distribute and modify it.

2
carg_parser.c
View file

@ -1,5 +1,5 @@
/* Arg_parser - POSIX/GNU command line argument parser. (C version) /* Arg_parser - POSIX/GNU command line argument parser. (C version)
Copyright (C) 2006-2017 Antonio Diaz Diaz. Copyright (C) 2006-2018 Antonio Diaz Diaz.
This library is free software. Redistribution and use in source and This library is free software. Redistribution and use in source and
binary forms, with or without modification, are permitted provided binary forms, with or without modification, are permitted provided

2
carg_parser.h
View file

@ -1,5 +1,5 @@
/* Arg_parser - POSIX/GNU command line argument parser. (C version) /* Arg_parser - POSIX/GNU command line argument parser. (C version)
Copyright (C) 2006-2017 Antonio Diaz Diaz. Copyright (C) 2006-2018 Antonio Diaz Diaz.
This library is free software. Redistribution and use in source and This library is free software. Redistribution and use in source and
binary forms, with or without modification, are permitted provided binary forms, with or without modification, are permitted provided

6
configure vendored
View file

@ -1,12 +1,12 @@
#! /bin/sh #! /bin/sh
# configure script for Clzip - LZMA lossless data compressor # configure script for Clzip - LZMA lossless data compressor
# Copyright (C) 2010-2017 Antonio Diaz Diaz. # Copyright (C) 2010-2018 Antonio Diaz Diaz.
# #
# This configure script is free software: you have unlimited permission # This configure script is free software: you have unlimited permission
# to copy, distribute and modify it. # to copy, distribute and modify it.
pkgname=clzip pkgname=clzip
pkgversion=1.9 pkgversion=1.10
progname=clzip progname=clzip
srctrigger=doc/${pkgname}.texi srctrigger=doc/${pkgname}.texi
@ -168,7 +168,7 @@ echo "LDFLAGS = ${LDFLAGS}"
rm -f Makefile rm -f Makefile
cat > Makefile << EOF cat > Makefile << EOF
# Makefile for Clzip - LZMA lossless data compressor # Makefile for Clzip - LZMA lossless data compressor
# Copyright (C) 2010-2017 Antonio Diaz Diaz. # Copyright (C) 2010-2018 Antonio Diaz Diaz.
# This file was generated automatically by configure. Don't edit. # This file was generated automatically by configure. Don't edit.
# #
# This Makefile is free software: you have unlimited permission # This Makefile is free software: you have unlimited permission

70
decoder.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -29,24 +29,6 @@
#include "decoder.h" #include "decoder.h"
void Pp_show_msg( struct Pretty_print * const pp, const char * const msg )
{
if( verbosity >= 0 )
{
if( pp->first_post )
{
unsigned i;
pp->first_post = false;
fprintf( stderr, " %s: ", pp->name );
for( i = strlen( pp->name ); i < pp->longest_name; ++i )
fputc( ' ', stderr );
if( !msg ) fflush( stderr );
}
if( msg ) fprintf( stderr, "%s\n", msg );
}
}
/* Returns the number of bytes really read. /* Returns the number of bytes really read.
If (returned value < size) and (errno == 0), means EOF was reached. If (returned value < size) and (errno == 0), means EOF was reached.
*/ */
@ -94,6 +76,7 @@ bool Rd_read_block( struct Range_decoder * const rdec )
rdec->at_stream_end = ( rdec->stream_pos < rd_buffer_size ); rdec->at_stream_end = ( rdec->stream_pos < rd_buffer_size );
rdec->partial_member_pos += rdec->pos; rdec->partial_member_pos += rdec->pos;
rdec->pos = 0; rdec->pos = 0;
show_dprogress( 0, 0, 0, 0 );
} }
return rdec->pos < rdec->stream_pos; return rdec->pos < rdec->stream_pos;
} }
@ -122,6 +105,8 @@ static bool LZd_verify_trailer( struct LZ_decoder * const d,
int size = Rd_read_data( d->rdec, trailer, Ft_size ); int size = Rd_read_data( d->rdec, trailer, Ft_size );
const unsigned long long data_size = LZd_data_position( d ); const unsigned long long data_size = LZd_data_position( d );
const unsigned long long member_size = Rd_member_position( d->rdec ); const unsigned long long member_size = Rd_member_position( d->rdec );
unsigned td_crc;
unsigned long long td_size, tm_size;
bool error = false; bool error = false;
if( size < Ft_size ) if( size < Ft_size )
@ -136,45 +121,56 @@ static bool LZd_verify_trailer( struct LZ_decoder * const d,
while( size < Ft_size ) trailer[size++] = 0; while( size < Ft_size ) trailer[size++] = 0;
} }
if( Ft_get_data_crc( trailer ) != LZd_crc( d ) ) td_crc = Ft_get_data_crc( trailer );
if( td_crc != LZd_crc( d ) )
{ {
error = true; error = true;
if( verbosity >= 0 ) if( verbosity >= 0 )
{ {
Pp_show_msg( pp, 0 ); Pp_show_msg( pp, 0 );
fprintf( stderr, "CRC mismatch; trailer says %08X, data CRC is %08X\n", fprintf( stderr, "CRC mismatch; stored %08X, computed %08X\n",
Ft_get_data_crc( trailer ), LZd_crc( d ) ); td_crc, LZd_crc( d ) );
} }
} }
if( Ft_get_data_size( trailer ) != data_size ) td_size = Ft_get_data_size( trailer );
if( td_size != data_size )
{ {
error = true; error = true;
if( verbosity >= 0 ) if( verbosity >= 0 )
{ {
Pp_show_msg( pp, 0 ); Pp_show_msg( pp, 0 );
fprintf( stderr, "Data size mismatch; trailer says %llu, data size is %llu (0x%llX)\n", fprintf( stderr, "Data size mismatch; stored %llu (0x%llX), computed %llu (0x%llX)\n",
Ft_get_data_size( trailer ), data_size, data_size ); td_size, td_size, data_size, data_size );
} }
} }
if( Ft_get_member_size( trailer ) != member_size ) tm_size = Ft_get_member_size( trailer );
if( tm_size != member_size )
{ {
error = true; error = true;
if( verbosity >= 0 ) if( verbosity >= 0 )
{ {
Pp_show_msg( pp, 0 ); Pp_show_msg( pp, 0 );
fprintf( stderr, "Member size mismatch; trailer says %llu, member size is %llu (0x%llX)\n", fprintf( stderr, "Member size mismatch; stored %llu (0x%llX), computed %llu (0x%llX)\n",
Ft_get_member_size( trailer ), member_size, member_size ); tm_size, tm_size, member_size, member_size );
} }
} }
if( !error && verbosity >= 2 && data_size > 0 && member_size > 0 ) if( error ) return false;
fprintf( stderr, "%6.3f:1, %6.3f bits/byte, %5.2f%% saved. ", if( verbosity >= 2 )
(double)data_size / member_size, {
( 8.0 * member_size ) / data_size, if( verbosity >= 4 ) show_header( d->dictionary_size );
100.0 * ( 1.0 - ( (double)member_size / data_size ) ) ); if( data_size == 0 || member_size == 0 )
if( !error && verbosity >= 4 ) fputs( "no data compressed. ", stderr );
fprintf( stderr, "CRC %08X, decompressed %9llu, compressed %8llu. ", else
LZd_crc( d ), data_size, member_size ); fprintf( stderr, "%6.3f:1, %5.2f%% ratio, %5.2f%% saved. ",
return !error; (double)data_size / member_size,
( 100.0 * member_size ) / data_size,
100.0 - ( ( 100.0 * member_size ) / data_size ) );
if( verbosity >= 4 ) fprintf( stderr, "CRC %08X, ", td_crc );
if( verbosity >= 3 )
fprintf( stderr, "decompressed %9llu, compressed %8llu. ",
data_size, member_size );
}
return true;
} }

10
decoder.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -239,11 +239,7 @@ struct LZ_decoder
void LZd_flush_data( struct LZ_decoder * const d ); void LZd_flush_data( struct LZ_decoder * const d );
static inline uint8_t LZd_peek_prev( const struct LZ_decoder * const d ) static inline uint8_t LZd_peek_prev( const struct LZ_decoder * const d )
{ { return d->buffer[((d->pos > 0) ? d->pos : d->dictionary_size)-1]; }
if( d->pos > 0 ) return d->buffer[d->pos-1];
if( d->pos_wrapped ) return d->buffer[d->dictionary_size-1];
return 0; /* prev_byte of first byte */
}
static inline uint8_t LZd_peek( const struct LZ_decoder * const d, static inline uint8_t LZd_peek( const struct LZ_decoder * const d,
const unsigned distance ) const unsigned distance )
@ -305,6 +301,8 @@ static inline bool LZd_init( struct LZ_decoder * const d,
d->crc = 0xFFFFFFFFU; d->crc = 0xFFFFFFFFU;
d->outfd = ofd; d->outfd = ofd;
d->pos_wrapped = false; d->pos_wrapped = false;
/* prev_byte of first byte; also for LZd_peek( 0 ) on corrupt file */
d->buffer[d->dictionary_size-1] = 0;
return true; return true;
} }

9
doc/clzip.1
View file

@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1. .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1.
.TH CLZIP "1" "April 2017" "clzip 1.9" "User Commands" .TH CLZIP "1" "February 2018" "clzip 1.10" "User Commands"
.SH NAME .SH NAME
clzip \- reduces the size of files clzip \- reduces the size of files
.SH SYNOPSIS .SH SYNOPSIS
@ -52,7 +52,7 @@ suppress all messages
set dictionary size limit in bytes [8 MiB] set dictionary size limit in bytes [8 MiB]
.TP .TP
\fB\-S\fR, \fB\-\-volume\-size=\fR<bytes> \fB\-S\fR, \fB\-\-volume\-size=\fR<bytes>
set volume size limit in bytes set volume size limit in bytes, implies \fB\-k\fR
.TP .TP
\fB\-t\fR, \fB\-\-test\fR \fB\-t\fR, \fB\-\-test\fR
test compressed file integrity test compressed file integrity
@ -68,6 +68,9 @@ alias for \fB\-0\fR
.TP .TP
\fB\-\-best\fR \fB\-\-best\fR
alias for \fB\-9\fR alias for \fB\-9\fR
.TP
\fB\-\-loose\-trailing\fR
allow trailing data seeming corrupt header
.PP .PP
If no file names are given, or if a file is '\-', clzip compresses or If no file names are given, or if a file is '\-', clzip compresses or
decompresses from standard input to standard output. decompresses from standard input to standard output.
@ -90,7 +93,7 @@ Report bugs to lzip\-bug@nongnu.org
.br .br
Clzip home page: http://www.nongnu.org/lzip/clzip.html Clzip home page: http://www.nongnu.org/lzip/clzip.html
.SH COPYRIGHT .SH COPYRIGHT
Copyright \(co 2017 Antonio Diaz Diaz. Copyright \(co 2018 Antonio Diaz Diaz.
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html> License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>
.br .br
This is free software: you are free to change and redistribute it. This is free software: you are free to change and redistribute it.

283
doc/clzip.info
View file

@ -11,11 +11,12 @@ File: clzip.info, Node: Top, Next: Introduction, Up: (dir)
Clzip Manual Clzip Manual
************ ************
This manual is for Clzip (version 1.9, 13 April 2017). This manual is for Clzip (version 1.10, 6 February 2018).
* Menu: * Menu:
* Introduction:: Purpose and features of clzip * Introduction:: Purpose and features of clzip
* Output:: Meaning of clzip's output
* Invoking clzip:: Command line interface * Invoking clzip:: Command line interface
* Quality assurance:: Design, development and testing of lzip * Quality assurance:: Design, development and testing of lzip
* File format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
@ -28,13 +29,13 @@ This manual is for Clzip (version 1.9, 13 April 2017).
* Concept index:: Index of concepts * Concept index:: Index of concepts
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission to This manual is free documentation: you have unlimited permission to
copy, distribute and modify it. copy, distribute and modify it.
 
File: clzip.info, Node: Introduction, Next: Invoking clzip, Prev: Top, Up: Top File: clzip.info, Node: Introduction, Next: Output, Prev: Top, Up: Top
1 Introduction 1 Introduction
************** **************
@ -55,7 +56,7 @@ archiving, taking into account both data integrity and decoder
availability: availability:
* The lzip format provides very safe integrity checking and some data * The lzip format provides very safe integrity checking and some data
recovery means. The lziprecover program can repair bit-flip errors recovery means. The lziprecover program can repair bit flip errors
(one of the most common forms of data corruption) in lzip files, (one of the most common forms of data corruption) in lzip files,
and provides data recovery capabilities, including error-checked and provides data recovery capabilities, including error-checked
merging of damaged copies of a file. *Note Data safety: merging of damaged copies of a file. *Note Data safety:
@ -129,7 +130,7 @@ entirely incomprehensible and therefore pointless.
Clzip will correctly decompress a file which is the concatenation of Clzip will correctly decompress a file which is the concatenation of
two or more compressed files. The result is the concatenation of the two or more compressed files. The result is the concatenation of the
corresponding uncompressed files. Integrity testing of concatenated corresponding decompressed files. Integrity testing of concatenated
compressed files is also supported. compressed files is also supported.
Clzip can produce multimember files, and lziprecover can safely Clzip can produce multimember files, and lziprecover can safely
@ -142,14 +143,58 @@ multivolume compressed tar archives.
automatically creating multimember output. The members so created are automatically creating multimember output. The members so created are
large, about 2 PiB each. large, about 2 PiB each.

File: clzip.info, Node: Output, Next: Invoking clzip, Prev: Introduction, Up: Top
2 Meaning of clzip's output
***************************
The output of clzip looks like this:
clzip -v foo
foo: 6.676:1, 14.98% ratio, 85.02% saved, 450560 in, 67493 out.
clzip -tvv foo.lz
foo.lz: 6.676:1, 14.98% ratio, 85.02% saved. ok
The meaning of each field is as follows:
'N:1'
The compression ratio (uncompressed_size / compressed_size), shown
as N to 1.
'ratio'
The inverse compression ratio
(compressed_size / uncompressed_size), shown as a percentage. A
decimal ratio is easily obtained by moving the decimal point two
places to the left; 14.98% = 0.1498.
'saved'
The space saved by compression (1 - ratio), shown as a percentage.
'in'
The size of the uncompressed data. When decompressing or testing,
it is shown as 'decompressed'. Note that clzip always prints the
uncompressed size before the compressed size when compressing,
decompressing, testing or listing.
'out'
The size of the compressed data. When decompressing or testing, it
is shown as 'compressed'.
When decompressing or testing at verbosity level 4 (-vvvv), the
dictionary size used to compress the file and the CRC32 of the
uncompressed data are also shown.
LANGUAGE NOTE: Uncompressed = not compressed = plain data; it may LANGUAGE NOTE: Uncompressed = not compressed = plain data; it may
never have been compressed. Decompressed is used to refer to data which never have been compressed. Decompressed is used to refer to data which
have undergone the process of decompression. have undergone the process of decompression.
 
File: clzip.info, Node: Invoking clzip, Next: Quality assurance, Prev: Introduction, Up: Top File: clzip.info, Node: Invoking clzip, Next: Quality assurance, Prev: Output, Up: Top
2 Invoking clzip 3 Invoking clzip
**************** ****************
The format for running clzip is: The format for running clzip is:
@ -160,7 +205,7 @@ The format for running clzip is:
other FILES and is read just once, the first time it appears in the other FILES and is read just once, the first time it appears in the
command line. command line.
Clzip supports the following options: clzip supports the following options:
'-h' '-h'
'--help' '--help'
@ -179,9 +224,10 @@ command line.
'-b BYTES' '-b BYTES'
'--member-size=BYTES' '--member-size=BYTES'
Set the member size limit to BYTES. A small member size may When compressing, set the member size limit to BYTES. A small
degrade compression ratio, so use it only when needed. Valid values member size may degrade compression ratio, so use it only when
range from 100 kB to 2 PiB. Defaults to 2 PiB. needed. Valid values range from 100 kB to 2 PiB. Defaults to
2 PiB.
'-c' '-c'
'--stdout' '--stdout'
@ -189,15 +235,15 @@ command line.
unchanged. If compressing several files, each file is compressed unchanged. If compressing several files, each file is compressed
independently. This option is needed when reading from a named independently. This option is needed when reading from a named
pipe (fifo) or from a device. Use it also to recover as much of pipe (fifo) or from a device. Use it also to recover as much of
the uncompressed data as possible when decompressing a corrupt the decompressed data as possible when decompressing a corrupt
file. file.
'-d' '-d'
'--decompress' '--decompress'
Decompress the specified file(s). If a file does not exist or Decompress the specified files. If a file does not exist or can't
can't be opened, clzip continues decompressing the rest of the be opened, clzip continues decompressing the rest of the files. If
files. If a file fails to decompress, clzip exits immediately a file fails to decompress, or is a terminal, clzip exits
without decompressing the rest of the files. immediately without decompressing the rest of the files.
'-f' '-f'
'--force' '--force'
@ -205,8 +251,8 @@ command line.
'-F' '-F'
'--recompress' '--recompress'
Force re-compression of files whose name already has the '.lz' or When compressing, force re-compression of files whose name already
'.tlz' suffix. has the '.lz' or '.tlz' suffix.
'-k' '-k'
'--keep' '--keep'
@ -216,7 +262,7 @@ command line.
'-l' '-l'
'--list' '--list'
Print the uncompressed size, compressed size and percentage saved Print the uncompressed size, compressed size and percentage saved
of the specified file(s). Trailing data are ignored. The values of the specified files. Trailing data are ignored. The values
produced are correct even for multimember files. If more than one produced are correct even for multimember files. If more than one
file is given, a final line containing the cumulative sizes is file is given, a final line containing the cumulative sizes is
printed. With '-v', the dictionary size, the number of members in printed. With '-v', the dictionary size, the number of members in
@ -230,19 +276,20 @@ command line.
'-m BYTES' '-m BYTES'
'--match-length=BYTES' '--match-length=BYTES'
Set the match length limit in bytes. After a match this long is When compressing, set the match length limit in bytes. After a
found, the search is finished. Valid values range from 5 to 273. match this long is found, the search is finished. Valid values
Larger values usually give better compression ratios but longer range from 5 to 273. Larger values usually give better compression
compression times. ratios but longer compression times.
'-o FILE' '-o FILE'
'--output=FILE' '--output=FILE'
When reading from standard input and '--stdout' has not been When reading from standard input and '--stdout' has not been
specified, use 'FILE' as the virtual name of the uncompressed specified, use 'FILE' as the virtual name of the uncompressed
file. This produces a file named 'FILE' when decompressing, a file file. This produces a file named 'FILE' when decompressing, or a
named 'FILE.lz' when compressing, and several files named file named 'FILE.lz' when compressing. A second '.lz' extension is
'FILE00001.lz', 'FILE00002.lz', etc, when compressing and not added if 'FILE' already ends in '.lz' or '.tlz'. When
splitting the output in volumes. compressing and splitting the output in volumes, several files
named 'FILE00001.lz', 'FILE00002.lz', etc, are created.
'-q' '-q'
'--quiet' '--quiet'
@ -250,13 +297,13 @@ command line.
'-s BYTES' '-s BYTES'
'--dictionary-size=BYTES' '--dictionary-size=BYTES'
Set the dictionary size limit in bytes. Clzip will use the smallest When compressing, set the dictionary size limit in bytes. Clzip
possible dictionary size for each file without exceeding this will use the smallest possible dictionary size for each file
limit. Valid values range from 4 KiB to 512 MiB. Values 12 to 29 without exceeding this limit. Valid values range from 4 KiB to
are interpreted as powers of two, meaning 2^12 to 2^29 bytes. Note 512 MiB. Values 12 to 29 are interpreted as powers of two, meaning
that dictionary sizes are quantized. If the specified size does 2^12 to 2^29 bytes. Note that dictionary sizes are quantized. If
not match one of the valid sizes, it will be rounded upwards by the specified size does not match one of the valid sizes, it will
adding up to (BYTES / 8) to it. be rounded upwards by adding up to (BYTES / 8) to it.
For maximum compression you should use a dictionary size limit as For maximum compression you should use a dictionary size limit as
large as possible, but keep in mind that the decompression memory large as possible, but keep in mind that the decompression memory
@ -265,38 +312,40 @@ command line.
'-S BYTES' '-S BYTES'
'--volume-size=BYTES' '--volume-size=BYTES'
Split the compressed output into several volume files with names When compressing, split the compressed output into several volume
'original_name00001.lz', 'original_name00002.lz', etc, and set the files with names 'original_name00001.lz', 'original_name00002.lz',
volume size limit to BYTES. Each volume is a complete, maybe etc, and set the volume size limit to BYTES. Input files are kept
multimember, lzip file. A small volume size may degrade compression unchanged. Each volume is a complete, maybe multimember, lzip
ratio, so use it only when needed. Valid values range from 100 kB file. A small volume size may degrade compression ratio, so use it
to 4 EiB. only when needed. Valid values range from 100 kB to 4 EiB.
'-t' '-t'
'--test' '--test'
Check integrity of the specified file(s), but don't decompress Check integrity of the specified files, but don't decompress them.
them. This really performs a trial decompression and throws away This really performs a trial decompression and throws away the
the result. Use it together with '-v' to see information about result. Use it together with '-v' to see information about the
the file(s). If a file fails the test, does not exist, can't be files. If a file fails the test, does not exist, can't be opened,
opened, or is a terminal, clzip continues checking the rest of the or is a terminal, clzip continues checking the rest of the files.
files. A final diagnostic is shown at verbosity level 1 or higher if any
file fails the test when testing multiple files.
'-v' '-v'
'--verbose' '--verbose'
Verbose mode. Verbose mode.
When compressing, show the compression ratio for each file When compressing, show the compression ratio and size for each file
processed. A second '-v' shows the progress of compression. processed.
When decompressing or testing, further -v's (up to 4) increase the When decompressing or testing, further -v's (up to 4) increase the
verbosity level, showing status, compression ratio, dictionary verbosity level, showing status, compression ratio, dictionary
size, trailer contents (CRC, data size, member size), and up to 6 size, trailer contents (CRC, data size, member size), and up to 6
bytes of trailing data (if any) both in hexadecimal and as a bytes of trailing data (if any) both in hexadecimal and as a
string of printable ASCII characters. string of printable ASCII characters.
Two or more '-v' options show the progress of (de)compression.
'-0 .. -9' '-0 .. -9'
Set the compression parameters (dictionary size and match length Set the compression parameters (dictionary size and match length
limit) as shown in the table below. The default compression level limit) as shown in the table below. The default compression level
is '-6'. Note that '-9' can be much slower than '-0'. These is '-6'. Note that '-9' can be much slower than '-0'. These
options have no effect when decompressing. options have no effect when decompressing, testing or listing.
The bidimensional parameter space of LZMA can't be mapped to a The bidimensional parameter space of LZMA can't be mapped to a
linear scale optimal for all files. If your files are large, very linear scale optimal for all files. If your files are large, very
@ -319,6 +368,13 @@ command line.
'--best' '--best'
Aliases for GNU gzip compatibility. Aliases for GNU gzip compatibility.
'--loose-trailing'
When decompressing, testing or listing, allow trailing data whose
first bytes are so similar to the magic bytes of a lzip header
that they can be confused with a corrupt header. Use this option
if a file triggers a "corrupt header" error and the cause is not
indeed a corrupt header.
Numbers given as arguments to options may be followed by a multiplier Numbers given as arguments to options may be followed by a multiplier
and an optional 'B' for "byte". and an optional 'B' for "byte".
@ -344,7 +400,7 @@ caused clzip to panic.
 
File: clzip.info, Node: Quality assurance, Next: File format, Prev: Invoking clzip, Up: Top File: clzip.info, Node: Quality assurance, Next: File format, Prev: Invoking clzip, Up: Top
3 Design, development and testing of lzip 4 Design, development and testing of lzip
***************************************** *****************************************
There are two ways of constructing a software design: One way is to make There are two ways of constructing a software design: One way is to make
@ -359,7 +415,7 @@ describes the lessons learned from previous compressors (gzip and
bzip2), and their application to the design of lzip. bzip2), and their application to the design of lzip.
3.1 Format design 4.1 Format design
================= =================
When gzip was designed in 1992, computers and operating systems were When gzip was designed in 1992, computers and operating systems were
@ -377,7 +433,7 @@ one of gzip.
Probably the worst defect of the gzip format from the point of view Probably the worst defect of the gzip format from the point of view
of data safety is the variable size of its header. If the byte at of data safety is the variable size of its header. If the byte at
offset 3 (flags) of a gzip member gets corrupted, it may become very offset 3 (flags) of a gzip member gets corrupted, it may become
difficult to recover the data, even if the compressed blocks are difficult to recover the data, even if the compressed blocks are
intact, because it can't be known with certainty where the compressed intact, because it can't be known with certainty where the compressed
blocks begin. blocks begin.
@ -399,8 +455,8 @@ error detection. Any distance larger than the dictionary size acts as a
forbidden symbol, allowing the decompressor to detect the approximate forbidden symbol, allowing the decompressor to detect the approximate
position of errors, and leaving very little work for the check sequence position of errors, and leaving very little work for the check sequence
(CRC and data sizes) in the detection of errors. Lzip is usually able (CRC and data sizes) in the detection of errors. Lzip is usually able
to detect all posible bit-flips in the compressed data without to detect all posible bit flips in the compressed data without
resorting to the check sequence. It would be very difficult to write an resorting to the check sequence. It would be difficult to write an
automatic recovery tool like lziprecover for the gzip format. And, as automatic recovery tool like lziprecover for the gzip format. And, as
far as I know, it has never been written. far as I know, it has never been written.
@ -409,15 +465,14 @@ decompressed data because it provides more accurate error detection than
CRC64 up to a compressed size of about 16 GiB, a size larger than that CRC64 up to a compressed size of about 16 GiB, a size larger than that
of most files. In the case of lzip, the additional detection capability of most files. In the case of lzip, the additional detection capability
of the decompressor reduces the probability of undetected errors more of the decompressor reduces the probability of undetected errors more
than a million times, making CRC32 more accurate than CRC64 up to about than a million times beyond what the CRC32 alone provides.
20 PiB of compressed size.
The lzip format is designed for long-term archiving. Therefore it The lzip format is designed for long-term archiving. Therefore it
excludes any unneeded features that may interfere with the future excludes any unneeded features that may interfere with the future
extraction of the uncompressed data. extraction of the decompressed data.
3.1.1 Gzip format (mis)features not present in lzip 4.1.1 Gzip format (mis)features not present in lzip
--------------------------------------------------- ---------------------------------------------------
'Multiple algorithms' 'Multiple algorithms'
@ -438,16 +493,22 @@ extraction of the uncompressed data.
compressed blocks. compressed blocks.
'Optional CRC for the header' 'Optional CRC for the header'
Using an optional checksum for the header is not only a bad idea, Using an optional CRC for the header is not only a bad idea, it is
it is an error; it may prevent the extraction of perfectly good an error; it circumvents the HD of the CRC and may prevent the
data. For example, if the checksum is used and the bit enabling it extraction of perfectly good data. For example, if the CRC is used
is reset by a bit-flip, the header will appear to be intact (in and the bit enabling it is reset by a bit flip, the header will
spite of being corrupt) while the compressed blocks will appear to appear to be intact (in spite of being corrupt) while the
be totally unrecoverable (in spite of being intact). Very compressed blocks will appear to be totally unrecoverable (in
misleading indeed. spite of being intact). Very misleading indeed.
'Metadata'
The gzip format stores some metadata, like the modification time
of the original file or the operating system on which compression
took place. This complicates reproducible compression (obtaining
identical compressed output from identical input).
3.1.2 Lzip format improvements over gzip and bzip2 4.1.2 Lzip format improvements over gzip and bzip2
-------------------------------------------------- --------------------------------------------------
'64-bit size field' '64-bit size field'
@ -475,7 +536,7 @@ extraction of the uncompressed data.
total uncompressed size. total uncompressed size.
3.2 Quality of implementation 4.2 Quality of implementation
============================= =============================
'Accurate and robust error detection' 'Accurate and robust error detection'
@ -521,7 +582,7 @@ extraction of the uncompressed data.
 
File: clzip.info, Node: File format, Next: Algorithm, Prev: Quality assurance, Up: Top File: clzip.info, Node: File format, Next: Algorithm, Prev: Quality assurance, Up: Top
4 File format 5 File format
************* *************
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -592,7 +653,7 @@ additional information before, between, or after them.
 
File: clzip.info, Node: Algorithm, Next: Stream format, Prev: File format, Up: Top File: clzip.info, Node: Algorithm, Next: Stream format, Prev: File format, Up: Top
5 Algorithm 6 Algorithm
*********** ***********
In spite of its name (Lempel-Ziv-Markov chain-Algorithm), LZMA is not a In spite of its name (Lempel-Ziv-Markov chain-Algorithm), LZMA is not a
@ -658,7 +719,7 @@ LZMA), and Julian Seward (for bzip2's CLI).
 
File: clzip.info, Node: Stream format, Next: Trailing data, Prev: Algorithm, Up: Top File: clzip.info, Node: Stream format, Next: Trailing data, Prev: Algorithm, Up: Top
6 Format of the LZMA stream in lzip files 7 Format of the LZMA stream in lzip files
***************************************** *****************************************
The LZMA algorithm has three parameters, called "special LZMA The LZMA algorithm has three parameters, called "special LZMA
@ -698,7 +759,7 @@ the lzip download directory. The source code of lzd is included in
appendix A. *Note Reference source code::. appendix A. *Note Reference source code::.
6.1 What is coded 7.1 What is coded
================= =================
The LZMA stream includes literals, matches and repeated matches (matches The LZMA stream includes literals, matches and repeated matches (matches
@ -773,7 +834,7 @@ slot + direct_bits distances from 4 to 127
slot + (direct_bits - 4) + 4 bits distances from 128 to 2^32 - 1 slot + (direct_bits - 4) + 4 bits distances from 128 to 2^32 - 1
6.2 The coding contexts 7.2 The coding contexts
======================= =======================
These contexts ('Bit_model' in the source), are integers or arrays of These contexts ('Bit_model' in the source), are integers or arrays of
@ -863,7 +924,7 @@ difference is found, the rest of the byte is decoded using the normal
bit tree context. (See 'decode_matched' in the source). bit tree context. (See 'decode_matched' in the source).
6.3 The range decoder 7.3 The range decoder
===================== =====================
The LZMA stream is consumed one byte at a time by the range decoder. The LZMA stream is consumed one byte at a time by the range decoder.
@ -883,7 +944,7 @@ range decoder. This is done by shifting 5 bytes in the initialization of
source). source).
6.4 Decoding the LZMA stream 7.4 Decoding the LZMA stream
============================ ============================
After decoding the member header and obtaining the dictionary size, the After decoding the member header and obtaining the dictionary size, the
@ -896,7 +957,7 @@ Stream" marker is decoded.
 
File: clzip.info, Node: Trailing data, Next: Examples, Prev: Stream format, Up: Top File: clzip.info, Node: Trailing data, Next: Examples, Prev: Stream format, Up: Top
7 Extra data appended to the file 8 Extra data appended to the file
********************************* *********************************
Sometimes extra data are found appended to a lzip file after the last Sometimes extra data are found appended to a lzip file after the last
@ -908,10 +969,11 @@ member. Such trailing data may be:
* Useful data added by the user; a cryptographically secure hash, a * Useful data added by the user; a cryptographically secure hash, a
description of file contents, etc. It is safe to append any amount description of file contents, etc. It is safe to append any amount
of text to a lzip file as long as the text does not begin with the of text to a lzip file as long as none of the first four bytes of
string "LZIP", and does not contain any zero bytes (null the text match the corresponding byte in the string "LZIP", and
characters). Nonzero bytes and zero bytes can't be safely mixed in the text does not contain any zero bytes (null characters).
trailing data. Nonzero bytes and zero bytes can't be safely mixed in trailing
data.
* Garbage added by some not totally successful copy operation. * Garbage added by some not totally successful copy operation.
@ -919,12 +981,17 @@ member. Such trailing data may be:
and hash value (for a chosen hash) coincide with those of another and hash value (for a chosen hash) coincide with those of another
file. file.
* In very rare cases, trailing data could be the corrupt header of * In rare cases, trailing data could be the corrupt header of another
another member. In multimember or concatenated files the member. In multimember or concatenated files the probability of
probability of corruption happening in the magic bytes is 5 times corruption happening in the magic bytes is 5 times smaller than the
smaller than the probability of getting a false positive caused by probability of getting a false positive caused by the corruption
the corruption of the integrity information itself. Therefore it of the integrity information itself. Therefore it can be
can be considered to be below the noise level. considered to be below the noise level. Additionally, the test
used by clzip to discriminate trailing data from a corrupt header
has a Hamming distance (HD) of 3, and the 3 bit flips must happen
in different magic bytes for the test to fail. In any case, the
option '--trailing-error' guarantees that any corrupt header will
be detected.
Trailing data are in no way part of the lzip file format, but tools Trailing data are in no way part of the lzip file format, but tools
reading lzip files are expected to behave as correctly and usefully as reading lzip files are expected to behave as correctly and usefully as
@ -938,7 +1005,7 @@ cases where a file containing trailing data must be rejected, the option
 
File: clzip.info, Node: Examples, Next: Problems, Prev: Trailing data, Up: Top File: clzip.info, Node: Examples, Next: Problems, Prev: Trailing data, Up: Top
8 A small tutorial with examples 9 A small tutorial with examples
******************************** ********************************
WARNING! Even if clzip is bug-free, other causes may result in a corrupt WARNING! Even if clzip is bug-free, other causes may result in a corrupt
@ -1020,8 +1087,8 @@ file with a member size of 32 MiB.
 
File: clzip.info, Node: Problems, Next: Reference source code, Prev: Examples, Up: Top File: clzip.info, Node: Problems, Next: Reference source code, Prev: Examples, Up: Top
9 Reporting bugs 10 Reporting bugs
**************** *****************
There are probably bugs in clzip. There are certainly errors and There are probably bugs in clzip. There are certainly errors and
omissions in this manual. If you report them, they will get fixed. If omissions in this manual. If you report them, they will get fixed. If
@ -1039,7 +1106,7 @@ Appendix A Reference source code
******************************** ********************************
/* Lzd - Educational decompressor for the lzip format /* Lzd - Educational decompressor for the lzip format
Copyright (C) 2013-2017 Antonio Diaz Diaz. Copyright (C) 2013-2018 Antonio Diaz Diaz.
This program is free software. Redistribution and use in source and This program is free software. Redistribution and use in source and
binary forms, with or without modification, are permitted provided binary forms, with or without modification, are permitted provided
@ -1355,9 +1422,9 @@ bool LZ_decoder::decode_member() // Returns false if error
Bit_model bm_align[dis_align_size]; Bit_model bm_align[dis_align_size];
Len_model match_len_model; Len_model match_len_model;
Len_model rep_len_model; Len_model rep_len_model;
unsigned rep0 = 0; // rep[0-3] latest four distances unsigned rep0 = 0; // rep[0-3] latest four distances
unsigned rep1 = 0; // used for efficient coding of unsigned rep1 = 0; // used for efficient coding of
unsigned rep2 = 0; // repeated distances unsigned rep2 = 0; // repeated distances
unsigned rep3 = 0; unsigned rep3 = 0;
State state; State state;
@ -1452,7 +1519,7 @@ int main( const int argc, const char * const argv[] )
"It is not safe to use lzd for any real work.\n" "It is not safe to use lzd for any real work.\n"
"\nUsage: %s < file.lz > file\n", argv[0] ); "\nUsage: %s < file.lz > file\n", argv[0] );
std::printf( "Lzd decompresses from standard input to standard output.\n" std::printf( "Lzd decompresses from standard input to standard output.\n"
"\nCopyright (C) 2017 Antonio Diaz Diaz.\n" "\nCopyright (C) 2018 Antonio Diaz Diaz.\n"
"This is free software: you are free to change and redistribute it.\n" "This is free software: you are free to change and redistribute it.\n"
"There is NO WARRANTY, to the extent permitted by law.\n" "There is NO WARRANTY, to the extent permitted by law.\n"
"Report bugs to lzip-bug@nongnu.org\n" "Report bugs to lzip-bug@nongnu.org\n"
@ -1497,7 +1564,7 @@ int main( const int argc, const char * const argv[] )
} }
if( std::fclose( stdout ) != 0 ) if( std::fclose( stdout ) != 0 )
{ std::fprintf( stderr, "Can't close stdout: %s\n", std::strerror( errno ) ); { std::fprintf( stderr, "Error closing stdout: %s\n", std::strerror( errno ) );
return 1; } return 1; }
return 0; return 0;
} }
@ -1520,6 +1587,7 @@ Concept index
* introduction: Introduction. (line 6) * introduction: Introduction. (line 6)
* invoking: Invoking clzip. (line 6) * invoking: Invoking clzip. (line 6)
* options: Invoking clzip. (line 6) * options: Invoking clzip. (line 6)
* output: Output. (line 6)
* quality assurance: Quality assurance. (line 6) * quality assurance: Quality assurance. (line 6)
* reference source code: Reference source code. (line 6) * reference source code: Reference source code. (line 6)
* trailing data: Trailing data. (line 6) * trailing data: Trailing data. (line 6)
@ -1530,19 +1598,20 @@ Concept index
 
Tag Table: Tag Table:
Node: Top210 Node: Top210
Node: Introduction1154 Node: Introduction1210
Node: Invoking clzip6630 Node: Output6491
Ref: --trailing-error7202 Node: Invoking clzip8011
Node: Quality assurance14125 Ref: --trailing-error8577
Node: File format22281 Node: Quality assurance16230
Node: Algorithm24686 Node: File format24640
Node: Stream format27516 Node: Algorithm27045
Node: Trailing data38257 Node: Stream format29875
Node: Examples40159 Node: Trailing data40616
Ref: concat-example41341 Node: Examples42894
Node: Problems42386 Ref: concat-example44076
Node: Reference source code42920 Node: Problems45121
Node: Concept index57238 Node: Reference source code45657
Node: Concept index59974
 
End Tag Table End Tag Table

235
doc/clzip.texi
View file

@ -6,8 +6,8 @@
@finalout @finalout
@c %**end of header @c %**end of header
@set UPDATED 13 April 2017 @set UPDATED 6 February 2018
@set VERSION 1.9 @set VERSION 1.10
@dircategory Data Compression @dircategory Data Compression
@direntry @direntry
@ -36,6 +36,7 @@ This manual is for Clzip (version @value{VERSION}, @value{UPDATED}).
@menu @menu
* Introduction:: Purpose and features of clzip * Introduction:: Purpose and features of clzip
* Output:: Meaning of clzip's output
* Invoking clzip:: Command line interface * Invoking clzip:: Command line interface
* Quality assurance:: Design, development and testing of lzip * Quality assurance:: Design, development and testing of lzip
* File format:: Detailed format of the compressed file * File format:: Detailed format of the compressed file
@ -49,7 +50,7 @@ This manual is for Clzip (version @value{VERSION}, @value{UPDATED}).
@end menu @end menu
@sp 1 @sp 1
Copyright @copyright{} 2010-2017 Antonio Diaz Diaz. Copyright @copyright{} 2010-2018 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission This manual is free documentation: you have unlimited permission
to copy, distribute and modify it. to copy, distribute and modify it.
@ -79,7 +80,7 @@ availability:
The lzip format provides very safe integrity checking and some data The lzip format provides very safe integrity checking and some data
recovery means. The recovery means. The
@uref{http://www.nongnu.org/lzip/manual/lziprecover_manual.html#Data-safety,,lziprecover} @uref{http://www.nongnu.org/lzip/manual/lziprecover_manual.html#Data-safety,,lziprecover}
program can repair bit-flip errors (one of the most common forms of data program can repair bit flip errors (one of the most common forms of data
corruption) in lzip files, and provides data recovery capabilities, corruption) in lzip files, and provides data recovery capabilities,
including error-checked merging of damaged copies of a file. including error-checked merging of damaged copies of a file.
@ifnothtml @ifnothtml
@ -128,9 +129,9 @@ choice of dictionary size limit.
The amount of memory required for compression is about 1 or 2 times the The amount of memory required for compression is about 1 or 2 times the
dictionary size limit (1 if input file size is less than dictionary size dictionary size limit (1 if input file size is less than dictionary size
limit, else 2) plus 9 times the dictionary size really used. The option limit, else 2) plus 9 times the dictionary size really used. The option
@samp{-0} is special and only requires about 1.5 MiB at most. The amount @samp{-0} is special and only requires about @w{1.5 MiB} at most. The
of memory required for decompression is about 46 kB larger than the amount of memory required for decompression is about @w{46 kB} larger
dictionary size really used. than the dictionary size really used.
When compressing, clzip replaces every file given in the command line When compressing, clzip replaces every file given in the command line
with a compressed version of itself, with the name "original_name.lz". with a compressed version of itself, with the name "original_name.lz".
@ -159,7 +160,7 @@ incomprehensible and therefore pointless.
Clzip will correctly decompress a file which is the concatenation of two Clzip will correctly decompress a file which is the concatenation of two
or more compressed files. The result is the concatenation of the or more compressed files. The result is the concatenation of the
corresponding uncompressed files. Integrity testing of concatenated corresponding decompressed files. Integrity testing of concatenated
compressed files is also supported. compressed files is also supported.
Clzip can produce multimember files, and lziprecover can safely recover Clzip can produce multimember files, and lziprecover can safely recover
@ -170,7 +171,53 @@ compressed tar archives.
Clzip is able to compress and decompress streams of unlimited size by Clzip is able to compress and decompress streams of unlimited size by
automatically creating multimember output. The members so created are automatically creating multimember output. The members so created are
large, about 2 PiB each. large, about @w{2 PiB} each.
@node Output
@chapter Meaning of clzip's output
@cindex output
The output of clzip looks like this:
@example
clzip -v foo
foo: 6.676:1, 14.98% ratio, 85.02% saved, 450560 in, 67493 out.
clzip -tvv foo.lz
foo.lz: 6.676:1, 14.98% ratio, 85.02% saved. ok
@end example
The meaning of each field is as follows:
@table @code
@item N:1
The compression ratio @w{(uncompressed_size / compressed_size)}, shown
as N to 1.
@item ratio
The inverse compression ratio @w{(compressed_size / uncompressed_size)},
shown as a percentage. A decimal ratio is easily obtained by moving the
decimal point two places to the left; @w{14.98% = 0.1498}.
@item saved
The space saved by compression @w{(1 - ratio)}, shown as a percentage.
@item in
The size of the uncompressed data. When decompressing or testing, it is
shown as @code{decompressed}. Note that clzip always prints the
uncompressed size before the compressed size when compressing,
decompressing, testing or listing.
@item out
The size of the compressed data. When decompressing or testing, it is
shown as @code{compressed}.
@end table
When decompressing or testing at verbosity level 4 (-vvvv), the
dictionary size used to compress the file and the CRC32 of the
uncompressed data are also shown.
LANGUAGE NOTE: Uncompressed = not compressed = plain data; it may never LANGUAGE NOTE: Uncompressed = not compressed = plain data; it may never
have been compressed. Decompressed is used to refer to data which have have been compressed. Decompressed is used to refer to data which have
@ -195,7 +242,7 @@ clzip [@var{options}] [@var{files}]
mixed with other @var{files} and is read just once, the first time it mixed with other @var{files} and is read just once, the first time it
appears in the command line. appears in the command line.
Clzip supports the following options: clzip supports the following options:
@table @code @table @code
@item -h @item -h
@ -215,24 +262,24 @@ garbage that can be safely ignored. @xref{concat-example}.
@item -b @var{bytes} @item -b @var{bytes}
@itemx --member-size=@var{bytes} @itemx --member-size=@var{bytes}
Set the member size limit to @var{bytes}. A small member size may When compressing, set the member size limit to @var{bytes}. A small
degrade compression ratio, so use it only when needed. Valid values member size may degrade compression ratio, so use it only when needed.
range from 100 kB to 2 PiB. Defaults to 2 PiB. Valid values range from @w{100 kB} to @w{2 PiB}. Defaults to @w{2 PiB}.
@item -c @item -c
@itemx --stdout @itemx --stdout
Compress or decompress to standard output; keep input files unchanged. Compress or decompress to standard output; keep input files unchanged.
If compressing several files, each file is compressed independently. If compressing several files, each file is compressed independently.
This option is needed when reading from a named pipe (fifo) or from a This option is needed when reading from a named pipe (fifo) or from a
device. Use it also to recover as much of the uncompressed data as device. Use it also to recover as much of the decompressed data as
possible when decompressing a corrupt file. possible when decompressing a corrupt file.
@item -d @item -d
@itemx --decompress @itemx --decompress
Decompress the specified file(s). If a file does not exist or can't be Decompress the specified files. If a file does not exist or can't be
opened, clzip continues decompressing the rest of the files. If a file opened, clzip continues decompressing the rest of the files. If a file
fails to decompress, clzip exits immediately without decompressing the fails to decompress, or is a terminal, clzip exits immediately without
rest of the files. decompressing the rest of the files.
@item -f @item -f
@itemx --force @itemx --force
@ -240,8 +287,8 @@ Force overwrite of output files.
@item -F @item -F
@itemx --recompress @itemx --recompress
Force re-compression of files whose name already has the @samp{.lz} or When compressing, force re-compression of files whose name already has
@samp{.tlz} suffix. the @samp{.lz} or @samp{.tlz} suffix.
@item -k @item -k
@itemx --keep @itemx --keep
@ -250,7 +297,7 @@ Keep (don't delete) input files during compression or decompression.
@item -l @item -l
@itemx --list @itemx --list
Print the uncompressed size, compressed size and percentage saved of the Print the uncompressed size, compressed size and percentage saved of the
specified file(s). Trailing data are ignored. The values produced are specified files. Trailing data are ignored. The values produced are
correct even for multimember files. If more than one file is given, a correct even for multimember files. If more than one file is given, a
final line containing the cumulative sizes is printed. With @samp{-v}, final line containing the cumulative sizes is printed. With @samp{-v},
the dictionary size, the number of members in the file, and the amount the dictionary size, the number of members in the file, and the amount
@ -263,18 +310,21 @@ verifies that none of the specified files contain trailing data.
@item -m @var{bytes} @item -m @var{bytes}
@itemx --match-length=@var{bytes} @itemx --match-length=@var{bytes}
Set the match length limit in bytes. After a match this long is found, When compressing, set the match length limit in bytes. After a match
the search is finished. Valid values range from 5 to 273. Larger values this long is found, the search is finished. Valid values range from 5 to
usually give better compression ratios but longer compression times. 273. Larger values usually give better compression ratios but longer
compression times.
@item -o @var{file} @item -o @var{file}
@itemx --output=@var{file} @itemx --output=@var{file}
When reading from standard input and @samp{--stdout} has not been When reading from standard input and @samp{--stdout} has not been
specified, use @samp{@var{file}} as the virtual name of the uncompressed specified, use @samp{@var{file}} as the virtual name of the uncompressed
file. This produces a file named @samp{@var{file}} when decompressing, a file. This produces a file named @samp{@var{file}} when decompressing,
file named @samp{@var{file}.lz} when compressing, and several files or a file named @samp{@var{file}.lz} when compressing. A second
named @samp{@var{file}00001.lz}, @samp{@var{file}00002.lz}, etc, when @samp{.lz} extension is not added if @samp{@var{file}} already ends in
compressing and splitting the output in volumes. @samp{.lz} or @samp{.tlz}. When compressing and splitting the output in
volumes, several files named @samp{@var{file}00001.lz},
@samp{@var{file}00002.lz}, etc, are created.
@item -q @item -q
@itemx --quiet @itemx --quiet
@ -282,12 +332,12 @@ Quiet operation. Suppress all messages.
@item -s @var{bytes} @item -s @var{bytes}
@itemx --dictionary-size=@var{bytes} @itemx --dictionary-size=@var{bytes}
Set the dictionary size limit in bytes. Clzip will use the smallest When compressing, set the dictionary size limit in bytes. Clzip will use
possible dictionary size for each file without exceeding this limit. the smallest possible dictionary size for each file without exceeding
Valid values range from 4 KiB to 512 MiB. Values 12 to 29 are this limit. Valid values range from @w{4 KiB} to @w{512 MiB}. Values 12
interpreted as powers of two, meaning 2^12 to 2^29 bytes. Note that to 29 are interpreted as powers of two, meaning 2^12 to 2^29 bytes. Note
dictionary sizes are quantized. If the specified size does not match one that dictionary sizes are quantized. If the specified size does not
of the valid sizes, it will be rounded upwards by adding up to match one of the valid sizes, it will be rounded upwards by adding up to
@w{(@var{bytes} / 8)} to it. @w{(@var{bytes} / 8)} to it.
For maximum compression you should use a dictionary size limit as large For maximum compression you should use a dictionary size limit as large
@ -296,37 +346,40 @@ is affected at compression time by the choice of dictionary size limit.
@item -S @var{bytes} @item -S @var{bytes}
@itemx --volume-size=@var{bytes} @itemx --volume-size=@var{bytes}
Split the compressed output into several volume files with names When compressing, split the compressed output into several volume files
@samp{original_name00001.lz}, @samp{original_name00002.lz}, etc, and set with names @samp{original_name00001.lz}, @samp{original_name00002.lz},
the volume size limit to @var{bytes}. Each volume is a complete, maybe etc, and set the volume size limit to @var{bytes}. Input files are kept
multimember, lzip file. A small volume size may degrade compression unchanged. Each volume is a complete, maybe multimember, lzip file. A
ratio, so use it only when needed. Valid values range from 100 kB to 4 small volume size may degrade compression ratio, so use it only when
EiB. needed. Valid values range from @w{100 kB} to @w{4 EiB}.
@item -t @item -t
@itemx --test @itemx --test
Check integrity of the specified file(s), but don't decompress them. Check integrity of the specified files, but don't decompress them. This
This really performs a trial decompression and throws away the result. really performs a trial decompression and throws away the result. Use it
Use it together with @samp{-v} to see information about the file(s). If together with @samp{-v} to see information about the files. If a file
a file fails the test, does not exist, can't be opened, or is a fails the test, does not exist, can't be opened, or is a terminal, clzip
terminal, clzip continues checking the rest of the files. continues checking the rest of the files. A final diagnostic is shown at
verbosity level 1 or higher if any file fails the test when testing
multiple files.
@item -v @item -v
@itemx --verbose @itemx --verbose
Verbose mode.@* Verbose mode.@*
When compressing, show the compression ratio for each file processed. A When compressing, show the compression ratio and size for each file
second @samp{-v} shows the progress of compression.@* processed.@*
When decompressing or testing, further -v's (up to 4) increase the When decompressing or testing, further -v's (up to 4) increase the
verbosity level, showing status, compression ratio, dictionary size, verbosity level, showing status, compression ratio, dictionary size,
trailer contents (CRC, data size, member size), and up to 6 bytes of trailer contents (CRC, data size, member size), and up to 6 bytes of
trailing data (if any) both in hexadecimal and as a string of printable trailing data (if any) both in hexadecimal and as a string of printable
ASCII characters. ASCII characters.@*
Two or more @samp{-v} options show the progress of (de)compression.
@item -0 .. -9 @item -0 .. -9
Set the compression parameters (dictionary size and match length limit) Set the compression parameters (dictionary size and match length limit)
as shown in the table below. The default compression level is @samp{-6}. as shown in the table below. The default compression level is @samp{-6}.
Note that @samp{-9} can be much slower than @samp{-0}. These options Note that @samp{-9} can be much slower than @samp{-0}. These options
have no effect when decompressing. have no effect when decompressing, testing or listing.
The bidimensional parameter space of LZMA can't be mapped to a linear The bidimensional parameter space of LZMA can't be mapped to a linear
scale optimal for all files. If your files are large, very repetitive, scale optimal for all files. If your files are large, very repetitive,
@ -351,6 +404,12 @@ etc, you may need to use the @samp{--dictionary-size} and
@itemx --best @itemx --best
Aliases for GNU gzip compatibility. Aliases for GNU gzip compatibility.
@item --loose-trailing
When decompressing, testing or listing, allow trailing data whose first
bytes are so similar to the magic bytes of a lzip header that they can
be confused with a corrupt header. Use this option if a file triggers a
"corrupt header" error and the cause is not indeed a corrupt header.
@end table @end table
Numbers given as arguments to options may be followed by a multiplier Numbers given as arguments to options may be followed by a multiplier
@ -410,7 +469,7 @@ of gzip.
Probably the worst defect of the gzip format from the point of view of Probably the worst defect of the gzip format from the point of view of
data safety is the variable size of its header. If the byte at offset 3 data safety is the variable size of its header. If the byte at offset 3
(flags) of a gzip member gets corrupted, it may become very difficult to (flags) of a gzip member gets corrupted, it may become difficult to
recover the data, even if the compressed blocks are intact, because it recover the data, even if the compressed blocks are intact, because it
can't be known with certainty where the compressed blocks begin. can't be known with certainty where the compressed blocks begin.
@ -431,22 +490,21 @@ distance larger than the dictionary size acts as a forbidden symbol,
allowing the decompressor to detect the approximate position of errors, allowing the decompressor to detect the approximate position of errors,
and leaving very little work for the check sequence (CRC and data sizes) and leaving very little work for the check sequence (CRC and data sizes)
in the detection of errors. Lzip is usually able to detect all posible in the detection of errors. Lzip is usually able to detect all posible
bit-flips in the compressed data without resorting to the check bit flips in the compressed data without resorting to the check
sequence. It would be very difficult to write an automatic recovery tool sequence. It would be difficult to write an automatic recovery tool like
like lziprecover for the gzip format. And, as far as I know, it has lziprecover for the gzip format. And, as far as I know, it has never
never been written. been written.
Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the
decompressed data because it provides more accurate error detection than decompressed data because it provides more accurate error detection than
CRC64 up to a compressed size of about 16 GiB, a size larger than that CRC64 up to a compressed size of about @w{16 GiB}, a size larger than
of most files. In the case of lzip, the additional detection capability that of most files. In the case of lzip, the additional detection
of the decompressor reduces the probability of undetected errors more capability of the decompressor reduces the probability of undetected
than a million times, making CRC32 more accurate than CRC64 up to about errors more than a million times beyond what the CRC32 alone provides.
20 PiB of compressed size.
The lzip format is designed for long-term archiving. Therefore it The lzip format is designed for long-term archiving. Therefore it
excludes any unneeded features that may interfere with the future excludes any unneeded features that may interfere with the future
extraction of the uncompressed data. extraction of the decompressed data.
@sp 1 @sp 1
@subsection Gzip format (mis)features not present in lzip @subsection Gzip format (mis)features not present in lzip
@ -472,12 +530,20 @@ header CRC nor the compressed blocks.
@item Optional CRC for the header @item Optional CRC for the header
Using an optional checksum for the header is not only a bad idea, it is Using an optional CRC for the header is not only a bad idea, it is an
an error; it may prevent the extraction of perfectly good data. For error; it circumvents the HD of the CRC and may prevent the extraction
example, if the checksum is used and the bit enabling it is reset by a of perfectly good data. For example, if the CRC is used and the bit
bit-flip, the header will appear to be intact (in spite of being enabling it is reset by a bit flip, the header will appear to be intact
corrupt) while the compressed blocks will appear to be totally (in spite of being corrupt) while the compressed blocks will appear to
unrecoverable (in spite of being intact). Very misleading indeed. be totally unrecoverable (in spite of being intact). Very misleading
indeed.
@item Metadata
The gzip format stores some metadata, like the modification time of the
original file or the operating system on which compression took place.
This complicates reproducible compression (obtaining identical
compressed output from identical input).
@end table @end table
@ -488,7 +554,7 @@ unrecoverable (in spite of being intact). Very misleading indeed.
Probably the most frequently reported shortcoming of the gzip format is Probably the most frequently reported shortcoming of the gzip format is
that it only stores the least significant 32 bits of the uncompressed that it only stores the least significant 32 bits of the uncompressed
size. The size of any file larger than 4 GiB gets truncated. size. The size of any file larger than @w{4 GiB} gets truncated.
Bzip2 does not store the uncompressed size of the file. Bzip2 does not store the uncompressed size of the file.
@ -965,9 +1031,10 @@ padding zero bytes to a lzip file.
@item @item
Useful data added by the user; a cryptographically secure hash, a Useful data added by the user; a cryptographically secure hash, a
description of file contents, etc. It is safe to append any amount of description of file contents, etc. It is safe to append any amount of
text to a lzip file as long as the text does not begin with the string text to a lzip file as long as none of the first four bytes of the text
"LZIP", and does not contain any zero bytes (null characters). Nonzero match the corresponding byte in the string "LZIP", and the text does not
bytes and zero bytes can't be safely mixed in trailing data. contain any zero bytes (null characters). Nonzero bytes and zero bytes
can't be safely mixed in trailing data.
@item @item
Garbage added by some not totally successful copy operation. Garbage added by some not totally successful copy operation.
@ -977,12 +1044,16 @@ Malicious data added to the file in order to make its total size and
hash value (for a chosen hash) coincide with those of another file. hash value (for a chosen hash) coincide with those of another file.
@item @item
In very rare cases, trailing data could be the corrupt header of another In rare cases, trailing data could be the corrupt header of another
member. In multimember or concatenated files the probability of member. In multimember or concatenated files the probability of
corruption happening in the magic bytes is 5 times smaller than the corruption happening in the magic bytes is 5 times smaller than the
probability of getting a false positive caused by the corruption of the probability of getting a false positive caused by the corruption of the
integrity information itself. Therefore it can be considered to be below integrity information itself. Therefore it can be considered to be below
the noise level. the noise level. Additionally, the test used by clzip to discriminate
trailing data from a corrupt header has a Hamming distance (HD) of 3,
and the 3 bit flips must happen in different magic bytes for the test to
fail. In any case, the option @samp{--trailing-error} guarantees that
any corrupt header will be detected.
@end itemize @end itemize
Trailing data are in no way part of the lzip file format, but tools Trailing data are in no way part of the lzip file format, but tools
@ -1018,7 +1089,7 @@ clzip -v file
@sp 1 @sp 1
@noindent @noindent
Example 2: Like example 1 but the created @samp{file.lz} is multimember Example 2: Like example 1 but the created @samp{file.lz} is multimember
with a member size of 1 MiB. The compression ratio is not shown. with a member size of @w{1 MiB}. The compression ratio is not shown.
@example @example
clzip -b 1MiB file clzip -b 1MiB file
@ -1067,7 +1138,7 @@ Do this instead
@sp 1 @sp 1
@noindent @noindent
Example 7: Decompress @samp{file.lz} partially until 10 KiB of Example 7: Decompress @samp{file.lz} partially until @w{10 KiB} of
decompressed data are produced. decompressed data are produced.
@example @example
@ -1086,7 +1157,7 @@ clzip -cd file.lz | dd bs=1000 skip=10 count=5
@sp 1 @sp 1
@noindent @noindent
Example 9: Create a multivolume compressed tar archive with a volume Example 9: Create a multivolume compressed tar archive with a volume
size of 1440 KiB. size of @w{1440 KiB}.
@example @example
tar -c some_directory | clzip -S 1440KiB -o volume_name tar -c some_directory | clzip -S 1440KiB -o volume_name
@ -1103,8 +1174,8 @@ clzip -cd volume_name*.lz | tar -xf -
@sp 1 @sp 1
@noindent @noindent
Example 11: Create a multivolume compressed backup of a large database Example 11: Create a multivolume compressed backup of a large database
file with a volume size of 650 MB, where each volume is a multimember file with a volume size of @w{650 MB}, where each volume is a
file with a member size of 32 MiB. multimember file with a member size of @w{32 MiB}.
@example @example
clzip -b 32MiB -S 650MB big_db clzip -b 32MiB -S 650MB big_db
@ -1132,7 +1203,7 @@ find by running @w{@code{clzip --version}}.
@verbatim @verbatim
/* Lzd - Educational decompressor for the lzip format /* Lzd - Educational decompressor for the lzip format
Copyright (C) 2013-2017 Antonio Diaz Diaz. Copyright (C) 2013-2018 Antonio Diaz Diaz.
This program is free software. Redistribution and use in source and This program is free software. Redistribution and use in source and
binary forms, with or without modification, are permitted provided binary forms, with or without modification, are permitted provided
@ -1448,9 +1519,9 @@ bool LZ_decoder::decode_member() // Returns false if error
Bit_model bm_align[dis_align_size]; Bit_model bm_align[dis_align_size];
Len_model match_len_model; Len_model match_len_model;
Len_model rep_len_model; Len_model rep_len_model;
unsigned rep0 = 0; // rep[0-3] latest four distances unsigned rep0 = 0; // rep[0-3] latest four distances
unsigned rep1 = 0; // used for efficient coding of unsigned rep1 = 0; // used for efficient coding of
unsigned rep2 = 0; // repeated distances unsigned rep2 = 0; // repeated distances
unsigned rep3 = 0; unsigned rep3 = 0;
State state; State state;
@ -1545,7 +1616,7 @@ int main( const int argc, const char * const argv[] )
"It is not safe to use lzd for any real work.\n" "It is not safe to use lzd for any real work.\n"
"\nUsage: %s < file.lz > file\n", argv[0] ); "\nUsage: %s < file.lz > file\n", argv[0] );
std::printf( "Lzd decompresses from standard input to standard output.\n" std::printf( "Lzd decompresses from standard input to standard output.\n"
"\nCopyright (C) 2017 Antonio Diaz Diaz.\n" "\nCopyright (C) 2018 Antonio Diaz Diaz.\n"
"This is free software: you are free to change and redistribute it.\n" "This is free software: you are free to change and redistribute it.\n"
"There is NO WARRANTY, to the extent permitted by law.\n" "There is NO WARRANTY, to the extent permitted by law.\n"
"Report bugs to lzip-bug@nongnu.org\n" "Report bugs to lzip-bug@nongnu.org\n"
@ -1590,7 +1661,7 @@ int main( const int argc, const char * const argv[] )
} }
if( std::fclose( stdout ) != 0 ) if( std::fclose( stdout ) != 0 )
{ std::fprintf( stderr, "Can't close stdout: %s\n", std::strerror( errno ) ); { std::fprintf( stderr, "Error closing stdout: %s\n", std::strerror( errno ) );
return 1; } return 1; }
return 0; return 0;
} }

2
encoder.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by

6
encoder.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -280,14 +280,14 @@ static inline bool LZe_init( struct LZ_encoder * const e,
const int dict_size, const int len_limit, const int dict_size, const int len_limit,
const int ifd, const int outfd ) const int ifd, const int outfd )
{ {
enum { before = max_num_trials, enum { before_size = max_num_trials,
/* bytes to keep in buffer after pos */ /* bytes to keep in buffer after pos */
after_size = ( 2 * max_match_len ) + 1, after_size = ( 2 * max_match_len ) + 1,
dict_factor = 2, dict_factor = 2,
num_prev_positions23 = num_prev_positions2 + num_prev_positions3, num_prev_positions23 = num_prev_positions2 + num_prev_positions3,
pos_array_factor = 2 }; pos_array_factor = 2 };
if( !LZeb_init( &e->eb, before, dict_size, after_size, dict_factor, if( !LZeb_init( &e->eb, before_size, dict_size, after_size, dict_factor,
num_prev_positions23, pos_array_factor, ifd, outfd ) ) num_prev_positions23, pos_array_factor, ifd, outfd ) )
return false; return false;
e->cycles = ( len_limit < max_match_len ) ? 16 + ( len_limit / 2 ) : 256; e->cycles = ( len_limit < max_match_len ) ? 16 + ( len_limit / 2 ) : 256;

29
encoder_base.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -69,21 +69,22 @@ void Mb_normalize_pos( struct Matchfinder_base * const mb )
} }
bool Mb_init( struct Matchfinder_base * const mb, const int before, bool Mb_init( struct Matchfinder_base * const mb, const int before_size,
const int dict_size, const int after_size, const int dict_size, const int after_size,
const int dict_factor, const int num_prev_positions23, const int dict_factor, const int num_prev_positions23,
const int pos_array_factor, const int ifd ) const int pos_array_factor, const int ifd )
{ {
const int buffer_size_limit = const int buffer_size_limit =
( dict_factor * dict_size ) + before + after_size; ( dict_factor * dict_size ) + before_size + after_size;
unsigned size; unsigned size;
int i; int i;
mb->partial_data_pos = 0; mb->partial_data_pos = 0;
mb->before_size = before; mb->before_size = before_size;
mb->pos = 0; mb->pos = 0;
mb->cyclic_pos = 0; mb->cyclic_pos = 0;
mb->stream_pos = 0; mb->stream_pos = 0;
mb->num_prev_positions23 = num_prev_positions23;
mb->infd = ifd; mb->infd = ifd;
mb->at_stream_end = false; mb->at_stream_end = false;
@ -111,8 +112,8 @@ bool Mb_init( struct Matchfinder_base * const mb, const int before,
size >>= 1; size >>= 1;
mb->key4_mask = size - 1; mb->key4_mask = size - 1;
size += num_prev_positions23; size += num_prev_positions23;
mb->num_prev_positions = size; mb->num_prev_positions = size;
mb->pos_array_size = pos_array_factor * ( mb->dictionary_size + 1 ); mb->pos_array_size = pos_array_factor * ( mb->dictionary_size + 1 );
size += mb->pos_array_size; size += mb->pos_array_size;
if( size * sizeof mb->prev_positions[0] <= size ) mb->prev_positions = 0; if( size * sizeof mb->prev_positions[0] <= size ) mb->prev_positions = 0;
@ -134,8 +135,20 @@ void Mb_reset( struct Matchfinder_base * const mb )
mb->stream_pos -= mb->pos; mb->stream_pos -= mb->pos;
mb->pos = 0; mb->pos = 0;
mb->cyclic_pos = 0; mb->cyclic_pos = 0;
for( i = 0; i < mb->num_prev_positions; ++i ) mb->prev_positions[i] = 0;
Mb_read_block( mb ); Mb_read_block( mb );
if( mb->at_stream_end && mb->stream_pos < mb->dictionary_size )
{
int size;
mb->dictionary_size = max( min_dictionary_size, mb->stream_pos );
size = 1 << max( 16, real_bits( mb->dictionary_size - 1 ) - 2 );
if( mb->dictionary_size > 1 << 26 ) /* 64 MiB */
size >>= 1;
mb->key4_mask = size - 1;
size += mb->num_prev_positions23;
mb->num_prev_positions = size;
mb->pos_array = mb->prev_positions + mb->num_prev_positions;
}
for( i = 0; i < mb->num_prev_positions; ++i ) mb->prev_positions[i] = 0;
} }
@ -148,7 +161,7 @@ void Re_flush_data( struct Range_encoder * const renc )
{ show_error( "Write error", errno, false ); cleanup_and_fail( 1 ); } { show_error( "Write error", errno, false ); cleanup_and_fail( 1 ); }
renc->partial_member_pos += renc->pos; renc->partial_member_pos += renc->pos;
renc->pos = 0; renc->pos = 0;
show_progress( 0, 0, 0, 0 ); show_cprogress( 0, 0, 0, 0 );
} }
} }
@ -188,5 +201,5 @@ void LZeb_reset( struct LZ_encoder_base * const eb )
Bm_array_init( eb->bm_align, dis_align_size ); Bm_array_init( eb->bm_align, dis_align_size );
Lm_init( &eb->match_len_model ); Lm_init( &eb->match_len_model );
Lm_init( &eb->rep_len_model ); Lm_init( &eb->rep_len_model );
Re_reset( &eb->renc ); Re_reset( &eb->renc, eb->mb.dictionary_size );
} }

16
encoder_base.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -170,6 +170,7 @@ struct Matchfinder_base
int stream_pos; /* first byte not yet read from file */ int stream_pos; /* first byte not yet read from file */
int pos_limit; /* when reached, a new block must be read */ int pos_limit; /* when reached, a new block must be read */
int key4_mask; int key4_mask;
int num_prev_positions23;
int num_prev_positions; /* size of prev_positions */ int num_prev_positions; /* size of prev_positions */
int pos_array_size; int pos_array_size;
int infd; /* input file descriptor */ int infd; /* input file descriptor */
@ -179,7 +180,7 @@ struct Matchfinder_base
bool Mb_read_block( struct Matchfinder_base * const mb ); bool Mb_read_block( struct Matchfinder_base * const mb );
void Mb_normalize_pos( struct Matchfinder_base * const mb ); void Mb_normalize_pos( struct Matchfinder_base * const mb );
bool Mb_init( struct Matchfinder_base * const mb, const int before, bool Mb_init( struct Matchfinder_base * const mb, const int before_size,
const int dict_size, const int after_size, const int dict_size, const int after_size,
const int dict_factor, const int num_prev_positions23, const int dict_factor, const int num_prev_positions23,
const int pos_array_factor, const int ifd ); const int pos_array_factor, const int ifd );
@ -262,7 +263,8 @@ static inline void Re_shift_low( struct Range_encoder * const renc )
renc->low = ( renc->low & 0x00FFFFFFU ) << 8; renc->low = ( renc->low & 0x00FFFFFFU ) << 8;
} }
static inline void Re_reset( struct Range_encoder * const renc ) static inline void Re_reset( struct Range_encoder * const renc,
const unsigned dictionary_size )
{ {
int i; int i;
renc->low = 0; renc->low = 0;
@ -271,6 +273,7 @@ static inline void Re_reset( struct Range_encoder * const renc )
renc->range = 0xFFFFFFFFU; renc->range = 0xFFFFFFFFU;
renc->ff_count = 0; renc->ff_count = 0;
renc->cache = 0; renc->cache = 0;
Fh_set_dictionary_size( renc->header, dictionary_size );
for( i = 0; i < Fh_size; ++i ) for( i = 0; i < Fh_size; ++i )
Re_put_byte( renc, renc->header[i] ); Re_put_byte( renc, renc->header[i] );
} }
@ -282,8 +285,7 @@ static inline bool Re_init( struct Range_encoder * const renc,
if( !renc->buffer ) return false; if( !renc->buffer ) return false;
renc->outfd = ofd; renc->outfd = ofd;
Fh_set_magic( renc->header ); Fh_set_magic( renc->header );
Fh_set_dictionary_size( renc->header, dictionary_size ); Re_reset( renc, dictionary_size );
Re_reset( renc );
return true; return true;
} }
@ -445,13 +447,13 @@ struct LZ_encoder_base
void LZeb_reset( struct LZ_encoder_base * const eb ); void LZeb_reset( struct LZ_encoder_base * const eb );
static inline bool LZeb_init( struct LZ_encoder_base * const eb, static inline bool LZeb_init( struct LZ_encoder_base * const eb,
const int before, const int dict_size, const int before_size, const int dict_size,
const int after_size, const int dict_factor, const int after_size, const int dict_factor,
const int num_prev_positions23, const int num_prev_positions23,
const int pos_array_factor, const int pos_array_factor,
const int ifd, const int outfd ) const int ifd, const int outfd )
{ {
if( !Mb_init( &eb->mb, before, dict_size, after_size, dict_factor, if( !Mb_init( &eb->mb, before_size, dict_size, after_size, dict_factor,
num_prev_positions23, pos_array_factor, ifd ) ) return false; num_prev_positions23, pos_array_factor, ifd ) ) return false;
if( !Re_init( &eb->renc, eb->mb.dictionary_size, outfd ) ) return false; if( !Re_init( &eb->renc, eb->mb.dictionary_size, outfd ) ) return false;
LZeb_reset( eb ); LZeb_reset( eb );

2
fast_encoder.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by

6
fast_encoder.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -49,7 +49,7 @@ static inline void FLZe_update_and_move( struct FLZ_encoder * const fe, int n )
static inline bool FLZe_init( struct FLZ_encoder * const fe, static inline bool FLZe_init( struct FLZ_encoder * const fe,
const int ifd, const int outfd ) const int ifd, const int outfd )
{ {
enum { before = 0, enum { before_size = 0,
dict_size = 65536, dict_size = 65536,
/* bytes to keep in buffer after pos */ /* bytes to keep in buffer after pos */
after_size = max_match_len, after_size = max_match_len,
@ -57,7 +57,7 @@ static inline bool FLZe_init( struct FLZ_encoder * const fe,
num_prev_positions23 = 0, num_prev_positions23 = 0,
pos_array_factor = 1 }; pos_array_factor = 1 };
return LZeb_init( &fe->eb, before, dict_size, after_size, dict_factor, return LZeb_init( &fe->eb, before_size, dict_size, after_size, dict_factor,
num_prev_positions23, pos_array_factor, ifd, outfd ); num_prev_positions23, pos_array_factor, ifd, outfd );
} }

32
file_index.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -108,7 +108,9 @@ static void Fi_set_num_error( struct File_index * const fi,
/* If successful, push last member and set pos to member header. */ /* If successful, push last member and set pos to member header. */
static bool Fi_skip_trailing_data( struct File_index * const fi, static bool Fi_skip_trailing_data( struct File_index * const fi,
const int fd, long long * const pos ) const int fd, long long * const pos,
const bool ignore_trailing,
const bool loose_trailing )
{ {
enum { block_size = 16384, enum { block_size = 16384,
buffer_size = block_size + Ft_size - 1 + Fh_size }; buffer_size = block_size + Ft_size - 1 + Fh_size };
@ -152,6 +154,11 @@ static bool Fi_skip_trailing_data( struct File_index * const fi,
add_error( fi, "Last member in input file is truncated or corrupt." ); add_error( fi, "Last member in input file is truncated or corrupt." );
fi->retval = 2; return false; fi->retval = 2; return false;
} }
if( !loose_trailing && bsize - i >= Fh_size &&
Fh_verify_corrupt( buffer + i ) )
{ add_error( fi, corrupt_mm_msg ); fi->retval = 2; return false; }
if( !ignore_trailing )
{ add_error( fi, trailing_msg ); fi->retval = 2; return false; }
*pos = ipos + i - member_size; *pos = ipos + i - member_size;
return push_back_member( fi, 0, Ft_get_data_size( *trailer ), *pos, return push_back_member( fi, 0, Ft_get_data_size( *trailer ), *pos,
member_size, dictionary_size ); member_size, dictionary_size );
@ -170,7 +177,7 @@ static bool Fi_skip_trailing_data( struct File_index * const fi,
bool Fi_init( struct File_index * const fi, const int infd, bool Fi_init( struct File_index * const fi, const int infd,
const bool ignore_trailing ) const bool ignore_trailing, const bool loose_trailing )
{ {
File_header header; File_header header;
long long pos; long long pos;
@ -211,12 +218,10 @@ bool Fi_init( struct File_index * const fi, const int infd,
member_size = Ft_get_member_size( trailer ); member_size = Ft_get_member_size( trailer );
if( member_size < min_member_size || member_size > (unsigned long long)pos ) if( member_size < min_member_size || member_size > (unsigned long long)pos )
{ {
if( fi->members > 0 ) if( fi->members <= 0 )
Fi_set_num_error( fi, "Member size in trailer is corrupt at pos ", { if( Fi_skip_trailing_data( fi, infd, &pos, ignore_trailing,
pos - 8 ); loose_trailing ) ) continue; else return false; }
else if( Fi_skip_trailing_data( fi, infd, &pos ) ) Fi_set_num_error( fi, "Member size in trailer is corrupt at pos ", pos - 8 );
{ if( ignore_trailing ) continue;
add_error( fi, trailing_msg ); fi->retval = 2; return false; }
break; break;
} }
if( seek_read( infd, header, Fh_size, pos - member_size ) != Fh_size ) if( seek_read( infd, header, Fh_size, pos - member_size ) != Fh_size )
@ -225,11 +230,10 @@ bool Fi_init( struct File_index * const fi, const int infd,
if( !Fh_verify_magic( header ) || !Fh_verify_version( header ) || if( !Fh_verify_magic( header ) || !Fh_verify_version( header ) ||
!isvalid_ds( dictionary_size ) ) !isvalid_ds( dictionary_size ) )
{ {
if( fi->members > 0 ) if( fi->members <= 0 )
Fi_set_num_error( fi, "Bad header at pos ", pos - member_size ); { if( Fi_skip_trailing_data( fi, infd, &pos, ignore_trailing,
else if( Fi_skip_trailing_data( fi, infd, &pos ) ) loose_trailing ) ) continue; else return false; }
{ if( ignore_trailing ) continue; Fi_set_num_error( fi, "Bad header at pos ", pos - member_size );
add_error( fi, trailing_msg ); fi->retval = 2; return false; }
break; break;
} }
pos -= member_size; pos -= member_size;

4
file_index.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -57,7 +57,7 @@ struct File_index
}; };
bool Fi_init( struct File_index * const fi, const int infd, bool Fi_init( struct File_index * const fi, const int infd,
const bool ignore_trailing ); const bool ignore_trailing, const bool loose_trailing );
void Fi_free( struct File_index * const fi ); void Fi_free( struct File_index * const fi );

8
list.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -34,7 +34,7 @@ static void list_line( const unsigned long long uncomp_size,
{ {
if( uncomp_size > 0 ) if( uncomp_size > 0 )
printf( "%15llu %15llu %6.2f%% %s\n", uncomp_size, comp_size, printf( "%15llu %15llu %6.2f%% %s\n", uncomp_size, comp_size,
100.0 * ( 1.0 - ( (double)comp_size / uncomp_size ) ), 100.0 - ( ( 100.0 * comp_size ) / uncomp_size ),
input_filename ); input_filename );
else else
printf( "%15llu %15llu -INF%% %s\n", uncomp_size, comp_size, printf( "%15llu %15llu -INF%% %s\n", uncomp_size, comp_size,
@ -43,7 +43,7 @@ static void list_line( const unsigned long long uncomp_size,
int list_files( const char * const filenames[], const int num_filenames, int list_files( const char * const filenames[], const int num_filenames,
const bool ignore_trailing ) const bool ignore_trailing, const bool loose_trailing )
{ {
unsigned long long total_comp = 0, total_uncomp = 0; unsigned long long total_comp = 0, total_uncomp = 0;
int files = 0, retval = 0; int files = 0, retval = 0;
@ -63,7 +63,7 @@ int list_files( const char * const filenames[], const int num_filenames,
open_instream( input_filename, &in_stats, true, true ); open_instream( input_filename, &in_stats, true, true );
if( infd < 0 ) { if( retval < 1 ) retval = 1; continue; } if( infd < 0 ) { if( retval < 1 ) retval = 1; continue; }
Fi_init( &file_index, infd, ignore_trailing ); Fi_init( &file_index, infd, ignore_trailing, loose_trailing );
close( infd ); close( infd );
if( file_index.retval != 0 ) if( file_index.retval != 0 )
{ {

57
lzip.h
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -22,6 +22,8 @@
#define min(x,y) ((x) <= (y) ? (x) : (y)) #define min(x,y) ((x) <= (y) ? (x) : (y))
#endif #endif
void * resize_buffer( void * buf, const unsigned min_size );
typedef int State; typedef int State;
enum { states = 12 }; enum { states = 12 };
@ -114,9 +116,13 @@ static inline void Lm_init( struct Len_model * const lm )
} }
/* defined in main.c */
extern int verbosity;
struct Pretty_print struct Pretty_print
{ {
const char * name; const char * name;
char * padded_name;
const char * stdin_name; const char * stdin_name;
unsigned longest_name; unsigned longest_name;
bool first_post; bool first_post;
@ -124,11 +130,12 @@ struct Pretty_print
static inline void Pp_init( struct Pretty_print * const pp, static inline void Pp_init( struct Pretty_print * const pp,
const char * const filenames[], const char * const filenames[],
const int num_filenames, const int verbosity ) const int num_filenames )
{ {
unsigned stdin_name_len; unsigned stdin_name_len;
int i; int i;
pp->name = 0; pp->name = 0;
pp->padded_name = 0;
pp->stdin_name = "(stdin)"; pp->stdin_name = "(stdin)";
pp->longest_name = 0; pp->longest_name = 0;
pp->first_post = false; pp->first_post = false;
@ -147,9 +154,19 @@ static inline void Pp_init( struct Pretty_print * const pp,
static inline void Pp_set_name( struct Pretty_print * const pp, static inline void Pp_set_name( struct Pretty_print * const pp,
const char * const filename ) const char * const filename )
{ {
unsigned name_len, padded_name_len, i = 0;
if( filename && filename[0] && strcmp( filename, "-" ) != 0 ) if( filename && filename[0] && strcmp( filename, "-" ) != 0 )
pp->name = filename; pp->name = filename;
else pp->name = pp->stdin_name; else pp->name = pp->stdin_name;
name_len = strlen( pp->name );
padded_name_len = max( name_len, pp->longest_name ) + 4;
pp->padded_name = resize_buffer( pp->padded_name, padded_name_len + 1 );
while( i < 2 ) pp->padded_name[i++] = ' ';
while( i < name_len + 2 ) { pp->padded_name[i] = pp->name[i-2]; ++i; }
pp->padded_name[i++] = ':';
while( i < padded_name_len ) pp->padded_name[i++] = ' ';
pp->padded_name[i] = 0;
pp->first_post = true; pp->first_post = true;
} }
@ -216,12 +233,21 @@ static inline void Fh_set_magic( File_header data )
static inline bool Fh_verify_magic( const File_header data ) static inline bool Fh_verify_magic( const File_header data )
{ return ( memcmp( data, magic_string, 4 ) == 0 ); } { return ( memcmp( data, magic_string, 4 ) == 0 ); }
/* detect truncated header */ /* detect (truncated) header */
static inline bool Fh_verify_prefix( const File_header data, const int size ) static inline bool Fh_verify_prefix( const File_header data, const int sz )
{ {
int i; for( i = 0; i < size && i < 4; ++i ) int i; for( i = 0; i < sz && i < 4; ++i )
if( data[i] != magic_string[i] ) return false; if( data[i] != magic_string[i] ) return false;
return ( size > 0 ); return ( sz > 0 );
}
/* detect corrupt header */
static inline bool Fh_verify_corrupt( const File_header data )
{
int matches = 0;
int i; for( i = 0; i < 4; ++i )
if( data[i] == magic_string[i] ) ++matches;
return ( matches > 1 && matches < 4 );
} }
static inline uint8_t Fh_version( const File_header data ) static inline uint8_t Fh_version( const File_header data )
@ -295,6 +321,7 @@ static inline void Ft_set_member_size( File_trailer data, unsigned long long sz
static const char * const bad_magic_msg = "Bad magic number (file not in lzip format)."; static const char * const bad_magic_msg = "Bad magic number (file not in lzip format).";
static const char * const bad_dict_msg = "Invalid dictionary size in member header."; static const char * const bad_dict_msg = "Invalid dictionary size in member header.";
static const char * const corrupt_mm_msg = "Corrupt header in multimember file.";
static const char * const trailing_msg = "Trailing data not allowed."; static const char * const trailing_msg = "Trailing data not allowed.";
/* defined in decoder.c */ /* defined in decoder.c */
@ -303,23 +330,27 @@ int writeblock( const int fd, const uint8_t * const buf, const int size );
/* defined in list.c */ /* defined in list.c */
int list_files( const char * const filenames[], const int num_filenames, int list_files( const char * const filenames[], const int num_filenames,
const bool ignore_trailing ); const bool ignore_trailing, const bool loose_trailing );
/* defined in main.c */ /* defined in main.c */
extern int verbosity;
struct stat; struct stat;
const char * bad_version( const unsigned version ); const char * bad_version( const unsigned version );
const char * format_ds( const unsigned dictionary_size ); const char * format_ds( const unsigned dictionary_size );
void show_header( const unsigned dictionary_size );
int open_instream( const char * const name, struct stat * const in_statsp, int open_instream( const char * const name, struct stat * const in_statsp,
const bool no_ofile, const bool reg_only ); const bool no_ofile, const bool reg_only );
void * resize_buffer( void * buf, const unsigned min_size );
void cleanup_and_fail( const int retval ); void cleanup_and_fail( const int retval );
void show_error( const char * const msg, const int errcode, const bool help ); void show_error( const char * const msg, const int errcode, const bool help );
void show_file_error( const char * const filename, const char * const msg, void show_file_error( const char * const filename, const char * const msg,
const int errcode ); const int errcode );
void internal_error( const char * const msg ); void internal_error( const char * const msg );
struct Matchfinder_base; struct Matchfinder_base;
void show_progress( const unsigned long long partial_size, void show_cprogress( const unsigned long long cfile_size,
const struct Matchfinder_base * const m, const unsigned long long partial_size,
struct Pretty_print * const p, const struct Matchfinder_base * const m,
const unsigned long long cfile_size ); struct Pretty_print * const p );
struct Range_decoder;
void show_dprogress( const unsigned long long cfile_size,
const unsigned long long partial_size,
const struct Range_decoder * const d,
struct Pretty_print * const p );

340
main.c
View file

@ -1,5 +1,5 @@
/* Clzip - LZMA lossless data compressor /* Clzip - LZMA lossless data compressor
Copyright (C) 2010-2017 Antonio Diaz Diaz. Copyright (C) 2010-2018 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
@ -71,7 +71,7 @@ int verbosity = 0;
const char * const Program_name = "Clzip"; const char * const Program_name = "Clzip";
const char * const program_name = "clzip"; const char * const program_name = "clzip";
const char * const program_year = "2017"; const char * const program_year = "2018";
const char * invocation_name = 0; const char * invocation_name = 0;
const struct { const char * from; const char * to; } known_extensions[] = { const struct { const char * from; const char * to; } known_extensions[] = {
@ -111,12 +111,13 @@ static void show_help( void )
" -o, --output=<file> if reading standard input, write to <file>\n" " -o, --output=<file> if reading standard input, write to <file>\n"
" -q, --quiet suppress all messages\n" " -q, --quiet suppress all messages\n"
" -s, --dictionary-size=<bytes> set dictionary size limit in bytes [8 MiB]\n" " -s, --dictionary-size=<bytes> set dictionary size limit in bytes [8 MiB]\n"
" -S, --volume-size=<bytes> set volume size limit in bytes\n" " -S, --volume-size=<bytes> set volume size limit in bytes, implies -k\n"
" -t, --test test compressed file integrity\n" " -t, --test test compressed file integrity\n"
" -v, --verbose be verbose (a 2nd -v gives more)\n" " -v, --verbose be verbose (a 2nd -v gives more)\n"
" -0 .. -9 set compression level [default 6]\n" " -0 .. -9 set compression level [default 6]\n"
" --fast alias for -0\n" " --fast alias for -0\n"
" --best alias for -9\n" " --best alias for -9\n"
" --loose-trailing allow trailing data seeming corrupt header\n"
"If no file names are given, or if a file is '-', clzip compresses or\n" "If no file names are given, or if a file is '-', clzip compresses or\n"
"decompresses from standard input to standard output.\n" "decompresses from standard input to standard output.\n"
"Numbers may be followed by a multiplier: k = kB = 10^3 = 1000,\n" "Numbers may be followed by a multiplier: k = kB = 10^3 = 1000,\n"
@ -146,6 +147,35 @@ static void show_version( void )
} }
/* assure at least a minimum size for buffer 'buf' */
void * resize_buffer( void * buf, const unsigned min_size )
{
if( buf ) buf = realloc( buf, min_size );
else buf = malloc( min_size );
if( !buf )
{
show_error( "Not enough memory.", 0, false );
cleanup_and_fail( 1 );
}
return buf;
}
void Pp_show_msg( struct Pretty_print * const pp, const char * const msg )
{
if( verbosity >= 0 )
{
if( pp->first_post )
{
pp->first_post = false;
fputs( pp->padded_name, stderr );
if( !msg ) fflush( stderr );
}
if( msg ) fprintf( stderr, "%s\n", msg );
}
}
const char * bad_version( const unsigned version ) const char * bad_version( const unsigned version )
{ {
static char buf[80]; static char buf[80];
@ -163,10 +193,10 @@ const char * format_ds( const unsigned dictionary_size )
{ "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" }; { "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" };
const char * p = ""; const char * p = "";
const char * np = " "; const char * np = " ";
unsigned num = dictionary_size, i; unsigned num = dictionary_size;
bool exact = ( num % factor == 0 ); bool exact = ( num % factor == 0 );
for( i = 0; i < 8 && ( num > 9999 || ( exact && num >= factor ) ); ++i ) int i; for( i = 0; i < 8 && ( num > 9999 || ( exact && num >= factor ) ); ++i )
{ num /= factor; if( num % factor != 0 ) exact = false; { num /= factor; if( num % factor != 0 ) exact = false;
p = prefix[i]; np = ""; } p = prefix[i]; np = ""; }
snprintf( buf, bufsize, "%s%4u %sB", np, num, p ); snprintf( buf, bufsize, "%s%4u %sB", np, num, p );
@ -174,10 +204,9 @@ const char * format_ds( const unsigned dictionary_size )
} }
static void show_header( const unsigned dictionary_size ) void show_header( const unsigned dictionary_size )
{ {
if( verbosity >= 3 ) fprintf( stderr, "dictionary %s, ", format_ds( dictionary_size ) );
fprintf( stderr, "dictionary %s. ", format_ds( dictionary_size ) );
} }
@ -271,6 +300,43 @@ static int extension_index( const char * const name )
} }
static void set_c_outname( const char * const name, const bool force_ext,
const bool multifile )
{
output_filename = resize_buffer( output_filename, strlen( name ) + 5 +
strlen( known_extensions[0].from ) + 1 );
strcpy( output_filename, name );
if( multifile ) strcat( output_filename, "00001" );
if( force_ext || multifile || extension_index( output_filename ) < 0 )
strcat( output_filename, known_extensions[0].from );
}
static void set_d_outname( const char * const name, const int eindex )
{
const unsigned name_len = strlen( name );
if( eindex >= 0 )
{
const char * const from = known_extensions[eindex].from;
const unsigned from_len = strlen( from );
if( name_len > from_len )
{
output_filename = resize_buffer( output_filename, name_len +
strlen( known_extensions[eindex].to ) + 1 );
strcpy( output_filename, name );
strcpy( output_filename + name_len - from_len, known_extensions[eindex].to );
return;
}
}
output_filename = resize_buffer( output_filename, name_len + 4 + 1 );
strcpy( output_filename, name );
strcat( output_filename, ".out" );
if( verbosity >= 1 )
fprintf( stderr, "%s: Can't guess original name for '%s' -- using '%s'\n",
program_name, name, output_filename );
}
int open_instream( const char * const name, struct stat * const in_statsp, int open_instream( const char * const name, struct stat * const in_statsp,
const bool no_ofile, const bool reg_only ) const bool no_ofile, const bool reg_only )
{ {
@ -315,55 +381,6 @@ static int open_instream2( const char * const name, struct stat * const in_stats
} }
/* assure at least a minimum size for buffer 'buf' */
void * resize_buffer( void * buf, const unsigned min_size )
{
if( buf ) buf = realloc( buf, min_size );
else buf = malloc( min_size );
if( !buf )
{
show_error( "Not enough memory.", 0, false );
cleanup_and_fail( 1 );
}
return buf;
}
static void set_c_outname( const char * const name, const bool multifile )
{
output_filename = resize_buffer( output_filename, strlen( name ) + 5 +
strlen( known_extensions[0].from ) + 1 );
strcpy( output_filename, name );
if( multifile ) strcat( output_filename, "00001" );
strcat( output_filename, known_extensions[0].from );
}
static void set_d_outname( const char * const name, const int eindex )
{
const unsigned name_len = strlen( name );
if( eindex >= 0 )
{
const char * const from = known_extensions[eindex].from;
const unsigned from_len = strlen( from );
if( name_len > from_len )
{
output_filename = resize_buffer( output_filename, name_len +
strlen( known_extensions[eindex].to ) + 1 );
strcpy( output_filename, name );
strcpy( output_filename + name_len - from_len, known_extensions[eindex].to );
return;
}
}
output_filename = resize_buffer( output_filename, name_len + 4 + 1 );
strcpy( output_filename, name );
strcat( output_filename, ".out" );
if( verbosity >= 1 )
fprintf( stderr, "%s: Can't guess original name for '%s' -- using '%s'\n",
program_name, name, output_filename );
}
static bool open_outstream( const bool force, const bool from_stdin ) static bool open_outstream( const bool force, const bool from_stdin )
{ {
const mode_t usr_rw = S_IRUSR | S_IWUSR; const mode_t usr_rw = S_IRUSR | S_IWUSR;
@ -479,14 +496,13 @@ struct Poly_encoder
}; };
static int compress( const unsigned long long member_size, static int compress( const unsigned long long cfile_size,
const unsigned long long member_size,
const unsigned long long volume_size, const int infd, const unsigned long long volume_size, const int infd,
const struct Lzma_options * const encoder_options, const struct Lzma_options * const encoder_options,
struct Pretty_print * const pp, struct Pretty_print * const pp,
const struct stat * const in_statsp, const bool zero ) const struct stat * const in_statsp, const bool zero )
{ {
const unsigned long long cfile_size =
(in_statsp && S_ISREG( in_statsp->st_mode )) ? in_statsp->st_size / 100 : 0;
unsigned long long in_size = 0, out_size = 0, partial_volume_size = 0; unsigned long long in_size = 0, out_size = 0, partial_volume_size = 0;
int retval = 0; int retval = 0;
struct Poly_encoder encoder = { 0, 0, 0 }; /* polymorphic encoder */ struct Poly_encoder encoder = { 0, 0, 0 }; /* polymorphic encoder */
@ -524,7 +540,7 @@ static int compress( const unsigned long long member_size,
{ {
const unsigned long long size = ( volume_size > 0 ) ? const unsigned long long size = ( volume_size > 0 ) ?
min( member_size, volume_size - partial_volume_size ) : member_size; min( member_size, volume_size - partial_volume_size ) : member_size;
show_progress( in_size, &encoder.eb->mb, pp, cfile_size ); /* init */ show_cprogress( cfile_size, in_size, &encoder.eb->mb, pp ); /* init */
if( ( zero && !FLZe_encode_member( encoder.fe, size ) ) || if( ( zero && !FLZe_encode_member( encoder.fe, size ) ) ||
( !zero && !LZe_encode_member( encoder.e, size ) ) ) ( !zero && !LZe_encode_member( encoder.e, size ) ) )
{ Pp_show_msg( pp, "Encoder error." ); retval = 1; break; } { Pp_show_msg( pp, "Encoder error." ); retval = 1; break; }
@ -554,11 +570,11 @@ static int compress( const unsigned long long member_size,
if( in_size == 0 || out_size == 0 ) if( in_size == 0 || out_size == 0 )
fputs( " no data compressed.\n", stderr ); fputs( " no data compressed.\n", stderr );
else else
fprintf( stderr, "%6.3f:1, %6.3f bits/byte, " fprintf( stderr, "%6.3f:1, %5.2f%% ratio, %5.2f%% saved, "
"%5.2f%% saved, %llu in, %llu out.\n", "%llu in, %llu out.\n",
(double)in_size / out_size, (double)in_size / out_size,
( 8.0 * out_size ) / in_size, ( 100.0 * out_size ) / in_size,
100.0 * ( 1.0 - ( (double)out_size / in_size ) ), 100.0 - ( ( 100.0 * out_size ) / in_size ),
in_size, out_size ); in_size, out_size );
} }
LZeb_free( encoder.eb ); LZeb_free( encoder.eb );
@ -577,9 +593,9 @@ static unsigned char xdigit( const unsigned value )
static bool show_trailing_data( const uint8_t * const data, const int size, static bool show_trailing_data( const uint8_t * const data, const int size,
struct Pretty_print * const pp, const bool all, struct Pretty_print * const pp, const bool all,
const bool ignore_trailing ) const int ignore_trailing ) /* -1 = show */
{ {
if( verbosity >= 4 || !ignore_trailing ) if( verbosity >= 4 || ignore_trailing <= 0 )
{ {
int i; int i;
char buf[80]; char buf[80];
@ -597,14 +613,15 @@ static bool show_trailing_data( const uint8_t * const data, const int size,
if( len < sizeof buf ) buf[len++] = '\''; if( len < sizeof buf ) buf[len++] = '\'';
if( len < sizeof buf ) buf[len] = 0; else buf[sizeof buf - 1] = 0; if( len < sizeof buf ) buf[len] = 0; else buf[sizeof buf - 1] = 0;
Pp_show_msg( pp, buf ); Pp_show_msg( pp, buf );
if( !ignore_trailing ) show_file_error( pp->name, trailing_msg, 0 ); if( ignore_trailing == 0 ) show_file_error( pp->name, trailing_msg, 0 );
} }
return ignore_trailing; return ( ignore_trailing > 0 );
} }
static int decompress( const int infd, struct Pretty_print * const pp, static int decompress( const unsigned long long cfile_size, const int infd,
const bool ignore_trailing, const bool testing ) struct Pretty_print * const pp, const bool ignore_trailing,
const bool loose_trailing, const bool testing )
{ {
unsigned long long partial_file_pos = 0; unsigned long long partial_file_pos = 0;
struct Range_decoder rdec; struct Range_decoder rdec;
@ -626,8 +643,12 @@ static int decompress( const int infd, struct Pretty_print * const pp,
size = Rd_read_data( &rdec, header, Fh_size ); size = Rd_read_data( &rdec, header, Fh_size );
if( Rd_finished( &rdec ) ) /* End Of File */ if( Rd_finished( &rdec ) ) /* End Of File */
{ {
if( first_member || Fh_verify_prefix( header, size ) ) if( first_member )
{ Pp_show_msg( pp, "File ends unexpectedly at member header." ); { show_file_error( pp->name, "File ends unexpectedly at member header.", 0 );
retval = 2; }
else if( Fh_verify_prefix( header, size ) )
{ Pp_show_msg( pp, "Truncated header in multimember file." );
show_trailing_data( header, size, pp, true, -1 );
retval = 2; } retval = 2; }
else if( size > 0 && !show_trailing_data( header, size, pp, else if( size > 0 && !show_trailing_data( header, size, pp,
true, ignore_trailing ) ) true, ignore_trailing ) )
@ -638,24 +659,27 @@ static int decompress( const int infd, struct Pretty_print * const pp,
{ {
if( first_member ) if( first_member )
{ show_file_error( pp->name, bad_magic_msg, 0 ); retval = 2; } { show_file_error( pp->name, bad_magic_msg, 0 ); retval = 2; }
else if( !loose_trailing && Fh_verify_corrupt( header ) )
{ Pp_show_msg( pp, corrupt_mm_msg );
show_trailing_data( header, size, pp, false, -1 );
retval = 2; }
else if( !show_trailing_data( header, size, pp, false, ignore_trailing ) ) else if( !show_trailing_data( header, size, pp, false, ignore_trailing ) )
retval = 2; retval = 2;
break; break;
} }
if( !Fh_verify_version( header ) ) if( !Fh_verify_version( header ) )
{ { Pp_show_msg( pp, bad_version( Fh_version( header ) ) );
Pp_show_msg( pp, bad_version( Fh_version( header ) ) ); retval = 2; break; }
retval = 2; break;
}
dictionary_size = Fh_get_dictionary_size( header ); dictionary_size = Fh_get_dictionary_size( header );
if( !isvalid_ds( dictionary_size ) ) if( !isvalid_ds( dictionary_size ) )
{ Pp_show_msg( pp, bad_dict_msg ); retval = 2; break; } { Pp_show_msg( pp, bad_dict_msg ); retval = 2; break; }
if( verbosity >= 2 || ( verbosity == 1 && first_member ) ) if( verbosity >= 2 || ( verbosity == 1 && first_member ) )
{ Pp_show_msg( pp, 0 ); show_header( dictionary_size ); } Pp_show_msg( pp, 0 );
if( !LZd_init( &decoder, &rdec, dictionary_size, outfd ) ) if( !LZd_init( &decoder, &rdec, dictionary_size, outfd ) )
{ Pp_show_msg( pp, "Not enough memory." ); retval = 1; break; } { Pp_show_msg( pp, "Not enough memory." ); retval = 1; break; }
show_dprogress( cfile_size, partial_file_pos, &rdec, pp ); /* init */
result = LZd_decode_member( &decoder, pp ); result = LZd_decode_member( &decoder, pp );
partial_file_pos += Rd_member_position( &rdec ); partial_file_pos += Rd_member_position( &rdec );
LZd_free( &decoder ); LZd_free( &decoder );
@ -728,25 +752,61 @@ void internal_error( const char * const msg )
} }
void show_progress( const unsigned long long partial_size, void show_cprogress( const unsigned long long cfile_size,
const struct Matchfinder_base * const m, const unsigned long long partial_size,
struct Pretty_print * const p, const struct Matchfinder_base * const m,
const unsigned long long cfile_size ) struct Pretty_print * const p )
{ {
static unsigned long long csize = 0; /* file_size / 100 */ static unsigned long long csize = 0; /* file_size / 100 */
static unsigned long long psize = 0; static unsigned long long psize = 0;
static const struct Matchfinder_base * mb = 0; static const struct Matchfinder_base * mb = 0;
static struct Pretty_print * pp = 0; static struct Pretty_print * pp = 0;
static bool enabled = true;
if( verbosity < 2 ) return; if( !enabled ) return;
if( m ) /* initialize static vars */ if( p ) /* initialize static vars */
{ csize = cfile_size; psize = partial_size; mb = m; pp = p; } {
if( verbosity < 2 || !isatty( STDERR_FILENO ) ) { enabled = false; return; }
csize = cfile_size; psize = partial_size; mb = m; pp = p;
}
if( mb && pp ) if( mb && pp )
{ {
const unsigned long long pos = psize + Mb_data_position( mb ); const unsigned long long pos = psize + Mb_data_position( mb );
if( csize > 0 ) if( csize > 0 )
fprintf( stderr, "%4llu%%", pos / csize ); fprintf( stderr, "%4llu%% %.1f MB\r", pos / csize, pos / 1000000.0 );
fprintf( stderr, " %.1f MB\r", pos / 1000000.0 ); else
fprintf( stderr, " %.1f MB\r", pos / 1000000.0 );
Pp_reset( pp ); Pp_show_msg( pp, 0 ); /* restore cursor position */
}
}
void show_dprogress( const unsigned long long cfile_size,
const unsigned long long partial_size,
const struct Range_decoder * const d,
struct Pretty_print * const p )
{
static unsigned long long csize = 0; /* file_size / 100 */
static unsigned long long psize = 0;
static const struct Range_decoder * rdec = 0;
static struct Pretty_print * pp = 0;
static int counter = 0;
static bool enabled = true;
if( !enabled ) return;
if( p ) /* initialize static vars */
{
if( verbosity < 2 || !isatty( STDERR_FILENO ) ) { enabled = false; return; }
csize = cfile_size; psize = partial_size; rdec = d; pp = p; counter = 0;
}
if( rdec && pp && --counter <= 0 )
{
const unsigned long long pos = psize + Rd_member_position( rdec );
counter = 7; /* update display every 114688 bytes */
if( csize > 0 )
fprintf( stderr, "%4llu%% %.1f MB\r", pos / csize, pos / 1000000.0 );
else
fprintf( stderr, " %.1f MB\r", pos / 1000000.0 );
Pp_reset( pp ); Pp_show_msg( pp, 0 ); /* restore cursor position */ Pp_reset( pp ); Pp_show_msg( pp, 0 ); /* restore cursor position */
} }
} }
@ -758,7 +818,7 @@ int main( const int argc, const char * const argv[] )
to the corresponding LZMA compression modes. */ to the corresponding LZMA compression modes. */
const struct Lzma_options option_mapping[] = const struct Lzma_options option_mapping[] =
{ {
{ 1 << 16, 16 }, /* -0 entry values not used */ { 1 << 16, 16 }, /* -0 */
{ 1 << 20, 5 }, /* -1 */ { 1 << 20, 5 }, /* -1 */
{ 3 << 19, 6 }, /* -2 */ { 3 << 19, 6 }, /* -2 */
{ 1 << 21, 8 }, /* -3 */ { 1 << 21, 8 }, /* -3 */
@ -776,52 +836,55 @@ int main( const int argc, const char * const argv[] )
const char * default_output_filename = ""; const char * default_output_filename = "";
const char ** filenames = 0; const char ** filenames = 0;
int num_filenames = 0; int num_filenames = 0;
int infd = -1;
enum Mode program_mode = m_compress; enum Mode program_mode = m_compress;
int argind = 0; int argind = 0;
int failed_tests = 0;
int retval = 0; int retval = 0;
int i; int i;
bool filenames_given = false; bool filenames_given = false;
bool force = false; bool force = false;
bool ignore_trailing = true; bool ignore_trailing = true;
bool keep_input_files = false; bool keep_input_files = false;
bool loose_trailing = false;
bool recompress = false; bool recompress = false;
bool stdin_used = false; bool stdin_used = false;
bool to_stdout = false; bool to_stdout = false;
bool zero = false; bool zero = false;
struct Pretty_print pp; struct Pretty_print pp;
enum { opt_lt = 256 };
const struct ap_Option options[] = const struct ap_Option options[] =
{ {
{ '0', "fast", ap_no }, { '0', "fast", ap_no },
{ '1', 0, ap_no }, { '1', 0, ap_no },
{ '2', 0, ap_no }, { '2', 0, ap_no },
{ '3', 0, ap_no }, { '3', 0, ap_no },
{ '4', 0, ap_no }, { '4', 0, ap_no },
{ '5', 0, ap_no }, { '5', 0, ap_no },
{ '6', 0, ap_no }, { '6', 0, ap_no },
{ '7', 0, ap_no }, { '7', 0, ap_no },
{ '8', 0, ap_no }, { '8', 0, ap_no },
{ '9', "best", ap_no }, { '9', "best", ap_no },
{ 'a', "trailing-error", ap_no }, { 'a', "trailing-error", ap_no },
{ 'b', "member-size", ap_yes }, { 'b', "member-size", ap_yes },
{ 'c', "stdout", ap_no }, { 'c', "stdout", ap_no },
{ 'd', "decompress", ap_no }, { 'd', "decompress", ap_no },
{ 'f', "force", ap_no }, { 'f', "force", ap_no },
{ 'F', "recompress", ap_no }, { 'F', "recompress", ap_no },
{ 'h', "help", ap_no }, { 'h', "help", ap_no },
{ 'k', "keep", ap_no }, { 'k', "keep", ap_no },
{ 'l', "list", ap_no }, { 'l', "list", ap_no },
{ 'm', "match-length", ap_yes }, { 'm', "match-length", ap_yes },
{ 'n', "threads", ap_yes }, { 'n', "threads", ap_yes },
{ 'o', "output", ap_yes }, { 'o', "output", ap_yes },
{ 'q', "quiet", ap_no }, { 'q', "quiet", ap_no },
{ 's', "dictionary-size", ap_yes }, { 's', "dictionary-size", ap_yes },
{ 'S', "volume-size", ap_yes }, { 'S', "volume-size", ap_yes },
{ 't', "test", ap_no }, { 't', "test", ap_no },
{ 'v', "verbose", ap_no }, { 'v', "verbose", ap_no },
{ 'V', "version", ap_no }, { 'V', "version", ap_no },
{ 0 , 0, ap_no } }; { opt_lt, "loose-trailing", ap_no },
{ 0 , 0, ap_no } };
struct Arg_parser parser; struct Arg_parser parser;
@ -865,6 +928,7 @@ int main( const int argc, const char * const argv[] )
case 't': set_mode( &program_mode, m_test ); break; case 't': set_mode( &program_mode, m_test ); break;
case 'v': if( verbosity < 4 ) ++verbosity; break; case 'v': if( verbosity < 4 ) ++verbosity; break;
case 'V': show_version(); return 0; case 'V': show_version(); return 0;
case opt_lt: loose_trailing = true; break;
default : internal_error( "uncaught option." ); default : internal_error( "uncaught option." );
} }
} /* end process options */ } /* end process options */
@ -885,7 +949,7 @@ int main( const int argc, const char * const argv[] )
} }
if( program_mode == m_list ) if( program_mode == m_list )
return list_files( filenames, num_filenames, ignore_trailing ); return list_files( filenames, num_filenames, ignore_trailing, loose_trailing );
if( program_mode == m_test ) if( program_mode == m_test )
outfd = -1; outfd = -1;
@ -899,12 +963,14 @@ int main( const int argc, const char * const argv[] )
( filenames_given || default_output_filename[0] ) ) ( filenames_given || default_output_filename[0] ) )
set_signals(); set_signals();
Pp_init( &pp, filenames, num_filenames, verbosity ); Pp_init( &pp, filenames, num_filenames );
output_filename = resize_buffer( output_filename, 1 ); output_filename = resize_buffer( output_filename, 1 );
for( i = 0; i < num_filenames; ++i ) for( i = 0; i < num_filenames; ++i )
{ {
unsigned long long cfile_size;
const char * input_filename = ""; const char * input_filename = "";
int infd;
int tmp; int tmp;
struct stat in_stats; struct stat in_stats;
const struct stat * in_statsp; const struct stat * in_statsp;
@ -921,7 +987,7 @@ int main( const int argc, const char * const argv[] )
else else
{ {
if( program_mode == m_compress ) if( program_mode == m_compress )
set_c_outname( default_output_filename, volume_size > 0 ); set_c_outname( default_output_filename, false, volume_size > 0 );
else else
{ {
output_filename = resize_buffer( output_filename, output_filename = resize_buffer( output_filename,
@ -931,7 +997,7 @@ int main( const int argc, const char * const argv[] )
if( !open_outstream( force, true ) ) if( !open_outstream( force, true ) )
{ {
if( retval < 1 ) retval = 1; if( retval < 1 ) retval = 1;
close( infd ); infd = -1; close( infd );
continue; continue;
} }
} }
@ -949,12 +1015,12 @@ int main( const int argc, const char * const argv[] )
else else
{ {
if( program_mode == m_compress ) if( program_mode == m_compress )
set_c_outname( input_filename, volume_size > 0 ); set_c_outname( input_filename, true, volume_size > 0 );
else set_d_outname( input_filename, eindex ); else set_d_outname( input_filename, eindex );
if( !open_outstream( force, false ) ) if( !open_outstream( force, false ) )
{ {
if( retval < 1 ) retval = 1; if( retval < 1 ) retval = 1;
close( infd ); infd = -1; close( infd );
continue; continue;
} }
} }
@ -965,33 +1031,43 @@ int main( const int argc, const char * const argv[] )
if( !check_tty( pp.name, infd, program_mode ) ) if( !check_tty( pp.name, infd, program_mode ) )
{ {
if( retval < 1 ) retval = 1; if( retval < 1 ) retval = 1;
if( program_mode == m_test ) { close( infd ); infd = -1; continue; } if( program_mode == m_test ) { close( infd ); continue; }
cleanup_and_fail( retval ); cleanup_and_fail( retval );
} }
in_statsp = input_filename[0] ? &in_stats : 0; in_statsp = input_filename[0] ? &in_stats : 0;
cfile_size = ( in_statsp && S_ISREG( in_statsp->st_mode ) ) ?
( in_statsp->st_size + 99 ) / 100 : 0;
if( program_mode == m_compress ) if( program_mode == m_compress )
tmp = compress( member_size, volume_size, infd, &encoder_options, &pp, tmp = compress( cfile_size, member_size, volume_size, infd,
in_statsp, zero ); &encoder_options, &pp, in_statsp, zero );
else else
tmp = decompress( infd, &pp, ignore_trailing, program_mode == m_test ); tmp = decompress( cfile_size, infd, &pp, ignore_trailing,
loose_trailing, program_mode == m_test );
if( tmp > retval ) retval = tmp; if( tmp > retval ) retval = tmp;
if( tmp && program_mode != m_test ) cleanup_and_fail( retval ); if( tmp )
{ if( program_mode != m_test ) cleanup_and_fail( retval );
else ++failed_tests; }
if( delete_output_on_interrupt ) if( delete_output_on_interrupt )
close_and_set_permissions( in_statsp ); close_and_set_permissions( in_statsp );
if( input_filename[0] ) if( input_filename[0] )
{ {
close( infd ); infd = -1; close( infd );
if( !keep_input_files && !to_stdout && program_mode != m_test ) if( !keep_input_files && !to_stdout && program_mode != m_test &&
( program_mode != m_compress || volume_size == 0 ) )
remove( input_filename ); remove( input_filename );
} }
} }
if( outfd >= 0 && close( outfd ) != 0 ) if( outfd >= 0 && close( outfd ) != 0 )
{ {
show_error( "Can't close stdout", errno, false ); show_error( "Error closing stdout", errno, false );
if( retval < 1 ) retval = 1; if( retval < 1 ) retval = 1;
} }
if( failed_tests > 0 && verbosity >= 1 && num_filenames > 1 )
fprintf( stderr, "%s: warning: %d %s failed the test.\n",
program_name, failed_tests,
( failed_tests == 1 ) ? "file" : "files" );
free( output_filename ); free( output_filename );
free( filenames ); free( filenames );
ap_free( &parser ); ap_free( &parser );

70
testsuite/check.sh
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# check script for Clzip - LZMA lossless data compressor # check script for Clzip - LZMA lossless data compressor
# Copyright (C) 2010-2017 Antonio Diaz Diaz. # Copyright (C) 2010-2018 Antonio Diaz Diaz.
# #
# This script is free software: you have unlimited permission # This script is free software: you have unlimited permission
# to copy, distribute and modify it. # to copy, distribute and modify it.
@ -53,6 +53,8 @@ done
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -cdq < in "${LZIP}" -cdq < in
[ $? = 2 ] || test_failed $LINENO [ $? = 2 ] || test_failed $LINENO
"${LZIP}" -dq -o in < "${in_lz}"
[ $? = 1 ] || test_failed $LINENO
# these are for code coverage # these are for code coverage
"${LZIP}" -lt "${in_lz}" 2> /dev/null "${LZIP}" -lt "${in_lz}" 2> /dev/null
[ $? = 1 ] || test_failed $LINENO [ $? = 1 ] || test_failed $LINENO
@ -99,6 +101,11 @@ printf "to be overwritten" > copy || framework_failure
"${LZIP}" -df copy.lz "${LZIP}" -df copy.lz
{ [ $? = 0 ] && [ ! -e copy.lz ] && cmp in copy ; } || test_failed $LINENO { [ $? = 0 ] && [ ! -e copy.lz ] && cmp in copy ; } || test_failed $LINENO
rm -f copy
cat "${in_lz}" > copy.lz || framework_failure
"${LZIP}" -d -S100k copy.lz
{ [ $? = 0 ] && [ ! -e copy.lz ] && cmp in copy ; } || test_failed $LINENO
printf "to be overwritten" > copy || framework_failure printf "to be overwritten" > copy || framework_failure
"${LZIP}" -df -o copy < "${in_lz}" || test_failed $LINENO "${LZIP}" -df -o copy < "${in_lz}" || test_failed $LINENO
cmp in copy || test_failed $LINENO cmp in copy || test_failed $LINENO
@ -145,7 +152,7 @@ cat "${in_lz}" "${in_lz}" > in2.lz || framework_failure
"${LZIP}" -cd in2.lz > copy2 || test_failed $LINENO "${LZIP}" -cd in2.lz > copy2 || test_failed $LINENO
cmp in2 copy2 || test_failed $LINENO cmp in2 copy2 || test_failed $LINENO
"${LZIP}" --output=copy2 < in2 || test_failed $LINENO "${LZIP}" --output=copy2.lz < in2 || test_failed $LINENO
"${LZIP}" -lq copy2.lz || test_failed $LINENO "${LZIP}" -lq copy2.lz || test_failed $LINENO
"${LZIP}" -t copy2.lz || test_failed $LINENO "${LZIP}" -t copy2.lz || test_failed $LINENO
"${LZIP}" -cd copy2.lz > copy2 || test_failed $LINENO "${LZIP}" -cd copy2.lz > copy2 || test_failed $LINENO
@ -204,10 +211,14 @@ for i in s4Ki 0 1 2 3 4 5 6 7 8 9 ; do
done done
cat in in in in in in in in > in8 || framework_failure cat in in in in in in in in > in8 || framework_failure
"${LZIP}" -1s12 -S100k -o out < in8 || test_failed $LINENO "${LZIP}" -1s12 -S100k in8 || test_failed $LINENO
"${LZIP}" -t out00001.lz out00002.lz || test_failed $LINENO "${LZIP}" -t in800001.lz in800002.lz || test_failed $LINENO
"${LZIP}" -cd out00001.lz out00002.lz | cmp in8 - || test_failed $LINENO "${LZIP}" -cd in800001.lz in800002.lz | cmp in8 - || test_failed $LINENO
rm -f out00001.lz rm -f in800001.lz in800002.lz
"${LZIP}" -1s12 -S100k -o out.lz < in8 || test_failed $LINENO
"${LZIP}" -t out.lz00001.lz out.lz00002.lz || test_failed $LINENO
"${LZIP}" -cd out.lz00001.lz out.lz00002.lz | cmp in8 - || test_failed $LINENO
rm -f out.lz00001.lz out.lz00002.lz
"${LZIP}" -1ks4Ki -b100000 in8 || test_failed $LINENO "${LZIP}" -1ks4Ki -b100000 in8 || test_failed $LINENO
"${LZIP}" -t in8.lz || test_failed $LINENO "${LZIP}" -t in8.lz || test_failed $LINENO
"${LZIP}" -cd in8.lz | cmp in8 - || test_failed $LINENO "${LZIP}" -cd in8.lz | cmp in8 - || test_failed $LINENO
@ -215,7 +226,15 @@ rm -f in8
"${LZIP}" -0 -S100k -o out < in8.lz || test_failed $LINENO "${LZIP}" -0 -S100k -o out < in8.lz || test_failed $LINENO
"${LZIP}" -t out00001.lz out00002.lz || test_failed $LINENO "${LZIP}" -t out00001.lz out00002.lz || test_failed $LINENO
"${LZIP}" -cd out00001.lz out00002.lz | cmp in8.lz - || test_failed $LINENO "${LZIP}" -cd out00001.lz out00002.lz | cmp in8.lz - || test_failed $LINENO
rm -f out00001.lz
"${LZIP}" -1 -S100k -o out < in8.lz || test_failed $LINENO
"${LZIP}" -t out00001.lz out00002.lz || test_failed $LINENO
"${LZIP}" -cd out00001.lz out00002.lz | cmp in8.lz - || test_failed $LINENO
rm -f out00001.lz out00002.lz rm -f out00001.lz out00002.lz
"${LZIP}" -0 -F -S100k in8.lz || test_failed $LINENO
"${LZIP}" -t in8.lz00001.lz in8.lz00002.lz || test_failed $LINENO
"${LZIP}" -cd in8.lz00001.lz in8.lz00002.lz | cmp in8.lz - || test_failed $LINENO
rm -f in8.lz00001.lz in8.lz00002.lz
"${LZIP}" -0kF -b100k in8.lz || test_failed $LINENO "${LZIP}" -0kF -b100k in8.lz || test_failed $LINENO
"${LZIP}" -t in8.lz.lz || test_failed $LINENO "${LZIP}" -t in8.lz.lz || test_failed $LINENO
"${LZIP}" -cd in8.lz.lz | cmp in8.lz - || test_failed $LINENO "${LZIP}" -cd in8.lz.lz | cmp in8.lz - || test_failed $LINENO
@ -223,6 +242,41 @@ rm -f in8.lz in8.lz.lz
printf "\ntesting bad input..." printf "\ntesting bad input..."
headers='LZIp LZiP LZip LzIP LzIp LziP lZIP lZIp lZiP lzIP'
body='\001\014\000\203\377\373\377\377\300\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000$\000\000\000\000\000\000\000'
cat "${in_lz}" > in0.lz
printf "LZIP${body}" >> in0.lz
if "${LZIP}" -tq in0.lz ; then
for header in ${headers} ; do
printf "${header}${body}" > in0.lz # first member
"${LZIP}" -lq in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -tq in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -lq --loose-trailing in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -tq --loose-trailing in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
cat "${in_lz}" > in0.lz
printf "${header}${body}" >> in0.lz # trailing data
"${LZIP}" -lq in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -tq in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -lq --loose-trailing in0.lz
[ $? = 0 ] || test_failed $LINENO ${header}
"${LZIP}" -t --loose-trailing in0.lz
[ $? = 0 ] || test_failed $LINENO ${header}
"${LZIP}" -lq --loose-trailing --trailing-error in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
"${LZIP}" -tq --loose-trailing --trailing-error in0.lz
[ $? = 2 ] || test_failed $LINENO ${header}
done
else
printf "\nwarning: skipping header test: 'printf' does not work on your system."
fi
rm -f in0.lz
cat "${in_lz}" "${in_lz}" "${in_lz}" > in3.lz || framework_failure cat "${in_lz}" "${in_lz}" "${in_lz}" > in3.lz || framework_failure
if dd if=in3.lz of=trunc.lz bs=14752 count=1 2> /dev/null && if dd if=in3.lz of=trunc.lz bs=14752 count=1 2> /dev/null &&
[ -e trunc.lz ] && cmp in2.lz trunc.lz > /dev/null 2>&1 ; then [ -e trunc.lz ] && cmp in2.lz trunc.lz > /dev/null 2>&1 ; then
@ -230,7 +284,7 @@ if dd if=in3.lz of=trunc.lz bs=14752 count=1 2> /dev/null &&
dd if=in3.lz of=trunc.lz bs=$i count=1 2> /dev/null dd if=in3.lz of=trunc.lz bs=$i count=1 2> /dev/null
"${LZIP}" -lq trunc.lz "${LZIP}" -lq trunc.lz
[ $? = 2 ] || test_failed $LINENO $i [ $? = 2 ] || test_failed $LINENO $i
"${LZIP}" -t trunc.lz 2> /dev/null "${LZIP}" -tq trunc.lz
[ $? = 2 ] || test_failed $LINENO $i [ $? = 2 ] || test_failed $LINENO $i
"${LZIP}" -tq < trunc.lz "${LZIP}" -tq < trunc.lz
[ $? = 2 ] || test_failed $LINENO $i [ $? = 2 ] || test_failed $LINENO $i
@ -242,6 +296,7 @@ if dd if=in3.lz of=trunc.lz bs=14752 count=1 2> /dev/null &&
else else
printf "\nwarning: skipping truncation test: 'dd' does not work on your system." printf "\nwarning: skipping truncation test: 'dd' does not work on your system."
fi fi
rm -f in3.lz trunc.lz
cat "${in_lz}" > ingin.lz || framework_failure cat "${in_lz}" > ingin.lz || framework_failure
printf "g" >> ingin.lz || framework_failure printf "g" >> ingin.lz || framework_failure
@ -254,6 +309,7 @@ cmp in copy || test_failed $LINENO
"${LZIP}" -t < ingin.lz || test_failed $LINENO "${LZIP}" -t < ingin.lz || test_failed $LINENO
"${LZIP}" -d < ingin.lz > copy || test_failed $LINENO "${LZIP}" -d < ingin.lz > copy || test_failed $LINENO
cmp in copy || test_failed $LINENO cmp in copy || test_failed $LINENO
rm -f ingin.lz
echo echo
if [ ${fail} = 0 ] ; then if [ ${fail} = 0 ] ; then