lzip/lzlib
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.5~rc1.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-20 20:18:02 +01:00
parent 0384b57edf
commit 81ec456584
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
21 changed files with 318 additions and 279 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTHORS
View file

@ -4,4 +4,4 @@ The ideas embodied in lzlib are due to (at least) the following people:
Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrey Markov (for Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrey Markov (for
the definition of Markov chains), G.N.N. Martin (for the definition of the definition of Markov chains), G.N.N. Martin (for the definition of
range encoding), Igor Pavlov (for putting all the above together in range encoding), Igor Pavlov (for putting all the above together in
LZMA), and Julian Seward (for bzip2's CLI and the idea of unzcrash). LZMA), and Julian Seward (for bzip2's CLI).

7
ChangeLog
View file

@ -1,3 +1,10 @@
2013-07-28 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.5-rc1 released.
* Removed decompression support for version 0 files.
* The LZ_compress_sync_flush mechanism has been fixed (again).
* Minor fixes.
2013-05-28 Antonio Diaz Diaz <antonio@gnu.org> 2013-05-28 Antonio Diaz Diaz <antonio@gnu.org>
* Version 1.4 released. * Version 1.4 released.

6
INSTALL
View file

@ -1,7 +1,7 @@
Requirements Requirements
------------ ------------
You will need a C compiler. You will need a C compiler.
I use gcc 4.8.0 and 3.3.6, but the code should compile with any I use gcc 4.8.1 and 3.3.6, but the code should compile with any
standards compliant compiler. standards compliant compiler.
Gcc is available at http://gcc.gnu.org. Gcc is available at http://gcc.gnu.org.
@ -10,9 +10,9 @@ Procedure
--------- ---------
1. Unpack the archive if you have not done so already: 1. Unpack the archive if you have not done so already:
lzip -cd lzlib[version].tar.lz | tar -xf - tar -xf lzlib[version].tar.lz
or or
gzip -cd lzlib[version].tar.gz | tar -xf - lzip -cd lzlib[version].tar.lz | tar -xf -
This creates the directory ./lzlib[version] containing the source from This creates the directory ./lzlib[version] containing the source from
the main archive. the main archive.

23
NEWS
View file

@ -1,21 +1,8 @@
Changes in version 1.4: Changes in version 1.5:
Multi-step trials have been implemented. Decompression support for deprecated version 0 files has been removed.
Compression ratio has been slightly increased. A bug has been fixed that would make an instance of "struct LZ_Encoder"
unresponsive if "LZ_compress_sync_flush" is called at the wrong moment.
Compression time has been reduced by 8%. Minor fixes.
Decompression time has been reduced by 7%.
Arguments and return values of functions in lzlib.h have been changed
from 'long long' to 'unsigned long long'.
The minimum size of the input compression buffer has been reduced to 64KiB.
"LZ_decompress_read" now tells "LZ_header_error" from "LZ_unexpected_eof"
the same way as lzip does when the EOF happens at the header.
The target "install-as-lzip" has been added to the Makefile.
The target "install-bin" has been added to the Makefile.

19
README
View file

@ -5,6 +5,10 @@ and decompression functions, including integrity checking of the
decompressed data. The compressed data format used by the library is the decompressed data. The compressed data format used by the library is the
lzip format. Lzlib is written in C. lzip format. Lzlib is written in C.
The lzip file format is designed for long-term data archiving. It is
clean, provides very safe 4 factor integrity checking, and is backed by
the recovery capabilities of lziprecover.
The functions and variables forming the interface of the compression The functions and variables forming the interface of the compression
library are declared in the file lzlib.h. Usage examples of the library library are declared in the file lzlib.h. Usage examples of the library
are given in the files main.c and bbexample.c from the source are given in the files main.c and bbexample.c from the source
@ -31,9 +35,18 @@ any signal handler. The decoder checks the consistency of the compressed
data, so the library should never crash even in case of corrupted input. data, so the library should never crash even in case of corrupted input.
Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov
chain-Algorithm) algorithm. The original LZMA algorithm was designed by chain-Algorithm) algorithm. The high compression of LZMA comes from
Igor Pavlov. For a description of the LZMA algorithm, see the Lzip combining two basic, well-proven compression ideas: sliding dictionaries
manual. (LZ77/78) and markov models (the thing used by every compression
algorithm that uses a range encoder or similar order-0 entropy coder as
its last stage) with segregation of contexts according to what the bits
are used for.
The ideas embodied in lzlib are due to (at least) the following people:
Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrey Markov (for
the definition of Markov chains), G.N.N. Martin (for the definition of
range encoding), Igor Pavlov (for putting all the above together in
LZMA), and Julian Seward (for bzip2's CLI).
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.

2
bbexample.c
View file

@ -1,4 +1,4 @@
/* Buff to buff example - A test program for the lzlib library /* Buff to buff example - Test program for the lzlib library
Copyright (C) 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you have unlimited permission This program is free software: you have unlimited permission

2
cbuffer.c
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify

18
configure vendored
View file

@ -1,17 +1,17 @@
#! /bin/sh #! /bin/sh
# configure script for Lzlib - A compression library for lzip files # configure script for Lzlib - Compression library for lzip files
# Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. # Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
# #
# This configure script is free software: you have unlimited permission # This configure script is free software: you have unlimited permission
# to copy, distribute and modify it. # to copy, distribute and modify it.
pkgname=lzlib pkgname=lzlib
pkgversion=1.4 pkgversion=1.5-rc1
soversion=1 soversion=1
progname=minilzip progname=minilzip
progname_shared= progname_shared=
libname=lz libname=lz
srctrigger=${libname}lib.h srctrigger=doc/${pkgname}.texinfo
# clear some things potentially inherited from environment. # clear some things potentially inherited from environment.
LC_ALL=C LC_ALL=C
@ -113,14 +113,14 @@ while [ $# != 0 ] ; do
*=* | *-*-*) ;; *=* | *-*-*) ;;
*) *)
echo "configure: unrecognized option: '${option}'" 1>&2 echo "configure: unrecognized option: '${option}'" 1>&2
echo "Try 'configure --help' for more information." echo "Try 'configure --help' for more information." 1>&2
exit 1 ;; exit 1 ;;
esac esac
# Check if the option took a separate argument # Check if the option took a separate argument
if [ "${arg2}" = yes ] ; then if [ "${arg2}" = yes ] ; then
if [ $# != 0 ] ; then args="${args} \"$1\"" ; shift if [ $# != 0 ] ; then args="${args} \"$1\"" ; shift
else echo "configure: Missing argument to \"${option}\"" 1>&2 else echo "configure: Missing argument to '${option}'" 1>&2
exit 1 exit 1
fi fi
fi fi
@ -138,10 +138,8 @@ if [ -z "${srcdir}" ] ; then
fi fi
if [ ! -r "${srcdir}/${srctrigger}" ] ; then if [ ! -r "${srcdir}/${srctrigger}" ] ; then
exec 1>&2 echo "configure: Can't find sources in ${srcdir} ${srcdirtext}" 1>&2
echo echo "configure: (At least ${srctrigger} is missing)." 1>&2
echo "configure: Can't find sources in ${srcdir} ${srcdirtext}"
echo "configure: (At least ${srctrigger} is missing)."
exit 1 exit 1
fi fi
@ -181,7 +179,7 @@ echo "CFLAGS = ${CFLAGS}"
echo "LDFLAGS = ${LDFLAGS}" echo "LDFLAGS = ${LDFLAGS}"
rm -f Makefile rm -f Makefile
cat > Makefile << EOF cat > Makefile << EOF
# Makefile for Lzlib - A compression library for lzip files # Makefile for Lzlib - Compression library for lzip files
# Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. # Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
# This file was generated automatically by configure. Do not edit. # This file was generated automatically by configure. Do not edit.
# #

57
decoder.c
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -28,16 +28,13 @@
static bool LZd_verify_trailer( struct LZ_decoder * const decoder ) static bool LZd_verify_trailer( struct LZ_decoder * const decoder )
{ {
File_trailer trailer; File_trailer trailer;
const int trailer_size = Ft_versioned_size( decoder->member_version );
const unsigned long long member_size = const unsigned long long member_size =
decoder->rdec->member_position + trailer_size; decoder->rdec->member_position + Ft_size;
int size = Rd_read_data( decoder->rdec, trailer, trailer_size ); int size = Rd_read_data( decoder->rdec, trailer, Ft_size );
if( size < trailer_size ) if( size < Ft_size )
return false; return false;
if( decoder->member_version == 0 ) Ft_set_member_size( trailer, member_size );
return ( decoder->rdec->code == 0 && return ( decoder->rdec->code == 0 &&
Ft_get_data_crc( trailer ) == LZd_crc( decoder ) && Ft_get_data_crc( trailer ) == LZd_crc( decoder ) &&
Ft_get_data_size( trailer ) == LZd_data_position( decoder ) && Ft_get_data_size( trailer ) == LZd_data_position( decoder ) &&
@ -49,38 +46,39 @@ static bool LZd_verify_trailer( struct LZ_decoder * const decoder )
3 = trailer error, 4 = unknown marker found. */ 3 = trailer error, 4 = unknown marker found. */
static int LZd_decode_member( struct LZ_decoder * const decoder ) static int LZd_decode_member( struct LZ_decoder * const decoder )
{ {
struct Range_decoder * const rdec = decoder->rdec;
State * const state = &decoder->state; State * const state = &decoder->state;
if( decoder->member_finished ) return 0; if( decoder->member_finished ) return 0;
if( !Rd_try_reload( decoder->rdec, false ) ) return 0; if( !Rd_try_reload( rdec, false ) ) return 0;
if( decoder->verify_trailer_pending ) if( decoder->verify_trailer_pending )
{ {
if( Rd_available_bytes( decoder->rdec ) < Ft_versioned_size( decoder->member_version ) && if( Rd_available_bytes( rdec ) < Ft_size && !rdec->at_stream_end )
!decoder->rdec->at_stream_end )
return 0; return 0;
decoder->verify_trailer_pending = false; decoder->verify_trailer_pending = false;
decoder->member_finished = true; decoder->member_finished = true;
if( LZd_verify_trailer( decoder ) ) return 0; else return 3; if( LZd_verify_trailer( decoder ) ) return 0; else return 3;
} }
while( !Rd_finished( decoder->rdec ) ) while( !Rd_finished( rdec ) )
{ {
const int pos_state = LZd_data_position( decoder ) & pos_state_mask; const int pos_state = LZd_data_position( decoder ) & pos_state_mask;
if( !Rd_enough_available_bytes( decoder->rdec ) || if( !Rd_enough_available_bytes( rdec ) ||
!LZd_enough_free_bytes( decoder ) ) !LZd_enough_free_bytes( decoder ) )
return 0; return 0;
if( Rd_decode_bit( decoder->rdec, &decoder->bm_match[*state][pos_state] ) == 0 ) /* 1st bit */ if( Rd_decode_bit( rdec, &decoder->bm_match[*state][pos_state] ) == 0 ) /* 1st bit */
{ {
const uint8_t prev_byte = LZd_get_prev_byte( decoder ); const uint8_t prev_byte = LZd_get_prev_byte( decoder );
if( St_is_char( *state ) ) if( St_is_char( *state ) )
{ {
*state -= ( *state < 4 ) ? *state : 3; *state -= ( *state < 4 ) ? *state : 3;
LZd_put_byte( decoder, Rd_decode_tree( decoder->rdec, LZd_put_byte( decoder, Rd_decode_tree( rdec,
decoder->bm_literal[get_lit_state(prev_byte)], 8 ) ); decoder->bm_literal[get_lit_state(prev_byte)], 8 ) );
} }
else else
{ {
*state -= ( *state < 10 ) ? 3 : 6; *state -= ( *state < 10 ) ? 3 : 6;
LZd_put_byte( decoder, Rd_decode_matched( decoder->rdec, LZd_put_byte( decoder, Rd_decode_matched( rdec,
decoder->bm_literal[get_lit_state(prev_byte)], decoder->bm_literal[get_lit_state(prev_byte)],
LZd_get_byte( decoder, decoder->rep0 ) ) ); LZd_get_byte( decoder, decoder->rep0 ) ) );
} }
@ -88,22 +86,22 @@ static int LZd_decode_member( struct LZ_decoder * const decoder )
else else
{ {
int len; int len;
if( Rd_decode_bit( decoder->rdec, &decoder->bm_rep[*state] ) == 1 ) /* 2nd bit */ if( Rd_decode_bit( rdec, &decoder->bm_rep[*state] ) == 1 ) /* 2nd bit */
{ {
if( Rd_decode_bit( decoder->rdec, &decoder->bm_rep0[*state] ) == 0 ) /* 3rd bit */ if( Rd_decode_bit( rdec, &decoder->bm_rep0[*state] ) == 0 ) /* 3rd bit */
{ {
if( Rd_decode_bit( decoder->rdec, &decoder->bm_len[*state][pos_state] ) == 0 ) /* 4th bit */ if( Rd_decode_bit( rdec, &decoder->bm_len[*state][pos_state] ) == 0 ) /* 4th bit */
{ *state = St_set_short_rep( *state ); { *state = St_set_short_rep( *state );
LZd_put_byte( decoder, LZd_get_byte( decoder, decoder->rep0 ) ); continue; } LZd_put_byte( decoder, LZd_get_byte( decoder, decoder->rep0 ) ); continue; }
} }
else else
{ {
unsigned distance; unsigned distance;
if( Rd_decode_bit( decoder->rdec, &decoder->bm_rep1[*state] ) == 0 ) /* 4th bit */ if( Rd_decode_bit( rdec, &decoder->bm_rep1[*state] ) == 0 ) /* 4th bit */
distance = decoder->rep1; distance = decoder->rep1;
else else
{ {
if( Rd_decode_bit( decoder->rdec, &decoder->bm_rep2[*state] ) == 0 ) /* 5th bit */ if( Rd_decode_bit( rdec, &decoder->bm_rep2[*state] ) == 0 ) /* 5th bit */
distance = decoder->rep2; distance = decoder->rep2;
else else
{ distance = decoder->rep3; decoder->rep3 = decoder->rep2; } { distance = decoder->rep3; decoder->rep3 = decoder->rep2; }
@ -113,42 +111,41 @@ static int LZd_decode_member( struct LZ_decoder * const decoder )
decoder->rep0 = distance; decoder->rep0 = distance;
} }
*state = St_set_rep( *state ); *state = St_set_rep( *state );
len = min_match_len + Rd_decode_len( decoder->rdec, &decoder->rep_len_model, pos_state ); len = min_match_len + Rd_decode_len( rdec, &decoder->rep_len_model, pos_state );
} }
else else
{ {
int dis_slot; int dis_slot;
const unsigned rep0_saved = decoder->rep0; const unsigned rep0_saved = decoder->rep0;
len = min_match_len + Rd_decode_len( decoder->rdec, &decoder->match_len_model, pos_state ); len = min_match_len + Rd_decode_len( rdec, &decoder->match_len_model, pos_state );
dis_slot = Rd_decode_tree6( decoder->rdec, decoder->bm_dis_slot[get_dis_state(len)] ); dis_slot = Rd_decode_tree6( rdec, decoder->bm_dis_slot[get_dis_state(len)] );
if( dis_slot < start_dis_model ) decoder->rep0 = dis_slot; if( dis_slot < start_dis_model ) decoder->rep0 = dis_slot;
else else
{ {
const int direct_bits = ( dis_slot >> 1 ) - 1; const int direct_bits = ( dis_slot >> 1 ) - 1;
decoder->rep0 = ( 2 | ( dis_slot & 1 ) ) << direct_bits; decoder->rep0 = ( 2 | ( dis_slot & 1 ) ) << direct_bits;
if( dis_slot < end_dis_model ) if( dis_slot < end_dis_model )
decoder->rep0 += Rd_decode_tree_reversed( decoder->rdec, decoder->rep0 += Rd_decode_tree_reversed( rdec,
decoder->bm_dis + decoder->rep0 - dis_slot - 1, decoder->bm_dis + decoder->rep0 - dis_slot - 1,
direct_bits ); direct_bits );
else else
{ {
decoder->rep0 += Rd_decode( decoder->rdec, direct_bits - dis_align_bits ) << dis_align_bits; decoder->rep0 += Rd_decode( rdec, direct_bits - dis_align_bits ) << dis_align_bits;
decoder->rep0 += Rd_decode_tree_reversed4( decoder->rdec, decoder->bm_align ); decoder->rep0 += Rd_decode_tree_reversed4( rdec, decoder->bm_align );
if( decoder->rep0 == 0xFFFFFFFFU ) /* Marker found */ if( decoder->rep0 == 0xFFFFFFFFU ) /* Marker found */
{ {
decoder->rep0 = rep0_saved; decoder->rep0 = rep0_saved;
Rd_normalize( decoder->rdec ); Rd_normalize( rdec );
if( len == min_match_len ) /* End Of Stream marker */ if( len == min_match_len ) /* End Of Stream marker */
{ {
if( Rd_available_bytes( decoder->rdec ) < Ft_versioned_size( decoder->member_version ) && if( Rd_available_bytes( rdec ) < Ft_size && !rdec->at_stream_end )
!decoder->rdec->at_stream_end )
{ decoder->verify_trailer_pending = true; return 0; } { decoder->verify_trailer_pending = true; return 0; }
decoder->member_finished = true; decoder->member_finished = true;
if( LZd_verify_trailer( decoder ) ) return 0; else return 3; if( LZd_verify_trailer( decoder ) ) return 0; else return 3;
} }
if( len == min_match_len + 1 ) /* Sync Flush marker */ if( len == min_match_len + 1 ) /* Sync Flush marker */
{ {
if( Rd_try_reload( decoder->rdec, true ) ) continue; if( Rd_try_reload( rdec, true ) ) continue;
else return 0; else return 0;
} }
return 4; return 4;

8
decoder.h
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -33,8 +33,8 @@ struct Range_decoder
unsigned long long member_position; unsigned long long member_position;
uint32_t code; uint32_t code;
uint32_t range; uint32_t range;
bool reload_pending;
bool at_stream_end; bool at_stream_end;
bool reload_pending;
}; };
static inline bool Rd_init( struct Range_decoder * const rdec ) static inline bool Rd_init( struct Range_decoder * const rdec )
@ -43,8 +43,8 @@ static inline bool Rd_init( struct Range_decoder * const rdec )
rdec->member_position = 0; rdec->member_position = 0;
rdec->code = 0; rdec->code = 0;
rdec->range = 0xFFFFFFFFU; rdec->range = 0xFFFFFFFFU;
rdec->reload_pending = false;
rdec->at_stream_end = false; rdec->at_stream_end = false;
rdec->reload_pending = false;
return true; return true;
} }
@ -311,7 +311,6 @@ struct LZ_decoder
unsigned long long partial_data_pos; unsigned long long partial_data_pos;
int dictionary_size; int dictionary_size;
uint32_t crc; uint32_t crc;
int member_version;
bool member_finished; bool member_finished;
bool verify_trailer_pending; bool verify_trailer_pending;
unsigned rep0; /* rep[0-3] latest four distances */ unsigned rep0; /* rep[0-3] latest four distances */
@ -391,7 +390,6 @@ static inline bool LZd_init( struct LZ_decoder * const decoder,
return false; return false;
decoder->partial_data_pos = 0; decoder->partial_data_pos = 0;
decoder->crc = 0xFFFFFFFFU; decoder->crc = 0xFFFFFFFFU;
decoder->member_version = Fh_version( header );
decoder->member_finished = false; decoder->member_finished = false;
decoder->verify_trailer_pending = false; decoder->verify_trailer_pending = false;
decoder->rep0 = 0; decoder->rep0 = 0;

125
doc/lzlib.info
View file

@ -3,7 +3,7 @@ lzlib.texinfo.
INFO-DIR-SECTION Data Compression INFO-DIR-SECTION Data Compression
START-INFO-DIR-ENTRY START-INFO-DIR-ENTRY
* Lzlib: (lzlib). A compression library for lzip files * Lzlib: (lzlib). Compression library for lzip files
END-INFO-DIR-ENTRY END-INFO-DIR-ENTRY
 
@ -12,22 +12,22 @@ File: lzlib.info, Node: Top, Next: Introduction, Up: (dir)
Lzlib Manual Lzlib Manual
************ ************
This manual is for Lzlib (version 1.4, 28 May 2013). This manual is for Lzlib (version 1.5-rc1, 28 July 2013).
* Menu: * Menu:
* Introduction:: Purpose and features of Lzlib * Introduction:: Purpose and features of Lzlib
* Library Version:: Checking library version * Library version:: Checking library version
* Buffering:: Sizes of Lzlib's buffers * Buffering:: Sizes of Lzlib's buffers
* Parameter Limits:: Min / max values for some parameters * Parameter limits:: Min / max values for some parameters
* Compression Functions:: Descriptions of the compression functions * Compression functions:: Descriptions of the compression functions
* Decompression Functions:: Descriptions of the decompression functions * Decompression functions:: Descriptions of the decompression functions
* Error Codes:: Meaning of codes returned by functions * Error codes:: Meaning of codes returned by functions
* Error Messages:: Error messages corresponding to error codes * Error messages:: Error messages corresponding to error codes
* Data Format:: Detailed format of the compressed data * Data format:: Detailed format of the compressed data
* Examples:: A small tutorial with examples * Examples:: A small tutorial with examples
* Problems:: Reporting bugs * Problems:: Reporting bugs
* Concept Index:: Index of concepts * Concept index:: Index of concepts
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
@ -36,7 +36,7 @@ This manual is for Lzlib (version 1.4, 28 May 2013).
copy, distribute and modify it. copy, distribute and modify it.
 
File: lzlib.info, Node: Introduction, Next: Library Version, Prev: Top, Up: Top File: lzlib.info, Node: Introduction, Next: Library version, Prev: Top, Up: Top
1 Introduction 1 Introduction
************** **************
@ -46,6 +46,10 @@ and decompression functions, including integrity checking of the
decompressed data. The compressed data format used by the library is the decompressed data. The compressed data format used by the library is the
lzip format. Lzlib is written in C. lzip format. Lzlib is written in C.
The lzip file format is designed for long-term data archiving. It is
clean, provides very safe 4 factor integrity checking, and is backed by
the recovery capabilities of lziprecover.
The functions and variables forming the interface of the compression The functions and variables forming the interface of the compression
library are declared in the file `lzlib.h'. Usage examples of the library are declared in the file `lzlib.h'. Usage examples of the
library are given in the files `main.c' and `bbexample.c' from the library are given in the files `main.c' and `bbexample.c' from the
@ -73,14 +77,23 @@ compressed data, so the library should never crash even in case of
corrupted input. corrupted input.
Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov
chain-Algorithm) algorithm. The original LZMA algorithm was designed by chain-Algorithm) algorithm. The high compression of LZMA comes from
Igor Pavlov. For a description of the LZMA algorithm, see the Lzip combining two basic, well-proven compression ideas: sliding dictionaries
manual. (LZ77/78) and markov models (the thing used by every compression
algorithm that uses a range encoder or similar order-0 entropy coder as
its last stage) with segregation of contexts according to what the bits
are used for.
The ideas embodied in lzlib are due to (at least) the following
people: Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrey
Markov (for the definition of Markov chains), G.N.N. Martin (for the
definition of range encoding), Igor Pavlov (for putting all the above
together in LZMA), and Julian Seward (for bzip2's CLI).
 
File: lzlib.info, Node: Library Version, Next: Buffering, Prev: Introduction, Up: Top File: lzlib.info, Node: Library version, Next: Buffering, Prev: Introduction, Up: Top
2 Library Version 2 Library version
***************** *****************
-- Function: const char * LZ_version ( void ) -- Function: const char * LZ_version ( void )
@ -98,7 +111,7 @@ application.
error( "bad library version" ); error( "bad library version" );
 
File: lzlib.info, Node: Buffering, Next: Parameter Limits, Prev: Library Version, Up: Top File: lzlib.info, Node: Buffering, Next: Parameter limits, Prev: Library version, Up: Top
3 Buffering 3 Buffering
*********** ***********
@ -126,9 +139,9 @@ minimum sizes:
member currently being decompressed or 64KiB, whichever is larger. member currently being decompressed or 64KiB, whichever is larger.
 
File: lzlib.info, Node: Parameter Limits, Next: Compression Functions, Prev: Buffering, Up: Top File: lzlib.info, Node: Parameter limits, Next: Compression functions, Prev: Buffering, Up: Top
4 Parameter Limits 4 Parameter limits
****************** ******************
These functions provide minimum and maximum values for some parameters. These functions provide minimum and maximum values for some parameters.
@ -155,9 +168,9 @@ Current values are shown in square brackets.
Returns the largest valid match length limit [273]. Returns the largest valid match length limit [273].
 
File: lzlib.info, Node: Compression Functions, Next: Decompression Functions, Prev: Parameter Limits, Up: Top File: lzlib.info, Node: Compression functions, Next: Decompression functions, Prev: Parameter limits, Up: Top
5 Compression Functions 5 Compression functions
*********************** ***********************
These are the functions used to compress data. In case of error, all of These are the functions used to compress data. In case of error, all of
@ -254,7 +267,7 @@ calling `LZ_compress_errno' before using it.
-- Function: enum LZ_Errno LZ_compress_errno ( struct LZ_Encoder * -- Function: enum LZ_Errno LZ_compress_errno ( struct LZ_Encoder *
const ENCODER ) const ENCODER )
Returns the current error code for ENCODER (*note Error Codes::). Returns the current error code for ENCODER (*note Error codes::).
-- Function: int LZ_compress_finished ( struct LZ_Encoder * const -- Function: int LZ_compress_finished ( struct LZ_Encoder * const
ENCODER ) ENCODER )
@ -287,9 +300,9 @@ calling `LZ_compress_errno' before using it.
perhaps not yet read. perhaps not yet read.
 
File: lzlib.info, Node: Decompression Functions, Next: Error Codes, Prev: Compression Functions, Up: Top File: lzlib.info, Node: Decompression functions, Next: Error codes, Prev: Compression functions, Up: Top
6 Decompression Functions 6 Decompression functions
************************* *************************
These are the functions used to decompress data. In case of error, all These are the functions used to decompress data. In case of error, all
@ -370,7 +383,7 @@ verified by calling `LZ_decompress_errno' before using it.
-- Function: enum LZ_Errno LZ_decompress_errno ( struct LZ_Decoder * -- Function: enum LZ_Errno LZ_decompress_errno ( struct LZ_Decoder *
const DECODER ) const DECODER )
Returns the current error code for DECODER (*note Error Codes::). Returns the current error code for DECODER (*note Error codes::).
-- Function: int LZ_decompress_finished ( struct LZ_Decoder * const -- Function: int LZ_decompress_finished ( struct LZ_Decoder * const
DECODER ) DECODER )
@ -419,9 +432,9 @@ verified by calling `LZ_decompress_errno' before using it.
but perhaps not yet read. but perhaps not yet read.
 
File: lzlib.info, Node: Error Codes, Next: Error Messages, Prev: Decompression Functions, Up: Top File: lzlib.info, Node: Error codes, Next: Error messages, Prev: Decompression functions, Up: Top
7 Error Codes 7 Error codes
************* *************
Most library functions return -1 to indicate that they have failed. But Most library functions return -1 to indicate that they have failed. But
@ -471,9 +484,9 @@ whether a call failed. If the call failed, then you can examine
Problems::). Problems::).
 
File: lzlib.info, Node: Error Messages, Next: Data Format, Prev: Error Codes, Up: Top File: lzlib.info, Node: Error messages, Next: Data format, Prev: Error codes, Up: Top
8 Error Messages 8 Error messages
**************** ****************
-- Function: const char * LZ_strerror ( const enum LZ_Errno LZ_ERRNO ) -- Function: const char * LZ_strerror ( const enum LZ_Errno LZ_ERRNO )
@ -487,9 +500,9 @@ File: lzlib.info, Node: Error Messages, Next: Data Format, Prev: Error Codes,
`LZ_(de)compress_errno'. `LZ_(de)compress_errno'.
 
File: lzlib.info, Node: Data Format, Next: Examples, Prev: Error Messages, Up: Top File: lzlib.info, Node: Data format, Next: Examples, Prev: Error messages, Up: Top
9 Data Format 9 Data format
************* *************
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -545,6 +558,8 @@ with no additional information before, between, or after them.
The lzma stream, finished by an end of stream marker. Uses default The lzma stream, finished by an end of stream marker. Uses default
values for encoder properties. See the lzip manual for a full values for encoder properties. See the lzip manual for a full
description. description.
Lzip only uses the LZMA marker `2' ("End Of Stream" marker). Lzlib
also uses the LZMA marker `3' ("Sync Flush" marker).
`CRC32 (4 bytes)' `CRC32 (4 bytes)'
CRC of the uncompressed original data. CRC of the uncompressed original data.
@ -560,7 +575,7 @@ with no additional information before, between, or after them.
 
File: lzlib.info, Node: Examples, Next: Problems, Prev: Data Format, Up: Top File: lzlib.info, Node: Examples, Next: Problems, Prev: Data format, Up: Top
10 A small tutorial with examples 10 A small tutorial with examples
********************************* *********************************
@ -680,7 +695,7 @@ next member in case of data error.
7) LZ_decompress_close 7) LZ_decompress_close
 
File: lzlib.info, Node: Problems, Next: Concept Index, Prev: Examples, Up: Top File: lzlib.info, Node: Problems, Next: Concept index, Prev: Examples, Up: Top
11 Reporting Bugs 11 Reporting Bugs
***************** *****************
@ -696,9 +711,9 @@ by running `minilzip --version' or in `LZ_version_string' from
`lzlib.h'. `lzlib.h'.
 
File: lzlib.info, Node: Concept Index, Prev: Problems, Up: Top File: lzlib.info, Node: Concept index, Prev: Problems, Up: Top
Concept Index Concept index
************* *************
[index] [index]
@ -706,34 +721,34 @@ Concept Index
* buffering: Buffering. (line 6) * buffering: Buffering. (line 6)
* bugs: Problems. (line 6) * bugs: Problems. (line 6)
* compression functions: Compression Functions. (line 6) * compression functions: Compression functions. (line 6)
* data format: Data Format. (line 6) * data format: Data format. (line 6)
* decompression functions: Decompression Functions. * decompression functions: Decompression functions.
(line 6) (line 6)
* error codes: Error Codes. (line 6) * error codes: Error codes. (line 6)
* error messages: Error Messages. (line 6) * error messages: Error messages. (line 6)
* examples: Examples. (line 6) * examples: Examples. (line 6)
* getting help: Problems. (line 6) * getting help: Problems. (line 6)
* introduction: Introduction. (line 6) * introduction: Introduction. (line 6)
* library version: Library Version. (line 6) * library version: Library version. (line 6)
* parameter limits: Parameter Limits. (line 6) * parameter limits: Parameter limits. (line 6)
 
Tag Table: Tag Table:
Node: Top219 Node: Top217
Node: Introduction1319 Node: Introduction1322
Node: Library Version3165 Node: Library version3903
Node: Buffering3810 Node: Buffering4548
Node: Parameter Limits4929 Node: Parameter limits5667
Node: Compression Functions5886 Node: Compression functions6624
Node: Decompression Functions12096 Node: Decompression functions12834
Node: Error Codes18257 Node: Error codes18995
Node: Error Messages20196 Node: Error messages20934
Node: Data Format20775 Node: Data format21513
Node: Examples23268 Node: Examples24136
Node: Problems27351 Node: Problems28219
Node: Concept Index27923 Node: Concept index28791
 
End Tag Table End Tag Table

83
doc/lzlib.texinfo
View file

@ -6,19 +6,19 @@
@finalout @finalout
@c %**end of header @c %**end of header
@set UPDATED 28 May 2013 @set UPDATED 28 July 2013
@set VERSION 1.4 @set VERSION 1.5-rc1
@dircategory Data Compression @dircategory Data Compression
@direntry @direntry
* Lzlib: (lzlib). A compression library for lzip files * Lzlib: (lzlib). Compression library for lzip files
@end direntry @end direntry
@ifnothtml @ifnothtml
@titlepage @titlepage
@title Lzlib @title Lzlib
@subtitle A compression library for lzip files @subtitle Compression library for lzip files
@subtitle for Lzlib version @value{VERSION}, @value{UPDATED} @subtitle for Lzlib version @value{VERSION}, @value{UPDATED}
@author by Antonio Diaz Diaz @author by Antonio Diaz Diaz
@ -36,17 +36,17 @@ This manual is for Lzlib (version @value{VERSION}, @value{UPDATED}).
@menu @menu
* Introduction:: Purpose and features of Lzlib * Introduction:: Purpose and features of Lzlib
* Library Version:: Checking library version * Library version:: Checking library version
* Buffering:: Sizes of Lzlib's buffers * Buffering:: Sizes of Lzlib's buffers
* Parameter Limits:: Min / max values for some parameters * Parameter limits:: Min / max values for some parameters
* Compression Functions:: Descriptions of the compression functions * Compression functions:: Descriptions of the compression functions
* Decompression Functions:: Descriptions of the decompression functions * Decompression functions:: Descriptions of the decompression functions
* Error Codes:: Meaning of codes returned by functions * Error codes:: Meaning of codes returned by functions
* Error Messages:: Error messages corresponding to error codes * Error messages:: Error messages corresponding to error codes
* Data Format:: Detailed format of the compressed data * Data format:: Detailed format of the compressed data
* Examples:: A small tutorial with examples * Examples:: A small tutorial with examples
* Problems:: Reporting bugs * Problems:: Reporting bugs
* Concept Index:: Index of concepts * Concept index:: Index of concepts
@end menu @end menu
@sp 1 @sp 1
@ -65,6 +65,10 @@ and decompression functions, including integrity checking of the
decompressed data. The compressed data format used by the library is the decompressed data. The compressed data format used by the library is the
lzip format. Lzlib is written in C. lzip format. Lzlib is written in C.
The lzip file format is designed for long-term data archiving. It is
clean, provides very safe 4 factor integrity checking, and is backed by
the recovery capabilities of lziprecover.
The functions and variables forming the interface of the compression The functions and variables forming the interface of the compression
library are declared in the file @samp{lzlib.h}. Usage examples of the library are declared in the file @samp{lzlib.h}. Usage examples of the
library are given in the files @samp{main.c} and @samp{bbexample.c} from library are given in the files @samp{main.c} and @samp{bbexample.c} from
@ -91,13 +95,22 @@ any signal handler. The decoder checks the consistency of the compressed
data, so the library should never crash even in case of corrupted input. data, so the library should never crash even in case of corrupted input.
Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov Lzlib implements a simplified version of the LZMA (Lempel-Ziv-Markov
chain-Algorithm) algorithm. The original LZMA algorithm was designed by chain-Algorithm) algorithm. The high compression of LZMA comes from
Igor Pavlov. For a description of the LZMA algorithm, see the Lzip combining two basic, well-proven compression ideas: sliding dictionaries
manual. (LZ77/78) and markov models (the thing used by every compression
algorithm that uses a range encoder or similar order-0 entropy coder as
its last stage) with segregation of contexts according to what the bits
are used for.
The ideas embodied in lzlib are due to (at least) the following people:
Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrey Markov (for
the definition of Markov chains), G.N.N. Martin (for the definition of
range encoding), Igor Pavlov (for putting all the above together in
LZMA), and Julian Seward (for bzip2's CLI).
@node Library Version @node Library version
@chapter Library Version @chapter Library version
@cindex library version @cindex library version
@deftypefun {const char *} LZ_version ( void ) @deftypefun {const char *} LZ_version ( void )
@ -150,8 +163,8 @@ whichever is larger.
@end itemize @end itemize
@node Parameter Limits @node Parameter limits
@chapter Parameter Limits @chapter Parameter limits
@cindex parameter limits @cindex parameter limits
These functions provide minimum and maximum values for some parameters. These functions provide minimum and maximum values for some parameters.
@ -182,8 +195,8 @@ Returns the largest valid match length limit [273].
@end deftypefun @end deftypefun
@node Compression Functions @node Compression functions
@chapter Compression Functions @chapter Compression functions
@cindex compression functions @cindex compression functions
These are the functions used to compress data. In case of error, all of These are the functions used to compress data. In case of error, all of
@ -288,7 +301,7 @@ accept a @var{size} up to the returned number of bytes.
@deftypefun {enum LZ_Errno} LZ_compress_errno ( struct LZ_Encoder * const @var{encoder} ) @deftypefun {enum LZ_Errno} LZ_compress_errno ( struct LZ_Encoder * const @var{encoder} )
Returns the current error code for @var{encoder} (@pxref{Error Codes}). Returns the current error code for @var{encoder} (@pxref{Error codes}).
@end deftypefun @end deftypefun
@ -328,8 +341,8 @@ perhaps not yet read.
@end deftypefun @end deftypefun
@node Decompression Functions @node Decompression functions
@chapter Decompression Functions @chapter Decompression functions
@cindex decompression functions @cindex decompression functions
These are the functions used to decompress data. In case of error, all These are the functions used to decompress data. In case of error, all
@ -421,7 +434,7 @@ will accept a @var{size} up to the returned number of bytes.
@deftypefun {enum LZ_Errno} LZ_decompress_errno ( struct LZ_Decoder * const @var{decoder} ) @deftypefun {enum LZ_Errno} LZ_decompress_errno ( struct LZ_Decoder * const @var{decoder} )
Returns the current error code for @var{decoder} (@pxref{Error Codes}). Returns the current error code for @var{decoder} (@pxref{Error codes}).
@end deftypefun @end deftypefun
@ -480,8 +493,8 @@ perhaps not yet read.
@end deftypefun @end deftypefun
@node Error Codes @node Error codes
@chapter Error Codes @chapter Error codes
@cindex error codes @cindex error codes
Most library functions return -1 to indicate that they have failed. But Most library functions return -1 to indicate that they have failed. But
@ -538,8 +551,8 @@ A bug was detected in the library. Please, report it (@pxref{Problems}).
@end deftypevr @end deftypevr
@node Error Messages @node Error messages
@chapter Error Messages @chapter Error messages
@cindex error messages @cindex error messages
@deftypefun {const char *} LZ_strerror ( const enum LZ_Errno @var{lz_errno} ) @deftypefun {const char *} LZ_strerror ( const enum LZ_Errno @var{lz_errno} )
@ -553,8 +566,8 @@ The value of @var{lz_errno} normally comes from a call to
@end deftypefun @end deftypefun
@node Data Format @node Data format
@chapter Data Format @chapter Data format
@cindex data format @cindex data format
Perfection is reached, not when there is no longer anything to add, but Perfection is reached, not when there is no longer anything to add, but
@ -612,7 +625,9 @@ Valid values for dictionary size range from 4KiB to 512MiB.
@item Lzma stream @item Lzma stream
The lzma stream, finished by an end of stream marker. Uses default values The lzma stream, finished by an end of stream marker. Uses default values
for encoder properties. See the lzip manual for a full description. for encoder properties. See the lzip manual for a full description.@*
Lzip only uses the LZMA marker @samp{2} ("End Of Stream" marker). Lzlib
also uses the LZMA marker @samp{3} ("Sync Flush" marker).
@item CRC32 (4 bytes) @item CRC32 (4 bytes)
CRC of the uncompressed original data. CRC of the uncompressed original data.
@ -793,8 +808,8 @@ find by running @w{@samp{minilzip --version}} or in
@samp{LZ_version_string} from @samp{lzlib.h}. @samp{LZ_version_string} from @samp{lzlib.h}.
@node Concept Index @node Concept index
@unnumbered Concept Index @unnumbered Concept index
@printindex cp @printindex cp

6
doc/minilzip.1
View file

@ -1,12 +1,12 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1. .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1.
.TH MINILZIP "1" "May 2013" "Minilzip 1.4" "User Commands" .TH MINILZIP "1" "July 2013" "Minilzip 1.5-rc1" "User Commands"
.SH NAME .SH NAME
Minilzip \- reduces the size of files Minilzip \- reduces the size of files
.SH SYNOPSIS .SH SYNOPSIS
.B minilzip .B minilzip
[\fIoptions\fR] [\fIfiles\fR] [\fIoptions\fR] [\fIfiles\fR]
.SH DESCRIPTION .SH DESCRIPTION
Minilzip \- A test program for the lzlib library. Minilzip \- Test program for the lzlib library.
.SH OPTIONS .SH OPTIONS
.TP .TP
\fB\-h\fR, \fB\-\-help\fR \fB\-h\fR, \fB\-\-help\fR
@ -82,7 +82,7 @@ Report bugs to lzip\-bug@nongnu.org
Lzlib home page: http://www.nongnu.org/lzip/lzlib.html Lzlib home page: http://www.nongnu.org/lzip/lzlib.html
.SH COPYRIGHT .SH COPYRIGHT
Copyright \(co 2013 Antonio Diaz Diaz. Copyright \(co 2013 Antonio Diaz Diaz.
Using Lzlib 1.4 Using Lzlib 1.5\-rc1
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
.br .br
This is free software: you are free to change and redistribute it. This is free software: you are free to change and redistribute it.

75
encoder.c
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -51,7 +51,8 @@ static bool Mf_init( struct Matchfinder * const mf,
const int dict_size, const int match_len_limit ) const int dict_size, const int match_len_limit )
{ {
const int buffer_size_limit = ( 2 * dict_size ) + before_size + after_size; const int buffer_size_limit = ( 2 * dict_size ) + before_size + after_size;
int i, size; unsigned size;
int i;
mf->partial_data_pos = 0; mf->partial_data_pos = 0;
mf->match_len_limit = match_len_limit; mf->match_len_limit = match_len_limit;
@ -62,6 +63,7 @@ static bool Mf_init( struct Matchfinder * const mf,
16 + ( match_len_limit / 2 ) : 256; 16 + ( match_len_limit / 2 ) : 256;
mf->at_stream_end = false; mf->at_stream_end = false;
mf->been_flushed = false; mf->been_flushed = false;
mf->flushing = false;
mf->buffer_size = max( 65536, buffer_size_limit ); mf->buffer_size = max( 65536, buffer_size_limit );
mf->buffer = (uint8_t *)malloc( mf->buffer_size ); mf->buffer = (uint8_t *)malloc( mf->buffer_size );
@ -69,7 +71,7 @@ static bool Mf_init( struct Matchfinder * const mf,
mf->dictionary_size = dict_size; mf->dictionary_size = dict_size;
mf->pos_limit = mf->buffer_size - after_size; mf->pos_limit = mf->buffer_size - after_size;
size = 1 << max( 16, real_bits( mf->dictionary_size - 1 ) - 2 ); size = 1 << max( 16, real_bits( mf->dictionary_size - 1 ) - 2 );
if( mf->dictionary_size > 1 << 26 ) if( mf->dictionary_size > 1 << 26 ) /* 64 MiB */
size >>= 1; size >>= 1;
mf->key4_mask = size - 1; mf->key4_mask = size - 1;
size += num_prev_positions2; size += num_prev_positions2;
@ -77,7 +79,8 @@ static bool Mf_init( struct Matchfinder * const mf,
mf->num_prev_positions = size; mf->num_prev_positions = size;
size += ( 2 * ( mf->dictionary_size + 1 ) ); size += ( 2 * ( mf->dictionary_size + 1 ) );
mf->prev_positions = (int32_t *)malloc( size * sizeof (int32_t) ); if( size * sizeof (int32_t) <= size ) mf->prev_positions = 0;
else mf->prev_positions = (int32_t *)malloc( size * sizeof (int32_t) );
if( !mf->prev_positions ) { free( mf->buffer ); return false; } if( !mf->prev_positions ) { free( mf->buffer ); return false; }
mf->prev_pos_tree = mf->prev_positions + mf->num_prev_positions; mf->prev_pos_tree = mf->prev_positions + mf->num_prev_positions;
for( i = 0; i < mf->num_prev_positions; ++i ) mf->prev_positions[i] = -1; for( i = 0; i < mf->num_prev_positions; ++i ) mf->prev_positions[i] = -1;
@ -116,6 +119,7 @@ static void Mf_reset( struct Matchfinder * const mf )
mf->cyclic_pos = 0; mf->cyclic_pos = 0;
mf->at_stream_end = false; mf->at_stream_end = false;
mf->been_flushed = false; mf->been_flushed = false;
mf->flushing = false;
for( i = 0; i < mf->num_prev_positions; ++i ) mf->prev_positions[i] = -1; for( i = 0; i < mf->num_prev_positions; ++i ) mf->prev_positions[i] = -1;
} }
@ -264,18 +268,17 @@ static bool LZe_full_flush( struct LZ_encoder * const encoder, const State state
const int pos_state = Mf_data_position( encoder->matchfinder ) & pos_state_mask; const int pos_state = Mf_data_position( encoder->matchfinder ) & pos_state_mask;
File_trailer trailer; File_trailer trailer;
if( encoder->member_finished || if( encoder->member_finished ||
Cb_free_bytes( &encoder->range_encoder.cb ) < max_marker_size + Ft_size ) Cb_free_bytes( &encoder->renc.cb ) < max_marker_size + Ft_size )
return false; return false;
Re_encode_bit( &encoder->range_encoder, &encoder->bm_match[state][pos_state], 1 ); Re_encode_bit( &encoder->renc, &encoder->bm_match[state][pos_state], 1 );
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep[state], 0 ); Re_encode_bit( &encoder->renc, &encoder->bm_rep[state], 0 );
LZe_encode_pair( encoder, 0xFFFFFFFFU, min_match_len, pos_state ); LZe_encode_pair( encoder, 0xFFFFFFFFU, min_match_len, pos_state );
Re_flush( &encoder->range_encoder ); Re_flush( &encoder->renc );
Ft_set_data_crc( trailer, LZe_crc( encoder ) ); Ft_set_data_crc( trailer, LZe_crc( encoder ) );
Ft_set_data_size( trailer, Mf_data_position( encoder->matchfinder ) ); Ft_set_data_size( trailer, Mf_data_position( encoder->matchfinder ) );
Ft_set_member_size( trailer, Re_member_position( &encoder->range_encoder ) + Ft_set_member_size( trailer, Re_member_position( &encoder->renc ) + Ft_size );
Ft_size );
for( i = 0; i < Ft_size; ++i ) for( i = 0; i < Ft_size; ++i )
Cb_put_byte( &encoder->range_encoder.cb, trailer[i] ); Cb_put_byte( &encoder->renc.cb, trailer[i] );
return true; return true;
} }
@ -286,12 +289,12 @@ static bool LZe_sync_flush( struct LZ_encoder * const encoder )
const int pos_state = Mf_data_position( encoder->matchfinder ) & pos_state_mask; const int pos_state = Mf_data_position( encoder->matchfinder ) & pos_state_mask;
const State state = encoder->state; const State state = encoder->state;
if( encoder->member_finished || if( encoder->member_finished ||
Cb_free_bytes( &encoder->range_encoder.cb ) < max_marker_size ) Cb_free_bytes( &encoder->renc.cb ) < max_marker_size )
return false; return false;
Re_encode_bit( &encoder->range_encoder, &encoder->bm_match[state][pos_state], 1 ); Re_encode_bit( &encoder->renc, &encoder->bm_match[state][pos_state], 1 );
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep[state], 0 ); Re_encode_bit( &encoder->renc, &encoder->bm_rep[state], 0 );
LZe_encode_pair( encoder, 0xFFFFFFFFU, min_match_len + 1, pos_state ); LZe_encode_pair( encoder, 0xFFFFFFFFU, min_match_len + 1, pos_state );
Re_flush( &encoder->range_encoder ); Re_flush( &encoder->renc );
return true; return true;
} }
@ -363,7 +366,7 @@ static bool LZe_init( struct LZ_encoder * const encoder,
Bm_array_init( encoder->bm_align, dis_align_size ); Bm_array_init( encoder->bm_align, dis_align_size );
encoder->matchfinder = mf; encoder->matchfinder = mf;
if( !Re_init( &encoder->range_encoder ) ) return false; if( !Re_init( &encoder->renc ) ) return false;
Lee_init( &encoder->match_len_encoder, encoder->matchfinder->match_len_limit ); Lee_init( &encoder->match_len_encoder, encoder->matchfinder->match_len_limit );
Lee_init( &encoder->rep_len_encoder, encoder->matchfinder->match_len_limit ); Lee_init( &encoder->rep_len_encoder, encoder->matchfinder->match_len_limit );
encoder->num_dis_slots = encoder->num_dis_slots =
@ -376,13 +379,13 @@ static bool LZe_init( struct LZ_encoder * const encoder,
encoder->member_finished = false; encoder->member_finished = false;
for( i = 0; i < Fh_size; ++i ) for( i = 0; i < Fh_size; ++i )
Cb_put_byte( &encoder->range_encoder.cb, header[i] ); Cb_put_byte( &encoder->renc.cb, header[i] );
return true; return true;
} }
/* Return value == number of bytes advanced (ahead). /* Return value == number of bytes advanced (ahead).
trials[0]..trials[retval-1] contain the steps to encode. trials[0]..trials[ahead-1] contain the steps to encode.
( trials[0].dis == -1 && trials[0].price == 1 ) means literal. ( trials[0].dis == -1 && trials[0].price == 1 ) means literal.
*/ */
static int LZe_sequence_optimizer( struct LZ_encoder * const encoder, static int LZe_sequence_optimizer( struct LZ_encoder * const encoder,
@ -583,8 +586,7 @@ static int LZe_sequence_optimizer( struct LZ_encoder * const encoder,
if( St_is_char( cur_state ) ) if( St_is_char( cur_state ) )
next_price += LZe_price_literal( encoder, prev_byte, cur_byte ); next_price += LZe_price_literal( encoder, prev_byte, cur_byte );
else else
next_price += LZe_price_matched( encoder, next_price += LZe_price_matched( encoder, prev_byte, cur_byte, match_byte );
prev_byte, cur_byte, match_byte );
if( !Mf_move_pos( encoder->matchfinder ) ) return 0; if( !Mf_move_pos( encoder->matchfinder ) ) return 0;
/* try last updates to next trial */ /* try last updates to next trial */
@ -741,15 +743,15 @@ static int LZe_sequence_optimizer( struct LZ_encoder * const encoder,
} }
static bool LZe_encode_member( struct LZ_encoder * const encoder, static bool LZe_encode_member( struct LZ_encoder * const encoder )
const bool finish )
{ {
const int fill_count = const int fill_count =
( encoder->matchfinder->match_len_limit > 12 ) ? 128 : 512; ( encoder->matchfinder->match_len_limit > 12 ) ? 128 : 512;
int ahead, i; int ahead, i;
State * const state = &encoder->state; State * const state = &encoder->state;
if( encoder->member_finished ) return true; if( encoder->member_finished ) return true;
if( Re_member_position( &encoder->range_encoder ) >= encoder->member_size_limit ) if( Re_member_position( &encoder->renc ) >= encoder->member_size_limit )
{ {
if( LZe_full_flush( encoder, *state ) ) encoder->member_finished = true; if( LZe_full_flush( encoder, *state ) ) encoder->member_finished = true;
return true; return true;
@ -761,10 +763,10 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
const uint8_t prev_byte = 0; const uint8_t prev_byte = 0;
uint8_t cur_byte; uint8_t cur_byte;
if( Mf_available_bytes( encoder->matchfinder ) < max_match_len && if( Mf_available_bytes( encoder->matchfinder ) < max_match_len &&
!encoder->matchfinder->at_stream_end ) !Mf_flushing_or_end( encoder->matchfinder ) )
return true; return true;
cur_byte = Mf_peek( encoder->matchfinder, 0 ); cur_byte = Mf_peek( encoder->matchfinder, 0 );
Re_encode_bit( &encoder->range_encoder, &encoder->bm_match[*state][0], 0 ); Re_encode_bit( &encoder->renc, &encoder->bm_match[*state][0], 0 );
LZe_encode_literal( encoder, prev_byte, cur_byte ); LZe_encode_literal( encoder, prev_byte, cur_byte );
CRC32_update_byte( &encoder->crc, cur_byte ); CRC32_update_byte( &encoder->crc, cur_byte );
Mf_get_match_pairs( encoder->matchfinder, 0 ); Mf_get_match_pairs( encoder->matchfinder, 0 );
@ -774,7 +776,7 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
while( !Mf_finished( encoder->matchfinder ) ) while( !Mf_finished( encoder->matchfinder ) )
{ {
if( !Mf_enough_available_bytes( encoder->matchfinder ) || if( !Mf_enough_available_bytes( encoder->matchfinder ) ||
!Re_enough_free_bytes( &encoder->range_encoder ) ) return true; !Re_enough_free_bytes( &encoder->renc ) ) return true;
if( encoder->pending_num_pairs == 0 ) if( encoder->pending_num_pairs == 0 )
{ {
if( encoder->fill_counter <= 0 ) if( encoder->fill_counter <= 0 )
@ -794,7 +796,7 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
const int len = encoder->trials[i].price; const int len = encoder->trials[i].price;
bool bit = ( dis < 0 && len == 1 ); bool bit = ( dis < 0 && len == 1 );
Re_encode_bit( &encoder->range_encoder, Re_encode_bit( &encoder->renc,
&encoder->bm_match[*state][pos_state], !bit ); &encoder->bm_match[*state][pos_state], !bit );
if( bit ) /* literal byte */ if( bit ) /* literal byte */
{ {
@ -811,28 +813,28 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
} }
*state = St_set_char( *state ); *state = St_set_char( *state );
} }
else /* match or repeated match */ else /* match or repeated match */
{ {
CRC32_update_buf( &encoder->crc, Mf_ptr_to_current_pos( encoder->matchfinder ) - ahead, len ); CRC32_update_buf( &encoder->crc, Mf_ptr_to_current_pos( encoder->matchfinder ) - ahead, len );
LZe_mtf_reps( dis, encoder->rep_distances ); LZe_mtf_reps( dis, encoder->rep_distances );
bit = ( dis < num_rep_distances ); bit = ( dis < num_rep_distances );
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep[*state], bit ); Re_encode_bit( &encoder->renc, &encoder->bm_rep[*state], bit );
if( bit ) if( bit )
{ {
bit = ( dis == 0 ); bit = ( dis == 0 );
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep0[*state], !bit ); Re_encode_bit( &encoder->renc, &encoder->bm_rep0[*state], !bit );
if( bit ) if( bit )
Re_encode_bit( &encoder->range_encoder, &encoder->bm_len[*state][pos_state], len > 1 ); Re_encode_bit( &encoder->renc, &encoder->bm_len[*state][pos_state], len > 1 );
else else
{ {
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep1[*state], dis > 1 ); Re_encode_bit( &encoder->renc, &encoder->bm_rep1[*state], dis > 1 );
if( dis > 1 ) if( dis > 1 )
Re_encode_bit( &encoder->range_encoder, &encoder->bm_rep2[*state], dis > 2 ); Re_encode_bit( &encoder->renc, &encoder->bm_rep2[*state], dis > 2 );
} }
if( len == 1 ) *state = St_set_short_rep( *state ); if( len == 1 ) *state = St_set_short_rep( *state );
else else
{ {
Lee_encode( &encoder->rep_len_encoder, &encoder->range_encoder, len, pos_state ); Lee_encode( &encoder->rep_len_encoder, &encoder->renc, len, pos_state );
*state = St_set_rep( *state ); *state = St_set_rep( *state );
} }
} }
@ -844,7 +846,7 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
} }
} }
ahead -= len; i += len; ahead -= len; i += len;
if( Re_member_position( &encoder->range_encoder ) >= encoder->member_size_limit ) if( Re_member_position( &encoder->renc ) >= encoder->member_size_limit )
{ {
if( !Mf_dec_pos( encoder->matchfinder, ahead ) ) return false; if( !Mf_dec_pos( encoder->matchfinder, ahead ) ) return false;
if( LZe_full_flush( encoder, *state ) ) encoder->member_finished = true; if( LZe_full_flush( encoder, *state ) ) encoder->member_finished = true;
@ -853,7 +855,6 @@ static bool LZe_encode_member( struct LZ_encoder * const encoder,
if( ahead <= 0 ) break; if( ahead <= 0 ) break;
} }
} }
if( finish && LZe_full_flush( encoder, *state ) ) if( LZe_full_flush( encoder, *state ) ) encoder->member_finished = true;
encoder->member_finished = true;
return true; return true;
} }

48
encoder.h
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -239,10 +239,9 @@ struct Matchfinder
int num_prev_positions; /* size of prev_positions */ int num_prev_positions; /* size of prev_positions */
bool at_stream_end; /* stream_pos shows real end of file */ bool at_stream_end; /* stream_pos shows real end of file */
bool been_flushed; bool been_flushed;
bool flushing;
}; };
static bool Mf_normalize_pos( struct Matchfinder * const mf );
static int Mf_write_data( struct Matchfinder * const mf, static int Mf_write_data( struct Matchfinder * const mf,
const uint8_t * const inbuf, const int size ) const uint8_t * const inbuf, const int size )
{ {
@ -255,6 +254,8 @@ static int Mf_write_data( struct Matchfinder * const mf,
return sz; return sz;
} }
static bool Mf_normalize_pos( struct Matchfinder * const mf );
static inline void Mf_free( struct Matchfinder * const mf ) static inline void Mf_free( struct Matchfinder * const mf )
{ {
free( mf->prev_positions ); free( mf->prev_positions );
@ -271,16 +272,18 @@ static inline unsigned long long
Mf_data_position( const struct Matchfinder * const mf ) Mf_data_position( const struct Matchfinder * const mf )
{ return mf->partial_data_pos + mf->pos; } { return mf->partial_data_pos + mf->pos; }
static inline void Mf_finish( struct Matchfinder * const mf )
{ mf->at_stream_end = true; mf->flushing = false; }
static inline bool Mf_finished( const struct Matchfinder * const mf ) static inline bool Mf_finished( const struct Matchfinder * const mf )
{ return mf->at_stream_end && mf->pos >= mf->stream_pos; } { return mf->at_stream_end && mf->pos >= mf->stream_pos; }
static inline const uint8_t *
Mf_ptr_to_current_pos( const struct Matchfinder * const mf )
{ return mf->buffer + mf->pos; }
static inline void Mf_set_flushing( struct Matchfinder * const mf, static inline void Mf_set_flushing( struct Matchfinder * const mf,
const bool flushing ) const bool flushing )
{ mf->at_stream_end = flushing; } { mf->flushing = flushing; }
static inline bool Mf_flushing_or_end( const struct Matchfinder * const mf )
{ return mf->at_stream_end || mf->flushing; }
static inline int Mf_free_bytes( const struct Matchfinder * const mf ) static inline int Mf_free_bytes( const struct Matchfinder * const mf )
{ if( mf->at_stream_end ) return 0; return mf->buffer_size - mf->stream_pos; } { if( mf->at_stream_end ) return 0; return mf->buffer_size - mf->stream_pos; }
@ -288,9 +291,13 @@ static inline int Mf_free_bytes( const struct Matchfinder * const mf )
static inline bool Mf_enough_available_bytes( const struct Matchfinder * const mf ) static inline bool Mf_enough_available_bytes( const struct Matchfinder * const mf )
{ {
return ( mf->pos + after_size <= mf->stream_pos || return ( mf->pos + after_size <= mf->stream_pos ||
( mf->at_stream_end && mf->pos < mf->stream_pos ) ); ( Mf_flushing_or_end( mf ) && mf->pos < mf->stream_pos ) );
} }
static inline const uint8_t *
Mf_ptr_to_current_pos( const struct Matchfinder * const mf )
{ return mf->buffer + mf->pos; }
static inline bool Mf_dec_pos( struct Matchfinder * const mf, static inline bool Mf_dec_pos( struct Matchfinder * const mf,
const int ahead ) const int ahead )
{ {
@ -321,7 +328,6 @@ static inline bool Mf_move_pos( struct Matchfinder * const mf )
return true; return true;
} }
static void Mf_reset( struct Matchfinder * const mf );
static int Mf_get_match_pairs( struct Matchfinder * const mf, struct Pair * pairs ); static int Mf_get_match_pairs( struct Matchfinder * const mf, struct Pair * pairs );
@ -590,7 +596,7 @@ struct LZ_encoder
Bit_model bm_align[dis_align_size]; Bit_model bm_align[dis_align_size];
struct Matchfinder * matchfinder; struct Matchfinder * matchfinder;
struct Range_encoder range_encoder; struct Range_encoder renc;
struct Len_encoder match_len_encoder; struct Len_encoder match_len_encoder;
struct Len_encoder rep_len_encoder; struct Len_encoder rep_len_encoder;
@ -611,13 +617,12 @@ struct LZ_encoder
static inline bool LZe_member_finished( const struct LZ_encoder * const encoder ) static inline bool LZe_member_finished( const struct LZ_encoder * const encoder )
{ {
return ( encoder->member_finished && return ( encoder->member_finished && !Cb_used_bytes( &encoder->renc.cb ) );
!Cb_used_bytes( &encoder->range_encoder.cb ) );
} }
static inline void LZe_free( struct LZ_encoder * const encoder ) static inline void LZe_free( struct LZ_encoder * const encoder )
{ Re_free( &encoder->range_encoder ); } { Re_free( &encoder->renc ); }
static inline unsigned LZe_crc( const struct LZ_encoder * const encoder ) static inline unsigned LZe_crc( const struct LZ_encoder * const encoder )
{ return encoder->crc ^ 0xFFFFFFFFU; } { return encoder->crc ^ 0xFFFFFFFFU; }
@ -702,13 +707,13 @@ static inline int LZe_price_matched( const struct LZ_encoder * const encoder,
static inline void LZe_encode_literal( struct LZ_encoder * const encoder, static inline void LZe_encode_literal( struct LZ_encoder * const encoder,
uint8_t prev_byte, uint8_t symbol ) uint8_t prev_byte, uint8_t symbol )
{ Re_encode_tree( &encoder->range_encoder, { Re_encode_tree( &encoder->renc,
encoder->bm_literal[get_lit_state(prev_byte)], symbol, 8 ); } encoder->bm_literal[get_lit_state(prev_byte)], symbol, 8 ); }
static inline void LZe_encode_matched( struct LZ_encoder * const encoder, static inline void LZe_encode_matched( struct LZ_encoder * const encoder,
uint8_t prev_byte, uint8_t symbol, uint8_t prev_byte, uint8_t symbol,
uint8_t match_byte ) uint8_t match_byte )
{ Re_encode_matched( &encoder->range_encoder, { Re_encode_matched( &encoder->renc,
encoder->bm_literal[get_lit_state(prev_byte)], encoder->bm_literal[get_lit_state(prev_byte)],
symbol, match_byte ); } symbol, match_byte ); }
@ -717,9 +722,8 @@ static inline void LZe_encode_pair( struct LZ_encoder * const encoder,
const int pos_state ) const int pos_state )
{ {
const int dis_slot = get_slot( dis ); const int dis_slot = get_slot( dis );
Lee_encode( &encoder->match_len_encoder, &encoder->range_encoder, len, pos_state ); Lee_encode( &encoder->match_len_encoder, &encoder->renc, len, pos_state );
Re_encode_tree( &encoder->range_encoder, Re_encode_tree( &encoder->renc, encoder->bm_dis_slot[get_dis_state(len)],
encoder->bm_dis_slot[get_dis_state(len)],
dis_slot, dis_slot_bits ); dis_slot, dis_slot_bits );
if( dis_slot >= start_dis_model ) if( dis_slot >= start_dis_model )
@ -729,14 +733,14 @@ static inline void LZe_encode_pair( struct LZ_encoder * const encoder,
const uint32_t direct_dis = dis - base; const uint32_t direct_dis = dis - base;
if( dis_slot < end_dis_model ) if( dis_slot < end_dis_model )
Re_encode_tree_reversed( &encoder->range_encoder, Re_encode_tree_reversed( &encoder->renc,
encoder->bm_dis + base - dis_slot - 1, encoder->bm_dis + base - dis_slot - 1,
direct_dis, direct_bits ); direct_dis, direct_bits );
else else
{ {
Re_encode( &encoder->range_encoder, direct_dis >> dis_align_bits, Re_encode( &encoder->renc, direct_dis >> dis_align_bits,
direct_bits - dis_align_bits ); direct_bits - dis_align_bits );
Re_encode_tree_reversed( &encoder->range_encoder, encoder->bm_align, Re_encode_tree_reversed( &encoder->renc, encoder->bm_align,
direct_dis, dis_align_bits ); direct_dis, dis_align_bits );
--encoder->align_price_count; --encoder->align_price_count;
} }

4
lzcheck.c
View file

@ -1,4 +1,4 @@
/* Lzcheck - A test program for the lzlib library /* Lzcheck - Test program for the lzlib library
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you have unlimited permission This program is free software: you have unlimited permission
@ -37,7 +37,7 @@ uint8_t out_buffer[buffer_size];
int main( const int argc, const char * const argv[] ) int main( const int argc, const char * const argv[] )
{ {
const int dictionary_size = 1 << 20; const int dictionary_size = 1 << 20;
const int match_len_limit = 36; const int match_len_limit = 16;
const unsigned long long member_size = INT64_MAX; const unsigned long long member_size = INT64_MAX;
struct LZ_Encoder * encoder; struct LZ_Encoder * encoder;
struct LZ_Decoder * decoder; struct LZ_Decoder * decoder;

13
lzip.h
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -208,7 +208,7 @@ static inline uint8_t Fh_version( const File_header data )
{ return data[4]; } { return data[4]; }
static inline bool Fh_verify_version( const File_header data ) static inline bool Fh_verify_version( const File_header data )
{ return ( data[4] <= 1 ); } { return ( data[4] == 1 ); }
static inline unsigned Fh_get_dictionary_size( const File_header data ) static inline unsigned Fh_get_dictionary_size( const File_header data )
{ {
@ -218,15 +218,15 @@ static inline unsigned Fh_get_dictionary_size( const File_header data )
return sz; return sz;
} }
static inline bool Fh_set_dictionary_size( File_header data, const int sz ) static inline bool Fh_set_dictionary_size( File_header data, const unsigned sz )
{ {
if( sz >= min_dictionary_size && sz <= max_dictionary_size ) if( sz >= min_dictionary_size && sz <= max_dictionary_size )
{ {
data[5] = real_bits( sz - 1 ); data[5] = real_bits( sz - 1 );
if( sz > min_dictionary_size ) if( sz > min_dictionary_size )
{ {
const int base_size = 1 << data[5]; const unsigned base_size = 1 << data[5];
const int wedge = base_size / 16; const unsigned wedge = base_size / 16;
int i; int i;
for( i = 7; i >= 1; --i ) for( i = 7; i >= 1; --i )
if( base_size - ( i * wedge ) >= sz ) if( base_size - ( i * wedge ) >= sz )
@ -252,9 +252,6 @@ typedef uint8_t File_trailer[20];
enum { Ft_size = 20 }; enum { Ft_size = 20 };
static inline int Ft_versioned_size( const int version )
{ return ( ( version >= 1 ) ? 20 : 12 ); }
static inline unsigned Ft_get_data_crc( const File_trailer data ) static inline unsigned Ft_get_data_crc( const File_trailer data )
{ {
unsigned tmp = 0; unsigned tmp = 0;

34
lzlib.c
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -199,7 +199,7 @@ int LZ_compress_close( struct LZ_Encoder * const e )
int LZ_compress_finish( struct LZ_Encoder * const e ) int LZ_compress_finish( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) || e->fatal ) return -1; if( !verify_encoder( e ) || e->fatal ) return -1;
Mf_set_flushing( e->matchfinder, true ); Mf_finish( e->matchfinder );
e->flush_pending = 0; e->flush_pending = 0;
/* if (open --> write --> finish) use same dictionary size as lzip. */ /* if (open --> write --> finish) use same dictionary size as lzip. */
/* this does not save any memory. */ /* this does not save any memory. */
@ -220,16 +220,13 @@ int LZ_compress_restart_member( struct LZ_Encoder * const e,
{ e->lz_errno = LZ_bad_argument; return -1; } { e->lz_errno = LZ_bad_argument; return -1; }
e->partial_in_size += Mf_data_position( e->matchfinder ); e->partial_in_size += Mf_data_position( e->matchfinder );
e->partial_out_size += Re_member_position( &e->lz_encoder->range_encoder ); e->partial_out_size += Re_member_position( &e->lz_encoder->renc );
Mf_reset( e->matchfinder ); Mf_reset( e->matchfinder );
LZe_free( e->lz_encoder ); free( e->lz_encoder ); LZe_free( e->lz_encoder );
e->lz_encoder = (struct LZ_encoder *)malloc( sizeof (struct LZ_encoder) ); if( !LZe_init( e->lz_encoder, e->matchfinder, e->member_header, member_size ) )
if( !e->lz_encoder ||
!LZe_init( e->lz_encoder, e->matchfinder, e->member_header, member_size ) )
{ {
if( e->lz_encoder ) LZe_free( e->lz_encoder ); free( e->lz_encoder ); e->lz_encoder = 0;
{ LZe_free( e->lz_encoder ); free( e->lz_encoder ); e->lz_encoder = 0; }
e->lz_errno = LZ_mem_error; e->fatal = true; e->lz_errno = LZ_mem_error; e->fatal = true;
return -1; return -1;
} }
@ -241,11 +238,11 @@ int LZ_compress_restart_member( struct LZ_Encoder * const e,
int LZ_compress_sync_flush( struct LZ_Encoder * const e ) int LZ_compress_sync_flush( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) || e->fatal ) return -1; if( !verify_encoder( e ) || e->fatal ) return -1;
if( !e->flush_pending && !e->matchfinder->at_stream_end ) if( e->flush_pending <= 0 && !Mf_flushing_or_end( e->matchfinder ) )
{ {
e->flush_pending = 2; /* 2 consecutive markers guarantee decoding */ e->flush_pending = 2; /* 2 consecutive markers guarantee decoding */
Mf_set_flushing( e->matchfinder, true ); Mf_set_flushing( e->matchfinder, true );
if( !LZe_encode_member( e->lz_encoder, false ) ) if( !LZe_encode_member( e->lz_encoder ) )
{ e->lz_errno = LZ_library_error; e->fatal = true; return -1; } { e->lz_errno = LZ_library_error; e->fatal = true; return -1; }
while( e->flush_pending > 0 && LZe_sync_flush( e->lz_encoder ) ) while( e->flush_pending > 0 && LZe_sync_flush( e->lz_encoder ) )
{ if( --e->flush_pending <= 0 ) Mf_set_flushing( e->matchfinder, false ); } { if( --e->flush_pending <= 0 ) Mf_set_flushing( e->matchfinder, false ); }
@ -258,11 +255,11 @@ int LZ_compress_read( struct LZ_Encoder * const e,
uint8_t * const buffer, const int size ) uint8_t * const buffer, const int size )
{ {
if( !verify_encoder( e ) || e->fatal ) return -1; if( !verify_encoder( e ) || e->fatal ) return -1;
if( !LZe_encode_member( e->lz_encoder, !e->flush_pending ) ) if( !LZe_encode_member( e->lz_encoder ) )
{ e->lz_errno = LZ_library_error; e->fatal = true; return -1; } { e->lz_errno = LZ_library_error; e->fatal = true; return -1; }
while( e->flush_pending > 0 && LZe_sync_flush( e->lz_encoder ) ) while( e->flush_pending > 0 && LZe_sync_flush( e->lz_encoder ) )
{ if( --e->flush_pending <= 0 ) Mf_set_flushing( e->matchfinder, false ); } { if( --e->flush_pending <= 0 ) Mf_set_flushing( e->matchfinder, false ); }
return Re_read_data( &e->lz_encoder->range_encoder, buffer, size ); return Re_read_data( &e->lz_encoder->renc, buffer, size );
} }
@ -270,7 +267,7 @@ int LZ_compress_write( struct LZ_Encoder * const e,
const uint8_t * const buffer, const int size ) const uint8_t * const buffer, const int size )
{ {
if( !verify_encoder( e ) || e->fatal ) return -1; if( !verify_encoder( e ) || e->fatal ) return -1;
if( e->flush_pending ) return 0; if( e->flush_pending > 0 || size < 0 ) return 0;
return Mf_write_data( e->matchfinder, buffer, size ); return Mf_write_data( e->matchfinder, buffer, size );
} }
@ -278,7 +275,7 @@ int LZ_compress_write( struct LZ_Encoder * const e,
int LZ_compress_write_size( struct LZ_Encoder * const e ) int LZ_compress_write_size( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) || e->fatal ) return -1; if( !verify_encoder( e ) || e->fatal ) return -1;
if( e->flush_pending ) return 0; if( e->flush_pending > 0 ) return 0;
return Mf_free_bytes( e->matchfinder ); return Mf_free_bytes( e->matchfinder );
} }
@ -293,7 +290,7 @@ enum LZ_Errno LZ_compress_errno( struct LZ_Encoder * const e )
int LZ_compress_finished( struct LZ_Encoder * const e ) int LZ_compress_finished( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) ) return -1; if( !verify_encoder( e ) ) return -1;
return ( !e->flush_pending && Mf_finished( e->matchfinder ) && return ( e->flush_pending <= 0 && Mf_finished( e->matchfinder ) &&
LZe_member_finished( e->lz_encoder ) ); LZe_member_finished( e->lz_encoder ) );
} }
@ -315,7 +312,7 @@ unsigned long long LZ_compress_data_position( struct LZ_Encoder * const e )
unsigned long long LZ_compress_member_position( struct LZ_Encoder * const e ) unsigned long long LZ_compress_member_position( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) ) return 0; if( !verify_encoder( e ) ) return 0;
return Re_member_position( &e->lz_encoder->range_encoder ); return Re_member_position( &e->lz_encoder->renc );
} }
@ -329,8 +326,7 @@ unsigned long long LZ_compress_total_in_size( struct LZ_Encoder * const e )
unsigned long long LZ_compress_total_out_size( struct LZ_Encoder * const e ) unsigned long long LZ_compress_total_out_size( struct LZ_Encoder * const e )
{ {
if( !verify_encoder( e ) ) return 0; if( !verify_encoder( e ) ) return 0;
return e->partial_out_size + return e->partial_out_size + Re_member_position( &e->lz_encoder->renc );
Re_member_position( &e->lz_encoder->range_encoder );
} }

4
lzlib.h
View file

@ -1,4 +1,4 @@
/* Lzlib - A compression library for lzip files /* Lzlib - Compression library for lzip files
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This library is free software: you can redistribute it and/or modify This library is free software: you can redistribute it and/or modify
@ -29,7 +29,7 @@
extern "C" { extern "C" {
#endif #endif
static const char * const LZ_version_string = "1.4"; static const char * const LZ_version_string = "1.5-rc1";
enum LZ_Errno { LZ_ok = 0, LZ_bad_argument, LZ_mem_error, enum LZ_Errno { LZ_ok = 0, LZ_bad_argument, LZ_mem_error,
LZ_sequence_error, LZ_header_error, LZ_unexpected_eof, LZ_sequence_error, LZ_header_error, LZ_unexpected_eof,

21
main.c
View file

@ -1,4 +1,4 @@
/* Minilzip - A test program for the lzlib library /* Minilzip - Test program for the lzlib library
Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz.
This program is free software: you can redistribute it and/or modify This program is free software: you can redistribute it and/or modify
@ -147,7 +147,7 @@ static void Pp_show_msg( struct Pretty_print * const pp, const char * const msg
static void show_help( void ) static void show_help( void )
{ {
printf( "%s - A test program for the lzlib library.\n", Program_name ); printf( "%s - Test program for the lzlib library.\n", Program_name );
printf( "\nUsage: %s [options] [files]\n", invocation_name ); printf( "\nUsage: %s [options] [files]\n", invocation_name );
printf( "\nOptions:\n" printf( "\nOptions:\n"
" -h, --help display this help and exit\n" " -h, --help display this help and exit\n"
@ -196,7 +196,7 @@ static void show_version( void )
} }
void show_header( struct LZ_Decoder * const decoder ) static void show_header( struct LZ_Decoder * const decoder )
{ {
const char * const prefix[8] = const char * const prefix[8] =
{ "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" }; { "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" };
@ -209,8 +209,6 @@ void show_header( struct LZ_Decoder * const decoder )
for( i = 0; i < 8 && ( num > 9999 || ( exact && num >= factor ) ); ++i ) for( i = 0; i < 8 && ( num > 9999 || ( exact && num >= factor ) ); ++i )
{ num /= factor; if( num % factor != 0 ) exact = false; { num /= factor; if( num % factor != 0 ) exact = false;
p = prefix[i]; np = ""; } p = prefix[i]; np = ""; }
if( verbosity >= 4 )
fprintf( stderr, "version %d, ", LZ_decompress_member_version( decoder ) );
fprintf( stderr, "dictionary size %s%4u %sB. ", np, num, p ); fprintf( stderr, "dictionary size %s%4u %sB. ", np, num, p );
} }
@ -323,12 +321,13 @@ static int open_instream( const char * const name, struct stat * const in_statsp
const bool can_read = ( i == 0 && const bool can_read = ( i == 0 &&
( S_ISBLK( mode ) || S_ISCHR( mode ) || ( S_ISBLK( mode ) || S_ISCHR( mode ) ||
S_ISFIFO( mode ) || S_ISSOCK( mode ) ) ); S_ISFIFO( mode ) || S_ISSOCK( mode ) ) );
if( i != 0 || ( !S_ISREG( mode ) && ( !to_stdout || !can_read ) ) ) const bool no_ofile = to_stdout || program_mode == m_test;
if( i != 0 || ( !S_ISREG( mode ) && ( !can_read || !no_ofile ) ) )
{ {
if( verbosity >= 0 ) if( verbosity >= 0 )
fprintf( stderr, "%s: Input file '%s' is not a regular file%s.\n", fprintf( stderr, "%s: Input file '%s' is not a regular file%s.\n",
program_name, name, program_name, name,
( can_read && !to_stdout ) ? ( can_read && !no_ofile ) ?
" and '--stdout' was not specified" : "" ); " and '--stdout' was not specified" : "" );
close( infd ); close( infd );
infd = -1; infd = -1;
@ -468,7 +467,7 @@ static void close_and_set_permissions( const struct stat * const in_statsp )
/* Returns the number of bytes really read. /* Returns the number of bytes really read.
If (returned value < size) and (errno == 0), means EOF was reached. If (returned value < size) and (errno == 0), means EOF was reached.
*/ */
int readblock( const int fd, uint8_t * const buf, const int size ) static int readblock( const int fd, uint8_t * const buf, const int size )
{ {
int rest = size; int rest = size;
errno = 0; errno = 0;
@ -487,7 +486,7 @@ int readblock( const int fd, uint8_t * const buf, const int size )
/* Returns the number of bytes really written. /* Returns the number of bytes really written.
If (returned value < size), it is always an error. If (returned value < size), it is always an error.
*/ */
int writeblock( const int fd, const uint8_t * const buf, const int size ) static int writeblock( const int fd, const uint8_t * const buf, const int size )
{ {
int rest = size; int rest = size;
errno = 0; errno = 0;
@ -714,8 +713,8 @@ int do_decompress( struct LZ_Decoder * const decoder, const int infd,
if( lz_errno == LZ_header_error || ( first_member && out_size == 0 ) ) if( lz_errno == LZ_header_error || ( first_member && out_size == 0 ) )
{ {
if( !first_member ) break; /* trailing garbage */ if( !first_member ) break; /* trailing garbage */
Pp_show_msg( pp, "Error reading member header" ); Pp_show_msg( pp, "Bad magic number (file not in lzip format)" );
return 1; return 2;
} }
if( lz_errno == LZ_mem_error ) if( lz_errno == LZ_mem_error )
{ {

40
testsuite/check.sh
View file

@ -24,21 +24,34 @@ mkdir tmp
cd "${objdir}"/tmp cd "${objdir}"/tmp
cat "${testdir}"/test.txt > in || framework_failure cat "${testdir}"/test.txt > in || framework_failure
in_lz="${testdir}"/test.txt.lz
fail=0 fail=0
printf "testing lzlib-%s..." "$2" printf "testing lzlib-%s..." "$2"
"${LZIP}" -cqs-1 in > /dev/null "${LZIP}" -cqs-1 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs0 in > /dev/null "${LZIP}" -cqs0 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqs4095 in > /dev/null "${LZIP}" -cqs4095 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cqm274 in > /dev/null "${LZIP}" -cqm274 in > /dev/null
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -tq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cdq < in
if [ $? = 2 ] ; then printf . ; else fail=1 ; printf - ; fi
dd if="${in_lz}" bs=1 count=6 2> /dev/null | "${LZIP}" -tq
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
dd if="${in_lz}" bs=1 count=20 2> /dev/null | "${LZIP}" -tq
if [ $? = 2 ] ; then printf . ; else printf - ; fail=1 ; fi
"${LZIP}" -t "${testdir}"/test.txt.lz || fail=1 "${LZIP}" -t "${in_lz}" || fail=1
"${LZIP}" -cd "${testdir}"/test.txt.lz > copy || fail=1 "${LZIP}" -cd "${in_lz}" > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf . printf .
@ -47,9 +60,9 @@ printf .
cmp in copy || fail=1 cmp in copy || fail=1
printf . printf .
"${LZIP}" -cfq "${testdir}"/test.txt.lz > out "${LZIP}" -cfq "${in_lz}" > out
if [ $? != 1 ] ; then fail=1 ; printf - ; else printf . ; fi if [ $? = 1 ] ; then printf . ; else fail=1 ; printf - ; fi
"${LZIP}" -cF "${testdir}"/test.txt.lz > out || fail=1 "${LZIP}" -cF "${in_lz}" > out || fail=1
"${LZIP}" -cd out | "${LZIP}" -d > copy || fail=1 "${LZIP}" -cd out | "${LZIP}" -d > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf . printf .
@ -60,30 +73,30 @@ for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do
printf "garbage" >> copy.lz || fail=1 printf "garbage" >> copy.lz || fail=1
"${LZIP}" -df copy.lz || fail=1 "${LZIP}" -df copy.lz || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do
"${LZIP}" -c -$i in > out || fail=1 "${LZIP}" -c -$i in > out || fail=1
printf "g" >> out || fail=1 printf "g" >> out || fail=1
"${LZIP}" -cd out > copy || fail=1 "${LZIP}" -cd out > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do
"${LZIP}" -$i < in > out || fail=1 "${LZIP}" -$i < in > out || fail=1
"${LZIP}" -d < out > copy || fail=1 "${LZIP}" -d < out > copy || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do for i in s4Ki 0 1 2 3 4 5 6 7 8s16 9s16 ; do
"${LZIP}" -f -$i -o out < in || fail=1 "${LZIP}" -f -$i -o out < in || fail=1
"${LZIP}" -df -o copy < out.lz || fail=1 "${LZIP}" -df -o copy < out.lz || fail=1
cmp in copy || fail=1 cmp in copy || fail=1
printf .
done done
printf .
"${LZIP}" < in > anyothername || fail=1 "${LZIP}" < in > anyothername || fail=1
"${LZIP}" -d anyothername || fail=1 "${LZIP}" -d anyothername || fail=1
@ -97,7 +110,6 @@ cmp in2 copy2 || fail=1
printf . printf .
"${BBEXAMPLE}" in || fail=1 "${BBEXAMPLE}" in || fail=1
printf .
"${BBEXAMPLE}" out || fail=1 "${BBEXAMPLE}" out || fail=1
printf . printf .