lzip/lziprecover
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.18.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-21 11:26:24 +01:00
parent 1327a2b8ad
commit 734bd31e8b
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
45 changed files with 1576 additions and 774 deletions
Download patch file Download diff file Expand all files Collapse all files

212
doc/lziprecover.texi
View file

@ -6,8 +6,8 @@
@finalout
@c %**end of header
@set UPDATED 16 September 2015
@set VERSION 1.18-pre2
@set UPDATED 12 May 2016
@set VERSION 1.18
@dircategory Data Compression
@direntry
@ -50,7 +50,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}).
@end menu
@sp 1
Copyright @copyright{} 2009-2015 Antonio Diaz Diaz.
Copyright @copyright{} 2009-2016 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission
to copy, distribute and modify it.
@ -66,6 +66,9 @@ files, produce a correct file by merging the good parts of two or more
damaged copies, extract data from damaged files, decompress files and
test integrity of files.
Lziprecover is not a replacement for regular backups, but a last line of
defense for the case where the backups are also damaged.
The lzip file format is designed for data sharing and long-term
archiving, taking into account both data integrity and decoder
availability:
@ -96,6 +99,10 @@ repair the nearer it is from the beginning of the file. Therefore, with
the help of lziprecover, losing an entire archive just because of a
corrupt byte near the beginning is a thing of the past.
For compressible data, multiple lzip-compressed copies have a better
chance of surviving intact than one uncompressed copy using the same
amount of storage space.
Lziprecover is able to recover or decompress files produced by any of
the compressors in the lzip family; lzip, plzip, minilzip/lzlib, clzip
and pdlzip.
@ -113,12 +120,11 @@ garbage data may be produced at the end of each member):
lziprecover -D0 -i -o file -q file.lz
@end example
Lziprecover is able to efficiently extract a range of bytes from a
multi-member file, because it only decompresses the members containing
the desired data.
Lziprecover provides random access to the data in multimember files; it
only decompresses the members containing the desired data.
Lziprecover can print correct total file sizes and ratios even for
multi-member files.
multimember files.
When recovering data, lziprecover takes as arguments the names of the
damaged files and writes zero or more recovered files depending on the
@ -128,9 +134,6 @@ damaged files themselves are never modified.
When decompressing or testing file integrity, lziprecover behaves like
lzip or lunzip.
Lziprecover is not a replacement for regular backups, but a last line of
defense for the case where the backups are also damaged.
@node Invoking lziprecover
@chapter Invoking lziprecover
@ -142,6 +145,11 @@ The format for running lziprecover is:
lziprecover [@var{options}] [@var{files}]
@end example
@noindent
When decompressing or testing, @samp{-} used as a @var{file} argument
means standard input. It can be mixed with other @var{files} and is read
just once, the first time it appears in the command line.
Lziprecover supports the following options:
@table @code
@ -160,31 +168,54 @@ Exit with error status 2 if any remaining input is detected after
decompressing the last member. Such remaining input is usually trailing
garbage that can be safely ignored. @xref{concat-example}.
@item -A
@itemx --alone-to-lz
Convert lzma-alone files to lzip format without recompressing, just
adding a lzip header and trailer. The conversion minimizes the
dictionary size of the resulting file (and therefore the amount of
memory required to decompress it). Only streamed files with default LZMA
properties can be converted; non-streamed lzma-alone files lack the end
of stream marker required in lzip files.
The name of the converted lzip file is derived from that of the original
lzma-alone file as follows:
@multitable {filename.lzma} {becomes} {anyothername.lz}
@item filename.lzma @tab becomes @tab filename.lz
@item filename.tlz @tab becomes @tab filename.tar.lz
@item anyothername @tab becomes @tab anyothername.lz
@end multitable
@item -c
@itemx --stdout
Decompress to standard output. Needed when reading from a named pipe
(fifo) or from a device. Use it to recover as much of the uncompressed
data as possible when decompressing a corrupt file.
Write decompressed data to standard output; keep input files unchanged.
This option is needed when reading from a named pipe (fifo) or from a
device. Use it also to recover as much of the uncompressed data as
possible when decompressing a corrupt file.
@item -d
@itemx --decompress
Decompress the specified file(s). If a file fails to decompress,
lziprecover exits immediately without decompressing the rest of the
files.
Decompress the specified file(s). If a file does not exist or can't be
opened, lziprecover continues decompressing the rest of the files. If a
file fails to decompress, lziprecover exits immediately without
decompressing the rest of the files.
@item -D @var{range}
@itemx --range-decompress=@var{range}
Decompress only a range of bytes starting at decompressed byte position
@samp{@var{begin}} and up to byte position @w{@samp{@var{end} - 1}}.
Three formats of @var{range} are recognized, @samp{@var{begin}},
@samp{@var{begin}-@var{end}}, and @samp{@var{begin},@var{size}}. If only
@var{begin} is specified, @var{end} is taken as the end of the file. The
produced bytes are sent to standard output unless the @samp{--output}
option is used. In order to guarantee the correctness of the data
produced, all members containing any part of the desired data are
decompressed and their integrity is verified. This operation is more
efficient in multi-member files because it only decompresses the members
containing the desired data.
This option provides random access to the data in multimember files; it
only decompresses the members containing the desired data. In order to
guarantee the correctness of the data produced, all members containing
any part of the desired data are decompressed and their integrity is
verified.
Four formats of @var{range} are recognized, @samp{@var{begin}},
@samp{@var{begin}-@var{end}}, @samp{@var{begin},@var{size}}, and
@samp{,@var{size}}. If only @var{begin} is specified, @var{end} is taken
as the end of the file. If only @var{size} is specified, @var{begin} is
taken as the beginning of the file. The produced bytes are sent to
standard output unless the @samp{--output} option is used.
@item -f
@itemx --force
@ -194,7 +225,7 @@ Force overwrite of output files.
@itemx --ignore-errors
Make @samp{--range-decompress} ignore data errors and continue
decompressing the remaining members in the file. For example,
@w{@samp{lziprecover -i -D0 file.lz > file}} decompresses all the
@w{@samp{lziprecover -D0 -i file.lz > file}} decompresses all the
recoverable data in all members of @samp{file.lz} without having to
split it first.
@ -205,8 +236,8 @@ Keep (don't delete) input files during decompression.
@item -l
@itemx --list
Print total file sizes and ratios. The values produced are correct even
for multi-member files. Use it together with @samp{-v} to see
information about the members in the file.
for multimember files. Use it together with @samp{-v} to see information
about the members in the file.
@item -m
@itemx --merge
@ -223,7 +254,11 @@ Place the output into @samp{@var{file}} instead of into
produced are in the form @samp{rec01@var{file}}, @samp{rec02@var{file}},
etc. If decompressing from standard input and @samp{--stdout} has not
been specified, use @samp{@var{file}} as the name of the decompressed
file.
file. If converting a lzma-alone file from standard input and
@samp{--stdout} has not been specified, use @samp{@var{file}.lz} as the
name of the converted file. (Or plain @samp{@var{file}} if it already
ends in @samp{.lz} or @samp{.tlz}).
@item -q
@itemx --quiet
@ -299,15 +334,15 @@ caused lziprecover to panic.
@cindex data safety
There are 3 main types of data corruption that may cause data loss:
single-byte errors, multi-byte errors (generally affecting a whole
sector in a block device), and total device failure.
single-byte errors, multibyte errors (generally affecting a whole sector
in a block device), and total device failure.
Lziprecover protects natively against single-byte errors
(@pxref{Repairing files}), as long as file integrity is checked
frequently enough that a second single-byte error does not develop in
the same member before the first one is repaired.
Lziprecover also protects against multi-byte errors (@pxref{Merging
Lziprecover also protects against multibyte errors (@pxref{Merging
files}), if at least one backup copy of the file is made.
The only remedy for total device failure is storing backup copies in
@ -326,11 +361,11 @@ If you used bzip2, and if the file is large enough to contain more than
one compressed data block (usually larger than 900 kB uncompressed), and
if no block is damaged in both files, then the data can be manually
recovered by splitting the files with bzip2recover, verifying every
block and then copying the right blocks in the right order in another
block and then copying the right blocks in the right order into another
file.
But if you used lzip, the data can be automatically recovered as long as
no byte is damaged in both files.
the damaged areas don't overlap.
Note that each error in a bzip2 file makes a whole block unusable, but
each error in a lzip file only affects the damaged bytes, making it
@ -347,8 +382,8 @@ at all. If the reparation is successful, the repaired file will be
identical bit for bit to the original. This makes lzip files resistant
to bit-flip, one of the most common forms of data corruption.
The error may be located anywhere in the file except in the header
(first 6 bytes of each member) or in the @samp{Member size} field of the
The error may be located anywhere in the file except in the first 5
bytes of each member header or in the @samp{Member size} field of the
trailer (last 8 bytes of each member). If the error is in the header it
can be easily repaired with a text editor like GNU Moe (@pxref{File
format}). If the error is in the member size, it is enough to ignore the
@ -391,21 +426,14 @@ The merge will fail if the damaged areas overlap (at least one byte is
damaged in all copies), or are adjacent and the boundary can't be
determined, or if the copies have too many damaged areas.
All the copies must have the same size. If some of them have been
truncated and are therefore smaller than they should, they can be
extended to the correct size with the following command before merging
them with the other copies:
All the copies must have the same size. If any of them is larger or
smaller than it should, either because it has been truncated or because
it got some garbage data appended at the end, it can be brought to the
correct size with the following command before merging it with the other
copies:
@example
ddrescue --extend-outfile=<correct_size> small_file.lz extended_file.lz
@end example
If some of the copies have got garbage data at the end and are therefore
larger than they should, their sizes can be reduced to the correct value
with the following command before merging them with the other copies:
@example
ddrescue --size=<correct_size> large_file.lz reduced_file.lz
ddrescue -s<correct_size> -x<correct_size> file.lz correct_size_file.lz
@end example
To give you an idea of its possibilities, when merging two copies, each
@ -461,7 +489,7 @@ additional information before, between, or after them.
Each member has the following structure:
@verbatim
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size |
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
@end verbatim
@ -485,8 +513,8 @@ from the base size to obtain the dictionary size.@*
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
Valid values for dictionary size range from 4 KiB to 512 MiB.
@item Lzma stream
The lzma stream, finished by an end of stream marker. Uses default
@item LZMA stream
The LZMA stream, finished by an end of stream marker. Uses default
values for encoder properties.
@ifnothtml
@xref{Stream format,,,lzip},
@ -506,7 +534,7 @@ Size of the uncompressed original data.
@item Member size (8 bytes)
Total size of the member, including header and trailer. This field acts
as a distributed index, allows the verification of stream integrity, and
facilitates safe recovery of undamaged members from multi-member files.
facilitates safe recovery of undamaged members from multimember files.
@end table
@ -536,7 +564,7 @@ hash value (for a chosen hash) coincide with those of another file.
@item
In very rare cases, trailing data could be the corrupt header of another
member. In multi-member or concatenated files the probability of
member. In multimember or concatenated files the probability of
corruption happening in the magic bytes is 5 times smaller than the
probability of getting a false positive caused by the corruption of the
integrity information itself. Therefore it can be considered to be below
@ -544,8 +572,8 @@ the noise level.
@end itemize
Trailing data can be safely ignored in most cases. In some cases, like
user-added data, it is expected to be ignored. In those cases where a
file containing trailing data must be rejected, the option
that of user-added data, it is expected to be ignored. In those cases
where a file containing trailing data must be rejected, the option
@samp{--trailing-error} can be used. @xref{--trailing-error}.
@ -616,7 +644,7 @@ mv file_fixed.lz file.lz
@sp 1
@noindent
Example 7: Split the multi-member file @samp{file.lz} and write each
Example 7: Split the multimember file @samp{file.lz} and write each
member in its own @samp{recXXXfile.lz} file. Then use
@w{@samp{lziprecover -t}} to test the integrity of the resulting files.
@ -681,10 +709,22 @@ test robustness to decompression of corrupted data, inspired by
unzcrash.c from Julian Seward's bzip2. Type @samp{make unzcrash} in the
lziprecover source directory to build it.
Unzcrash reads the specified file and then repeatedly decompresses it,
increasing 256 times each byte of the compressed data, so as to test all
possible one-byte errors. This should not cause any invalid memory
accesses. If it does, please, report it as a bug.
By default, unzcrash reads the specified file and then repeatedly
decompresses it, increasing 256 times each byte of the compressed data,
so as to test all possible one-byte errors.
If the @code{--block} option is given, unzcrash reads the specified file
and then repeatedly decompresses it, setting all bytes in each
successive block to the value given, so as to test all possible full
sector errors.
If the @code{--truncate} option is given, unzcrash reads the specified
file and then repeatedly decompresses it, truncating the file to
increasing lengths, so as to test all possible truncation points.
None of the three test modes described above should cause any invalid
memory accesses. If any of them does, please, report it as a bug to the
maintainers of the decompressor being tested.
Unzcrash really executes as a subprocess the shell command specified in
the first non-option argument, and then writes the file specified in the
@ -693,6 +733,27 @@ modifying the corresponding byte each time. Therefore unzcrash can be
used to test any decompressor (not only lzip), or even other decoder
programs having a suitable command line syntax.
If the decompressor returns with zero status, unzcrash compares the
output of the decompressor for the original and corrupt files. If the
outputs differ, it means that the decompressor returned a false
negative; it failed to recognize the corruption and produced garbage
output. The only exception is when a multimember file is truncated just
after the last byte of a member, producing a shorter but valid
compressed file. Except in this latter case, please, report any false
negative as a bug.
In order to compare the outputs, unzcrash needs a @samp{zcmp} program
able to understand the format being tested. For example the one provided
by @samp{zutils}.
@ifnothtml
@xref{Zcmp,,,zutils},
@end ifnothtml
@ifhtml
See
@uref{http://www.nongnu.org/zutils/manual/zutils_manual.html#Zcmp,,zcmp}
@end ifhtml
The format for running unzcrash is:
@example
@ -728,9 +789,21 @@ The number of N-bit errors per byte (N = 1 to 8) is:
@item 1-3,5-8 @tab 1, 2, 3, 5, 6, 7 and 8
@end multitable
@item -B[@var{size}][,@var{value}]
@itemx --block[=@var{size}][,@var{value}]
Test block errors of given @var{size} aligned to a @var{size}-byte
boundary, simulating a whole sector I/O error. Block @var{size} defaults
to 512 bytes. @var{value} defaults to 0.
@item -d @var{n}
@itemx --delta=@var{n}
Test only one of every @var{n} bytes, blocks or truncation sizes,
instead of all of them.
@item -p @var{bytes}
@itemx --position=@var{bytes}
First byte position to test in the file. Defaults to 0.
First byte position to test in the file. Defaults to 0. Negative values
are relative to the end of the file.
@item -q
@itemx --quiet
@ -738,13 +811,24 @@ Quiet operation. Suppress all messages.
@item -s @var{bytes}
@itemx --size=@var{bytes}
Number of byte positions to test. If not specified, the whole file is
tested.
Number of byte positions to test. If not specified, the rest of the file
is tested (from @code{--position} to end of file). Negative values are
relative to the rest of the file.
@item -t
@itemx --truncate
Test all possible truncation points in the range specified by
@code{--position} and @code{--size}.
@item -v
@itemx --verbose
Verbose mode.
@item -z
@itemx --zcmp=<command>
Set zcmp command name and options. Defaults to @code{zcmp}. Use
@code{--zcmp=false} to disable comparisons.
@end table
Exit status: 0 for a normal exit, 1 for environmental problems (file not