lzip/lziprecover
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.18.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-21 11:26:24 +01:00
parent 1327a2b8ad
commit 734bd31e8b
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
45 changed files with 1576 additions and 774 deletions
Download patch file Download diff file Expand all files Collapse all files

223
doc/lziprecover.info
View file

@ -12,7 +12,7 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir)
Lziprecover Manual
******************
This manual is for Lziprecover (version 1.18-pre2, 16 September 2015).
This manual is for Lziprecover (version 1.18, 12 May 2016).
* Menu:
@ -30,7 +30,7 @@ This manual is for Lziprecover (version 1.18-pre2, 16 September 2015).
* Concept index:: Index of concepts
Copyright (C) 2009-2015 Antonio Diaz Diaz.
Copyright (C) 2009-2016 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission to
copy, distribute and modify it.
@ -47,6 +47,9 @@ files, produce a correct file by merging the good parts of two or more
damaged copies, extract data from damaged files, decompress files and
test integrity of files.
Lziprecover is not a replacement for regular backups, but a last
line of defense for the case where the backups are also damaged.
The lzip file format is designed for data sharing and long-term
archiving, taking into account both data integrity and decoder
availability:
@ -72,6 +75,10 @@ repair the nearer it is from the beginning of the file. Therefore, with
the help of lziprecover, losing an entire archive just because of a
corrupt byte near the beginning is a thing of the past.
For compressible data, multiple lzip-compressed copies have a better
chance of surviving intact than one uncompressed copy using the same
amount of storage space.
Lziprecover is able to recover or decompress files produced by any of
the compressors in the lzip family; lzip, plzip, minilzip/lzlib, clzip
and pdlzip.
@ -87,12 +94,11 @@ garbage data may be produced at the end of each member):
lziprecover -D0 -i -o file -q file.lz
Lziprecover is able to efficiently extract a range of bytes from a
multi-member file, because it only decompresses the members containing
the desired data.
Lziprecover provides random access to the data in multimember files;
it only decompresses the members containing the desired data.
Lziprecover can print correct total file sizes and ratios even for
multi-member files.
multimember files.
When recovering data, lziprecover takes as arguments the names of the
damaged files and writes zero or more recovered files depending on the
@ -102,9 +108,6 @@ damaged files themselves are never modified.
When decompressing or testing file integrity, lziprecover behaves
like lzip or lunzip.
Lziprecover is not a replacement for regular backups, but a last
line of defense for the case where the backups are also damaged.

File: lziprecover.info, Node: Invoking lziprecover, Next: Data safety, Prev: Introduction, Up: Top
@ -115,6 +118,10 @@ The format for running lziprecover is:
lziprecover [OPTIONS] [FILES]
When decompressing or testing, '-' used as a FILE argument means
standard input. It can be mixed with other FILES and is read just once,
the first time it appears in the command line.
Lziprecover supports the following options:
'-h'
@ -133,30 +140,51 @@ The format for running lziprecover is:
trailing garbage that can be safely ignored. *Note
concat-example::.
'-A'
'--alone-to-lz'
Convert lzma-alone files to lzip format without recompressing, just
adding a lzip header and trailer. The conversion minimizes the
dictionary size of the resulting file (and therefore the amount of
memory required to decompress it). Only streamed files with
default LZMA properties can be converted; non-streamed lzma-alone
files lack the end of stream marker required in lzip files.
The name of the converted lzip file is derived from that of the
original lzma-alone file as follows:
filename.lzma becomes filename.lz
filename.tlz becomes filename.tar.lz
anyothername becomes anyothername.lz
'-c'
'--stdout'
Decompress to standard output. Needed when reading from a named
pipe (fifo) or from a device. Use it to recover as much of the
Write decompressed data to standard output; keep input files
unchanged. This option is needed when reading from a named pipe
(fifo) or from a device. Use it also to recover as much of the
uncompressed data as possible when decompressing a corrupt file.
'-d'
'--decompress'
Decompress the specified file(s). If a file fails to decompress,
lziprecover exits immediately without decompressing the rest of the
files.
Decompress the specified file(s). If a file does not exist or
can't be opened, lziprecover continues decompressing the rest of
the files. If a file fails to decompress, lziprecover exits
immediately without decompressing the rest of the files.
'-D RANGE'
'--range-decompress=RANGE'
Decompress only a range of bytes starting at decompressed byte
position 'BEGIN' and up to byte position 'END - 1'. Three formats
of RANGE are recognized, 'BEGIN', 'BEGIN-END', and 'BEGIN,SIZE'.
If only BEGIN is specified, END is taken as the end of the file.
The produced bytes are sent to standard output unless the
'--output' option is used. In order to guarantee the correctness
of the data produced, all members containing any part of the
desired data are decompressed and their integrity is verified.
This operation is more efficient in multi-member files because it
only decompresses the members containing the desired data.
position 'BEGIN' and up to byte position 'END - 1'. This option
provides random access to the data in multimember files; it only
decompresses the members containing the desired data. In order to
guarantee the correctness of the data produced, all members
containing any part of the desired data are decompressed and their
integrity is verified.
Four formats of RANGE are recognized, 'BEGIN', 'BEGIN-END',
'BEGIN,SIZE', and ',SIZE'. If only BEGIN is specified, END is taken
as the end of the file. If only SIZE is specified, BEGIN is taken
as the beginning of the file. The produced bytes are sent to
standard output unless the '--output' option is used.
'-f'
'--force'
@ -166,7 +194,7 @@ The format for running lziprecover is:
'--ignore-errors'
Make '--range-decompress' ignore data errors and continue
decompressing the remaining members in the file. For example,
'lziprecover -i -D0 file.lz > file' decompresses all the
'lziprecover -D0 -i file.lz > file' decompresses all the
recoverable data in all members of 'file.lz' without having to
split it first.
@ -177,7 +205,7 @@ The format for running lziprecover is:
'-l'
'--list'
Print total file sizes and ratios. The values produced are correct
even for multi-member files. Use it together with '-v' to see
even for multimember files. Use it together with '-v' to see
information about the members in the file.
'-m'
@ -195,7 +223,10 @@ The format for running lziprecover is:
splitting, the names of the files produced are in the form
'rec01FILE', 'rec02FILE', etc. If decompressing from standard
input and '--stdout' has not been specified, use 'FILE' as the
name of the decompressed file.
name of the decompressed file. If converting a lzma-alone file
from standard input and '--stdout' has not been specified, use
'FILE.lz' as the name of the converted file. (Or plain 'FILE' if
it already ends in '.lz' or '.tlz').
'-q'
'--quiet'
@ -270,15 +301,15 @@ File: lziprecover.info, Node: Data safety, Next: Repairing files, Prev: Invok
**************************************
There are 3 main types of data corruption that may cause data loss:
single-byte errors, multi-byte errors (generally affecting a whole
sector in a block device), and total device failure.
single-byte errors, multibyte errors (generally affecting a whole sector
in a block device), and total device failure.
Lziprecover protects natively against single-byte errors (*note
Repairing files::), as long as file integrity is checked frequently
enough that a second single-byte error does not develop in the same
member before the first one is repaired.
Lziprecover also protects against multi-byte errors (*note Merging
Lziprecover also protects against multibyte errors (*note Merging
files::), if at least one backup copy of the file is made.
The only remedy for total device failure is storing backup copies in
@ -298,10 +329,10 @@ than one compressed data block (usually larger than 900 kB
uncompressed), and if no block is damaged in both files, then the data
can be manually recovered by splitting the files with bzip2recover,
verifying every block and then copying the right blocks in the right
order in another file.
order into another file.
But if you used lzip, the data can be automatically recovered as
long as no byte is damaged in both files.
long as the damaged areas don't overlap.
Note that each error in a bzip2 file makes a whole block unusable,
but each error in a lzip file only affects the damaged bytes, making it
@ -319,8 +350,8 @@ at all. If the reparation is successful, the repaired file will be
identical bit for bit to the original. This makes lzip files resistant
to bit-flip, one of the most common forms of data corruption.
The error may be located anywhere in the file except in the header
(first 6 bytes of each member) or in the 'Member size' field of the
The error may be located anywhere in the file except in the first 5
bytes of each member header or in the 'Member size' field of the
trailer (last 8 bytes of each member). If the error is in the header it
can be easily repaired with a text editor like GNU Moe (*note File
format::). If the error is in the member size, it is enough to ignore
@ -364,19 +395,13 @@ the file.
is damaged in all copies), or are adjacent and the boundary can't be
determined, or if the copies have too many damaged areas.
All the copies must have the same size. If some of them have been
truncated and are therefore smaller than they should, they can be
extended to the correct size with the following command before merging
them with the other copies:
All the copies must have the same size. If any of them is larger or
smaller than it should, either because it has been truncated or because
it got some garbage data appended at the end, it can be brought to the
correct size with the following command before merging it with the other
copies:
ddrescue --extend-outfile=<correct_size> small_file.lz extended_file.lz
If some of the copies have got garbage data at the end and are
therefore larger than they should, their sizes can be reduced to the
correct value with the following command before merging them with the
other copies:
ddrescue --size=<correct_size> large_file.lz reduced_file.lz
ddrescue -s<correct_size> -x<correct_size> file.lz correct_size_file.lz
To give you an idea of its possibilities, when merging two copies,
each of them with one damaged area affecting 1 percent of the copy, the
@ -427,7 +452,7 @@ additional information before, between, or after them.
Each member has the following structure:
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size |
| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size |
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
All multibyte values are stored in little endian order.
@ -450,8 +475,8 @@ additional information before, between, or after them.
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB
Valid values for dictionary size range from 4 KiB to 512 MiB.
'Lzma stream'
The lzma stream, finished by an end of stream marker. Uses default
'LZMA stream'
The LZMA stream, finished by an end of stream marker. Uses default
values for encoder properties. *Note Stream format: (lzip)Stream
format, for a complete description.
@ -465,7 +490,7 @@ additional information before, between, or after them.
Total size of the member, including header and trailer. This field
acts as a distributed index, allows the verification of stream
integrity, and facilitates safe recovery of undamaged members from
multi-member files.
multimember files.

@ -490,15 +515,15 @@ member. Such trailing data may be:
file.
* In very rare cases, trailing data could be the corrupt header of
another member. In multi-member or concatenated files the
another member. In multimember or concatenated files the
probability of corruption happening in the magic bytes is 5 times
smaller than the probability of getting a false positive caused by
the corruption of the integrity information itself. Therefore it
can be considered to be below the noise level.
Trailing data can be safely ignored in most cases. In some cases,
like user-added data, it is expected to be ignored. In those cases
where a file containing trailing data must be rejected, the option
like that of user-added data, it is expected to be ignored. In those
cases where a file containing trailing data must be rejected, the option
'--trailing-error' can be used. *Note --trailing-error::.

@ -550,7 +575,7 @@ are abridged diagnostic messages from lziprecover).
mv file_fixed.lz file.lz
Example 7: Split the multi-member file 'file.lz' and write each member
Example 7: Split the multimember file 'file.lz' and write each member
in its own 'recXXXfile.lz' file. Then use 'lziprecover -t' to test the
integrity of the resulting files.
@ -599,10 +624,22 @@ test robustness to decompression of corrupted data, inspired by
unzcrash.c from Julian Seward's bzip2. Type 'make unzcrash' in the
lziprecover source directory to build it.
Unzcrash reads the specified file and then repeatedly decompresses
it, increasing 256 times each byte of the compressed data, so as to
test all possible one-byte errors. This should not cause any invalid
memory accesses. If it does, please, report it as a bug.
By default, unzcrash reads the specified file and then repeatedly
decompresses it, increasing 256 times each byte of the compressed data,
so as to test all possible one-byte errors.
If the '--block' option is given, unzcrash reads the specified file
and then repeatedly decompresses it, setting all bytes in each
successive block to the value given, so as to test all possible full
sector errors.
If the '--truncate' option is given, unzcrash reads the specified
file and then repeatedly decompresses it, truncating the file to
increasing lengths, so as to test all possible truncation points.
None of the three test modes described above should cause any invalid
memory accesses. If any of them does, please, report it as a bug to the
maintainers of the decompressor being tested.
Unzcrash really executes as a subprocess the shell command specified
in the first non-option argument, and then writes the file specified in
@ -611,6 +648,19 @@ modifying the corresponding byte each time. Therefore unzcrash can be
used to test any decompressor (not only lzip), or even other decoder
programs having a suitable command line syntax.
If the decompressor returns with zero status, unzcrash compares the
output of the decompressor for the original and corrupt files. If the
outputs differ, it means that the decompressor returned a false
negative; it failed to recognize the corruption and produced garbage
output. The only exception is when a multimember file is truncated just
after the last byte of a member, producing a shorter but valid
compressed file. Except in this latter case, please, report any false
negative as a bug.
In order to compare the outputs, unzcrash needs a 'zcmp' program
able to understand the format being tested. For example the one provided
by 'zutils'. *Note Zcmp: (zutils)Zcmp,
The format for running unzcrash is:
unzcrash [OPTIONS] "lzip -tv" FILENAME.lz
@ -642,9 +692,21 @@ programs having a suitable command line syntax.
1,3-5,8 1, 3, 4, 5 and 8
1-3,5-8 1, 2, 3, 5, 6, 7 and 8
'-B[SIZE][,VALUE]'
'--block[=SIZE][,VALUE]'
Test block errors of given SIZE aligned to a SIZE-byte boundary,
simulating a whole sector I/O error. Block SIZE defaults to 512
bytes. VALUE defaults to 0.
'-d N'
'--delta=N'
Test only one of every N bytes, blocks or truncation sizes,
instead of all of them.
'-p BYTES'
'--position=BYTES'
First byte position to test in the file. Defaults to 0.
First byte position to test in the file. Defaults to 0. Negative
values are relative to the end of the file.
'-q'
'--quiet'
@ -652,13 +714,24 @@ programs having a suitable command line syntax.
'-s BYTES'
'--size=BYTES'
Number of byte positions to test. If not specified, the whole file
is tested.
Number of byte positions to test. If not specified, the rest of
the file is tested (from '--position' to end of file). Negative
values are relative to the rest of the file.
'-t'
'--truncate'
Test all possible truncation points in the range specified by
'--position' and '--size'.
'-v'
'--verbose'
Verbose mode.
'-z'
'--zcmp=<command>'
Set zcmp command name and options. Defaults to 'zcmp'. Use
'--zcmp=false' to disable comparisons.
Exit status: 0 for a normal exit, 1 for environmental problems (file
not found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or
@ -706,21 +779,21 @@ Concept index

Tag Table:
Node: Top231
Node: Introduction1278
Node: Invoking lziprecover4395
Ref: --trailing-error4860
Node: Data safety10294
Node: Repairing files12218
Node: Merging files14120
Node: File names15961
Node: File format16425
Node: Trailing data18854
Node: Examples20230
Ref: concat-example20661
Ref: ddrescue-example21725
Node: Unzcrash23015
Node: Problems25571
Node: Concept index26123
Node: Introduction1267
Node: Invoking lziprecover4525
Ref: --trailing-error5175
Node: Data safety11779
Node: Repairing files13702
Node: Merging files15602
Node: File names17217
Node: File format17681
Node: Trailing data20109
Node: Examples21492
Ref: concat-example21923
Ref: ddrescue-example22986
Node: Unzcrash24276
Node: Problems28786
Node: Concept index29338

End Tag Table