lzip/lziprecover
1
0
Fork 0
Code Issues Activity

Merging upstream version 1.21.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-21 11:29:36 +01:00
parent 4b818dc40b
commit 29d9f35b61
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
42 changed files with 2853 additions and 1586 deletions
Download patch file Download diff file Expand all files Collapse all files

41
doc/lziprecover.1
View file

@ -1,27 +1,30 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1.
.TH LZIPRECOVER "1" "February 2018" "lziprecover 1.20" "User Commands"
.TH LZIPRECOVER "1" "January 2019" "lziprecover 1.21" "User Commands"
.SH NAME
lziprecover \- recovers data from damaged lzip files
.SH SYNOPSIS
.B lziprecover
[\fI\,options\/\fR] [\fI\,files\/\fR]
.SH DESCRIPTION
Lziprecover \- Data recovery tool and decompressor for the lzip format.
Lziprecover is a data recovery tool and decompressor for files in the lzip
compressed data format (.lz). Lziprecover is able to repair slightly damaged
files, produce a correct file by merging the good parts of two or more
damaged copies, extract data from damaged files, decompress files and test
integrity of files.
.PP
Lziprecover can repair perfectly most files with small errors (up to one
single\-byte error per member), without the need of any extra redundance
at all. Losing an entire archive just because of a corrupt byte near the
beginning is a thing of the past.
.PP
Lziprecover can also produce a correct file by merging the good parts of
two or more damaged copies, extract data from damaged files, decompress
files and test integrity of files.
Lziprecover can remove the damaged members from multimember files, for
example multimember tar.lz archives.
.PP
Lziprecover provides random access to the data in multimember files; it
only decompresses the members containing the desired data.
Lziprecover provides random access to the data in multimember files; it only
decompresses the members containing the desired data.
.PP
Lziprecover facilitates the management of metadata stored as trailing
data in lzip files.
Lziprecover facilitates the management of metadata stored as trailing data
in lzip files.
.PP
Lziprecover is not a replacement for regular backups, but a last line of
defense for the case where the backups are also damaged.
@ -45,14 +48,14 @@ write to standard output, keep input files
\fB\-d\fR, \fB\-\-decompress\fR
decompress
.TP
\fB\-D\fR, \fB\-\-range\-decompress=\fR<range>
decompress a range of bytes (N\-M) to stdout
\fB\-D\fR, \fB\-\-range\-decompress=\fR<n\-m>
decompress a range of bytes to stdout
.TP
\fB\-f\fR, \fB\-\-force\fR
overwrite existing output files
.TP
\fB\-i\fR, \fB\-\-ignore\-errors\fR
make '\-\-range\-decompress' ignore data errors
all errors in \fB\-D\fR, format errors in \fB\-l\fR, \fB\-\-dump\fR
.TP
\fB\-k\fR, \fB\-\-keep\fR
keep (don't delete) input files
@ -84,14 +87,14 @@ be verbose (a 2nd \fB\-v\fR gives more)
\fB\-\-loose\-trailing\fR
allow trailing data seeming corrupt header
.TP
\fB\-\-dump\-tdata\fR
dump trailing data to standard output
\fB\-\-dump=\fR<list>:d:t
dump members listed/damaged, tdata to stdout
.TP
\fB\-\-remove\-tdata\fR
remove trailing data from files in place
\fB\-\-remove=\fR<list>:d:t
remove members, tdata from files in place
.TP
\fB\-\-strip\-tdata\fR
copy files to stdout without trailing data
\fB\-\-strip=\fR<list>:d:t
copy files to stdout stripping members given
.PP
If no file names are given, or if a file is '\-', lziprecover decompresses
from standard input to standard output.
@ -107,7 +110,7 @@ Report bugs to lzip\-bug@nongnu.org
.br
Lziprecover home page: http://www.nongnu.org/lzip/lziprecover.html
.SH COPYRIGHT
Copyright \(co 2018 Antonio Diaz Diaz.
Copyright \(co 2019 Antonio Diaz Diaz.
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>
.br
This is free software: you are free to change and redistribute it.

302
doc/lziprecover.info
View file

@ -12,7 +12,7 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir)
Lziprecover Manual
******************
This manual is for Lziprecover (version 1.20, 12 February 2018).
This manual is for Lziprecover (version 1.21, 4 January 2019).
* Menu:
@ -21,6 +21,7 @@ This manual is for Lziprecover (version 1.20, 12 February 2018).
* Data safety:: Protecting data from accidental loss
* Repairing files:: Fixing bit flips and similar errors
* Merging files:: Fixing several damaged copies
* Tarlz:: Options supporting the tar.lz format
* File names:: Names of the files produced by lziprecover
* File format:: Detailed format of the compressed file
* Trailing data:: Extra data appended to the file
@ -30,7 +31,7 @@ This manual is for Lziprecover (version 1.20, 12 February 2018).
* Concept index:: Index of concepts
Copyright (C) 2009-2018 Antonio Diaz Diaz.
Copyright (C) 2009-2019 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission to
copy, distribute and modify it.
@ -43,13 +44,19 @@ File: lziprecover.info, Node: Introduction, Next: Invoking lziprecover, Prev:
Lziprecover is a data recovery tool and decompressor for files in the
lzip compressed data format (.lz). Lziprecover is able to repair
slightly damaged files, produce a correct file by merging the good parts
of two or more damaged copies, extract data from damaged files,
slightly damaged files, produce a correct file by merging the good
parts of two or more damaged copies, extract data from damaged files,
decompress files and test integrity of files.
Lziprecover can remove the damaged members from multimember files,
for example multimember tar.lz archives.
Lziprecover provides random access to the data in multimember files;
it only decompresses the members containing the desired data.
Lziprecover facilitates the management of metadata stored as trailing
data in lzip files.
Lziprecover is not a replacement for regular backups, but a last
line of defense for the case where the backups are also damaged.
@ -100,7 +107,7 @@ garbage data may be produced at the end of each member):
When recovering data, lziprecover takes as arguments the names of the
damaged files and writes zero or more recovered files depending on the
operation selected and whether the recovery succeeded or not. The
damaged files themselves are never modified.
damaged files themselves are kept unchanged.
When decompressing or testing file integrity, lziprecover behaves
like lzip or lunzip.
@ -132,7 +139,7 @@ the first time it appears in the command line.
'-V'
'--version'
Print the version number of lziprecover on the standard output and
exit.
exit. This version number should be included in all bug reports.
'-a'
'--trailing-error'
@ -194,12 +201,15 @@ the first time it appears in the command line.
'-i'
'--ignore-errors'
Make '--range-decompress' ignore data errors and continue
decompressing the remaining members in the file. For example,
Make '--range-decompress' ignore errors and continue decompressing
the remaining members in the file. For example,
'lziprecover -D0 -i file.lz > file' decompresses all the
recoverable data in all members of 'file.lz' without having to
split it first.
Make '--list', '--dump', '--remove' and '--strip' ignore format
errors.
'-k'
'--keep'
Keep (don't delete) input files during decompression.
@ -213,20 +223,23 @@ the first time it appears in the command line.
printed. With '-v', the dictionary size, the number of members in
the file, and the amount of trailing data (if any) are also
printed. With '-vv', the positions and sizes of each member in
multimember files are also printed. '-lq' can be used to verify
quickly (without decompressing) the structural integrity of the
specified files. (Use '--test' to verify the data integrity).
'-alq' additionally verifies that none of the specified files
contain trailing data.
multimember files are also printed. With '-i', format errors are
ignored, and with '-ivv', gaps between members are shown. The
member numbers shown coincide with the file numbers produced by
'--split'.
'-lq' can be used to verify quickly (without decompressing) the
structural integrity of the specified files. (Use '--test' to
verify the data integrity). '-alq' additionally verifies that none
of the specified files contain trailing data.
'-m'
'--merge'
Try to produce a correct file by merging the good parts of two or
more damaged copies. If successful, a repaired copy is written to
the file 'FILE_fixed.lz'. The exit status is 0 if a correct file
could be produced, 2 otherwise. See the chapter 'Merging files'
(*note Merging files::) for a complete description of the merge
mode.
could be produced, 2 otherwise. *Note Merging files::, for a
complete description of the merge mode.
'-o FILE'
'--output=FILE'
@ -248,17 +261,21 @@ the first time it appears in the command line.
Try to repair a file with small errors (up to one single-byte
error per member). If successful, a repaired copy is written to
the file 'FILE_fixed.lz'. 'FILE' is not modified at all. The exit
status is 0 if the file could be repaired, 2 otherwise. See the
chapter 'Repairing files' (*note Repairing files::) for a complete
description of the repair mode.
status is 0 if the file could be repaired, 2 otherwise. *Note
Repairing files::, for a complete description of the repair mode.
'-s'
'--split'
Search for members in 'FILE' and write each member in its own
'.lz' file. You can then use 'lziprecover -t' to test the
integrity of the resulting files, decompress those which are
undamaged, and try to repair or partially decompress those which
are damaged.
file. Gaps between members are detected and each gap is saved in
its own file. Trailing data (if any) are saved alone in the last
file. You can then use 'lziprecover -t' to test the integrity of
the resulting files, decompress those which are undamaged, and try
to repair or partially decompress those which are damaged. Gaps
may contain garbage or may be members with corrupt headers or
trailers. If other lziprecover functions fail to work on a
multimember FILE because of damage in headers or trailers, try to
split FILE and then work on each member individually.
The names of the files produced are in the form 'rec01FILE',
'rec02FILE', etc, and are designed so that the use of wildcards in
@ -297,33 +314,75 @@ the first time it appears in the command line.
if a file triggers a "corrupt header" error and the cause is not
indeed a corrupt header.
'--dump-tdata'
Dump the trailing data (if any) of one or more regular files to
'--dump=[MEMBER_LIST][:damaged][:tdata]'
Dump the members listed, the damaged members (if any), or the
trailing data (if any) of one or more regular multimember files to
standard output, or to a file if the '--output' option is used. If
more than one file is given, the trailing data of all files are
concatenated. If a file does not exist, can't be opened, or is not
regular, lziprecover continues processing the rest of the files.
If the dump fails in one file, lziprecover exits immediately
without processing the rest of the files.
more than one file is given, the elements dumped from all files
are concatenated. If a file does not exist, can't be opened, or
is not regular, lziprecover continues processing the rest of the
files. If the dump fails in one file, lziprecover exits
immediately without processing the rest of the files.
'--remove-tdata'
Remove the trailing data from regular files in place. The date of
each file is preserved if possible. If the removal fails in one
file, lziprecover continues processing the rest of the files. This
option may be dangerous if the file is corrupt or if the trailing
data contain a forbidden combination of characters. *Note Trailing
data::. Verify that 'lzip -cd file.lz | wc -c' and the
The argument to '--dump' is a colon-separated list of the following
element specifiers; a member list (1,3-6), a reverse member list
(r1,3-6), and the strings "damaged" and "tdata" (which may be
shortened to 'd' and 't' respectively). A member list selects the
members (or gaps) listed, whose numbers coincide with those shown
by '--list'. A reverse member list selects the members listed
counting from the last member in the file (r1). Negated versions
of both kinds of lists exist (^1,3-6:r^1,3-6) which selects all
the members except those in the list. The strings "damaged" and
"tdata" select the damaged members and the trailing data
respectively. If the same member is selected more than once, for
example by '1:r1' in a single-member file, it is dumped just once.
See the following examples:
'--dump' argument Elements dumped
---------------------------------------------------------------------
'1,3-6' members 1, 3, 4, 5 and 6
'r1-3' last 3 members in file
'^13,15' all but 13th and 15th members in file
'r^1' all but last member in file
'damaged' all damaged members in file
'tdata' trailing data
'1-5:r1:tdata' members 1 to 5, last member, trailing data
'damaged:tdata' damaged members, trailing data
'3,12:damaged:tdata' members 3, 12, damaged members, trailing data
'--remove=[MEMBER_LIST][:damaged][:tdata]'
Remove the members listed, the damaged members (if any), or the
trailing data (if any) from regular multimember files in place.
The date of each file is preserved if possible. If all members in
a file are selected to be removed, the file is left unchanged and
the exit status is set to 2. If a file does not exist, can't be
opened, is not regular, or is left unchanged, lziprecover
continues processing the rest of the files. In case of I/O error,
lziprecover exits immediately without processing the rest of the
files. See '--dump' above for a description of the argument.
This option may be dangerous even if only the trailing data is
being removed because the file may be corrupt or the trailing data
may contain a forbidden combination of characters. *Note Trailing
data::. It is advisable to make a backup before attempting the
removal. At least verify that 'lzip -cd file.lz | wc -c' and the
uncompressed size shown by 'lzip -l file.lz' match before
attempting the removal.
attempting the removal of trailing data.
'--strip-tdata'
Copy one or more regular files to standard output (or to a file if
the '--output' option is used), stripping the trailing data (if
any) from each file. If more than one file is given, the files are
concatenated. If a file does not exist, can't be opened, or is not
'--strip=[MEMBER_LIST][:damaged][:tdata]'
Copy one or more regular multimember files to standard output (or
to a file if the '--output' option is used), stripping the members
listed, the damaged members (if any), or the trailing data (if
any) from each file. If all members in a file are selected to be
stripped, the trailing data (if any) are also stripped even if
'tdata' is not specified. If more than one file is given, the
files are concatenated. In this case the trailing data are also
stripped from all but the last file even if 'tdata' is not
specified. If a file does not exist, can't be opened, or is not
regular, lziprecover continues processing the rest of the files.
If a file fails to copy, lziprecover exits immediately without
processing the rest of the files.
processing the rest of the files. See '--dump' above for a
description of the argument.
Numbers given as arguments to options may be followed by a multiplier
@ -431,7 +490,7 @@ cause much more loss of data than errors located near the end. So
lziprecover repairs more efficiently the worst errors.

File: lziprecover.info, Node: Merging files, Next: File names, Prev: Repairing files, Up: Top
File: lziprecover.info, Node: Merging files, Next: Tarlz, Prev: Repairing files, Up: Top
5 Merging files
***************
@ -489,9 +548,74 @@ lower than the number of corrupt bytes (3104) because contiguous
corrupt bytes are counted as a single multibyte error.

File: lziprecover.info, Node: File names, Next: File format, Prev: Merging files, Up: Top
File: lziprecover.info, Node: Tarlz, Next: File names, Prev: Merging files, Up: Top
6 Names of the files produced by lziprecover
6 Options supporting the tar.lz format
**************************************
Tarlz is an implementation of the tar archiver which by default creates
archives compressed with lzip on a per file basis. Tarlz can append
files to the end of such compressed archives because each tar member is
compressed in its own lzip member, as well as the end-of-file blocks.
Thus tarlz archives are multimember lzip files, which has some safety
advantages over solidly compressed tar.lz archives. For example, in
case of corruption, tarlz can extract all the undamaged members from
the tar.lz archive, skipping over the damaged members, just like the
standard (uncompressed) tar. In this chapter we'll explain the ways in
which lziprecover can recover and process multimember tar.lz archives.
*Note tarlz manual: (tarlz)Top.
6.1 Recovering damaged multimember tar.lz archives
==================================================
If you have several copies of the damaged archive, try merging them
first because merging has a high probability of success. If the command
below prints something like 'Input files merged successfully.' you are
done and 'archive.tar.lz' now contains the recovered archive:
lziprecover -m -v -o archive.tar.lz a/archive.tar.lz b/archive.tar.lz
If you only have one copy of the damaged archive, you may try to
repair the archive, but this has a lower probability of success. If the
command below prints something like
'Copy of input file repaired successfully.' you are done and
'archive_fixed.tar.lz' now contains the recovered archive:
lziprecover -v -R archive.tar.lz
If all the above fails, you may save the damaged members for later
and then copy the good members to another archive. If the two commands
below succeed, 'bad_members.tar.lz' will contain all the damaged members
and 'archive_cleaned.tar.lz' will contain a good archive with the
damaged members removed:
lziprecover -v --dump=damaged -o bad_members.tar.lz archive.tar.lz
lziprecover -v --strip=damaged -o archive_cleaned.tar.lz archive.tar.lz
You can then use 'tarlz --keep-damaged' to recover as much data as
possible from each damaged member in 'bad_members.tar.lz':
mkdir tmp
cd tmp
tarlz --keep-damaged -xvf ../bad_members.tar.lz
6.2 Processing multimember tar.lz archives
==========================================
Lziprecover is able to copy a list of members from a file to another.
For example the command
'lziprecover --dump=1-10:r1:tdata archive.tar.lz > subarch.tar.lz'
creates a subset archive containing the first ten members, the
end-of-file blocks, and the trailing data (if any) of 'archive.tar.lz'.
The 'r1' part selects the last member, which in an appendable tar.lz
archive contains the end-of-file blocks.

File: lziprecover.info, Node: File names, Next: File format, Prev: Tarlz, Up: Top
7 Names of the files produced by lziprecover
********************************************
The name of the fixed file produced by '--merge' and '--repair' is made
@ -502,7 +626,7 @@ original file name ends with one of the extensions '.tar.lz', '.lz' or

File: lziprecover.info, Node: File format, Next: Trailing data, Prev: File names, Up: Top
7 File format
8 File format
*************
Perfection is reached, not when there is no longer anything to add, but
@ -544,11 +668,11 @@ additional information before, between, or after them.
'DS (coded dictionary size, 1 byte)'
The dictionary size is calculated by taking a power of 2 (the base
size) and substracting from it a fraction between 0/16 and 7/16 of
size) and subtracting from it a fraction between 0/16 and 7/16 of
the base size.
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).
Bits 7-5 contain the numerator of the fraction (0 to 7) to
substract from the base size to obtain the dictionary size.
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
from the base size to obtain the dictionary size.
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB
Valid values for dictionary size range from 4 KiB to 512 MiB.
@ -573,7 +697,7 @@ additional information before, between, or after them.

File: lziprecover.info, Node: Trailing data, Next: Examples, Prev: File format, Up: Top
8 Extra data appended to the file
9 Extra data appended to the file
*********************************
Sometimes extra data are found appended to a lzip file after the last
@ -626,11 +750,11 @@ Example 1: Add a comment or description to a compressed file.
# First append the comment as trailing data to a lzip file
echo 'This file contains this and that' >> file.lz
# This command prints the comment to standard output
lziprecover --dump-tdata file.lz
lziprecover --dump=tdata file.lz
# This command outputs file.lz without the comment
lziprecover --strip-tdata file.lz
lziprecover --strip=tdata file.lz
# This command removes the comment from file.lz
lziprecover --remove-tdata file.lz
lziprecover --remove=tdata file.lz
Example 2: Add and verify a cryptographically secure hash. (This may be
@ -639,14 +763,14 @@ to guarantee that both file and hash have not been maliciously
replaced).
sha256sum < file.lz >> file.lz
lziprecover --strip-tdata file.lz | sha256sum -c \
<(lziprecover --dump-tdata file.lz)
lziprecover --strip=tdata file.lz | sha256sum -c \
<(lziprecover --dump=tdata file.lz)

File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: Trailing data, Up: Top
9 A small tutorial with examples
********************************
10 A small tutorial with examples
*********************************
Example 1: Restore a regular file from its compressed version
'file.lz'. If the operation is successful, 'file.lz' is removed.
@ -667,6 +791,10 @@ or more compressed files. *Note Trailing data::.
cat file1.lz file2.lz file3.lz | lziprecover -d
Do this instead
lziprecover -cd file1.lz file2.lz file3.lz
You may also concatenate the compressed files like this
lziprecover --strip=tdata file1.lz file2.lz file3.lz > file123.lz
Or keeping the trailing data of the last file like this
lziprecover --strip=damaged file1.lz file2.lz file3.lz > file123.lz
Example 4: Decompress 'file.lz' partially until 10 KiB of decompressed
@ -700,8 +828,8 @@ integrity of the resulting files.
Example 8: Recover a compressed backup from two copies on CD-ROM with
error-checked merging of copies. (*Note GNU ddrescue manual:
(ddrescue)Top, for details about ddrescue).
error-checked merging of copies. *Note GNU ddrescue manual:
(ddrescue)Top, for details about ddrescue.
ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1
mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage
@ -732,7 +860,7 @@ correct file produced is saved in 'big_db_00001.lz'.

File: lziprecover.info, Node: Unzcrash, Next: Problems, Prev: Examples, Up: Top
10 Testing the robustness of decompressors
11 Testing the robustness of decompressors
******************************************
The lziprecover package also includes unzcrash, a program written to
@ -775,14 +903,17 @@ after the last byte of a member, producing a shorter but valid
compressed file. Except in this latter case, please, report any false
negative as a bug.
In order to compare the outputs, unzcrash needs a 'zcmp' program
able to understand the format being tested. For example the one provided
by 'zutils'. *Note Zcmp: (zutils)Zcmp,
In order to compare the outputs, unzcrash needs a 'zcmp' program able
to understand the format being tested. For example the 'zcmp' provided
by 'zutils'. *Note Zcmp: (zutils)Zcmp.
The format for running unzcrash is:
unzcrash [OPTIONS] 'lzip -t' FILE.lz
FILE.lz must not contain errors and must be correctly decompressed by
the decompressor being tested for the comparisons to work.
unzcrash supports the following options:
'-h'
@ -792,7 +923,7 @@ by 'zutils'. *Note Zcmp: (zutils)Zcmp,
'-V'
'--version'
Print the version number of unzcrash on the standard output and
exit.
exit. This version number should be included in all bug reports.
'-b RANGE'
'--bits=RANGE'
@ -868,7 +999,10 @@ by 'zutils'. *Note Zcmp: (zutils)Zcmp,
'-z'
'--zcmp=<command>'
Set zcmp command name and options. Defaults to 'zcmp'. Use
'--zcmp=false' to disable comparisons.
'--zcmp=false' to disable comparisons. If testing a decompressor
different from the one used by default by zcmp, it is needed to
force unzcrash and zcmp to use the same decompressor with a
command like 'unzcrash --zcmp='zcmp --lz=plzip' 'plzip -t' FILE.lz'
Exit status: 0 for a normal exit, 1 for environmental problems (file
@ -879,7 +1013,7 @@ caused unzcrash to panic.

File: lziprecover.info, Node: Problems, Next: Concept index, Prev: Unzcrash, Up: Top
11 Reporting bugs
12 Reporting bugs
*****************
There are probably bugs in lziprecover. There are certainly errors and
@ -911,6 +1045,7 @@ Concept index
* merging files: Merging files. (line 6)
* options: Invoking lziprecover. (line 6)
* repairing files: Repairing files. (line 6)
* tarlz: Tarlz. (line 6)
* trailing data: Trailing data. (line 6)
* unzcrash: Unzcrash. (line 6)
* usage: Invoking lziprecover. (line 6)
@ -920,21 +1055,22 @@ Concept index

Tag Table:
Node: Top231
Node: Introduction1273
Node: Invoking lziprecover4650
Ref: --trailing-error5300
Node: Data safety14832
Node: Repairing files16783
Node: Merging files18706
Node: File names21468
Node: File format21932
Node: Trailing data24360
Node: Examples27595
Ref: concat-example28026
Ref: ddrescue-example29127
Node: Unzcrash30417
Node: Problems36055
Node: Concept index36607
Node: Introduction1335
Node: Invoking lziprecover4918
Ref: --trailing-error5628
Node: Data safety18371
Node: Repairing files20322
Node: Merging files22245
Node: Tarlz25002
Node: File names27857
Node: File format28313
Node: Trailing data30739
Node: Examples33974
Ref: concat-example34407
Ref: ddrescue-example35778
Node: Unzcrash37066
Node: Problems43130
Node: Concept index43682

End Tag Table

273
doc/lziprecover.texi
View file

@ -6,8 +6,8 @@
@finalout
@c %**end of header
@set UPDATED 12 February 2018
@set VERSION 1.20
@set UPDATED 4 January 2019
@set VERSION 1.21
@dircategory Data Compression
@direntry
@ -40,6 +40,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}).
* Data safety:: Protecting data from accidental loss
* Repairing files:: Fixing bit flips and similar errors
* Merging files:: Fixing several damaged copies
* Tarlz:: Options supporting the tar.lz format
* File names:: Names of the files produced by lziprecover
* File format:: Detailed format of the compressed file
* Trailing data:: Extra data appended to the file
@ -50,7 +51,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}).
@end menu
@sp 1
Copyright @copyright{} 2009-2018 Antonio Diaz Diaz.
Copyright @copyright{} 2009-2019 Antonio Diaz Diaz.
This manual is free documentation: you have unlimited permission
to copy, distribute and modify it.
@ -60,15 +61,22 @@ to copy, distribute and modify it.
@chapter Introduction
@cindex introduction
Lziprecover is a data recovery tool and decompressor for files in the
lzip compressed data format (.lz). Lziprecover is able to repair
slightly damaged files, produce a correct file by merging the good parts
of two or more damaged copies, extract data from damaged files,
decompress files and test integrity of files.
@uref{http://www.nongnu.org/lzip/lziprecover.html,,Lziprecover} is a
data recovery tool and decompressor for files in the lzip compressed
data format (.lz). Lziprecover is able to repair slightly damaged files,
produce a correct file by merging the good parts of two or more damaged
copies, extract data from damaged files, decompress files and test
integrity of files.
Lziprecover can remove the damaged members from multimember files, for
example multimember tar.lz archives.
Lziprecover provides random access to the data in multimember files; it
only decompresses the members containing the desired data.
Lziprecover facilitates the management of metadata stored as trailing
data in lzip files.
Lziprecover is not a replacement for regular backups, but a last line of
defense for the case where the backups are also damaged.
@ -126,7 +134,7 @@ lziprecover -D0 -i -o file -q file.lz
When recovering data, lziprecover takes as arguments the names of the
damaged files and writes zero or more recovered files depending on the
operation selected and whether the recovery succeeded or not. The
damaged files themselves are never modified.
damaged files themselves are kept unchanged.
When decompressing or testing file integrity, lziprecover behaves like
lzip or lunzip.
@ -164,6 +172,7 @@ Print an informative help message describing the options and exit.
@item -V
@itemx --version
Print the version number of lziprecover on the standard output and exit.
This version number should be included in all bug reports.
@anchor{--trailing-error}
@item -a
@ -227,12 +236,15 @@ Force overwrite of output files.
@item -i
@itemx --ignore-errors
Make @samp{--range-decompress} ignore data errors and continue
decompressing the remaining members in the file. For example,
Make @samp{--range-decompress} ignore errors and continue decompressing
the remaining members in the file. For example,
@w{@samp{lziprecover -D0 -i file.lz > file}} decompresses all the
recoverable data in all members of @samp{file.lz} without having to
split it first.
Make @samp{--list}, @samp{--dump}, @samp{--remove} and @samp{--strip}
ignore format errors.
@item -k
@itemx --keep
Keep (don't delete) input files during decompression.
@ -246,18 +258,22 @@ final line containing the cumulative sizes is printed. With @samp{-v},
the dictionary size, the number of members in the file, and the amount
of trailing data (if any) are also printed. With @samp{-vv}, the
positions and sizes of each member in multimember files are also
printed. @samp{-lq} can be used to verify quickly (without
decompressing) the structural integrity of the specified files. (Use
@samp{--test} to verify the data integrity). @samp{-alq} additionally
verifies that none of the specified files contain trailing data.
printed. With @samp{-i}, format errors are ignored, and with
@samp{-ivv}, gaps between members are shown. The member numbers shown
coincide with the file numbers produced by @samp{--split}.
@samp{-lq} can be used to verify quickly (without decompressing) the
structural integrity of the specified files. (Use @samp{--test} to
verify the data integrity). @samp{-alq} additionally verifies that none
of the specified files contain trailing data.
@item -m
@itemx --merge
Try to produce a correct file by merging the good parts of two or more
damaged copies. If successful, a repaired copy is written to the file
@samp{@var{file}_fixed.lz}. The exit status is 0 if a correct file could
be produced, 2 otherwise. See the chapter @samp{Merging files}
(@pxref{Merging files}) for a complete description of the merge mode.
be produced, 2 otherwise. @xref{Merging files}, for a complete
description of the merge mode.
@item -o @var{file}
@itemx --output=@var{file}
@ -280,16 +296,21 @@ Quiet operation. Suppress all messages.
Try to repair a file with small errors (up to one single-byte error per
member). If successful, a repaired copy is written to the file
@samp{@var{file}_fixed.lz}. @samp{@var{file}} is not modified at all.
The exit status is 0 if the file could be repaired, 2 otherwise. See the
chapter @samp{Repairing files} (@pxref{Repairing files}) for a complete
description of the repair mode.
The exit status is 0 if the file could be repaired, 2 otherwise.
@xref{Repairing files}, for a complete description of the repair mode.
@item -s
@itemx --split
Search for members in @samp{@var{file}} and write each member in its own
@samp{.lz} file. You can then use @samp{lziprecover -t} to test the
integrity of the resulting files, decompress those which are undamaged,
and try to repair or partially decompress those which are damaged.
file. Gaps between members are detected and each gap is saved in its own
file. Trailing data (if any) are saved alone in the last file. You can
then use @samp{lziprecover -t} to test the integrity of the resulting
files, decompress those which are undamaged, and try to repair or
partially decompress those which are damaged. Gaps may contain garbage
or may be members with corrupt headers or trailers. If other lziprecover
functions fail to work on a multimember @var{file} because of damage in
headers or trailers, try to split @var{file} and then work on each
member individually.
The names of the files produced are in the form @samp{rec01@var{file}},
@samp{rec02@var{file}}, etc, and are designed so that the use of
@ -326,33 +347,75 @@ bytes are so similar to the magic bytes of a lzip header that they can
be confused with a corrupt header. Use this option if a file triggers a
"corrupt header" error and the cause is not indeed a corrupt header.
@item --dump-tdata
Dump the trailing data (if any) of one or more regular files to standard
@item --dump=[@var{member_list}][:damaged][:tdata]
Dump the members listed, the damaged members (if any), or the trailing
data (if any) of one or more regular multimember files to standard
output, or to a file if the @samp{--output} option is used. If more than
one file is given, the trailing data of all files are concatenated. If a
file does not exist, can't be opened, or is not regular, lziprecover
continues processing the rest of the files. If the dump fails in one
file, lziprecover exits immediately without processing the rest of the
files.
@item --remove-tdata
Remove the trailing data from regular files in place. The date of each
file is preserved if possible. If the removal fails in one file,
lziprecover continues processing the rest of the files. This option may
be dangerous if the file is corrupt or if the trailing data contain a
forbidden combination of characters. @xref{Trailing data}. Verify that
@w{@samp{lzip -cd file.lz | wc -c}} and the uncompressed size shown by
@w{@samp{lzip -l file.lz}} match before attempting the removal.
@item --strip-tdata
Copy one or more regular files to standard output (or to a file if the
@samp{--output} option is used), stripping the trailing data (if any)
from each file. If more than one file is given, the files are
concatenated. If a file does not exist, can't be opened, or is not
regular, lziprecover continues processing the rest of the files. If a
file fails to copy, lziprecover exits immediately without processing the
one file is given, the elements dumped from all files are concatenated.
If a file does not exist, can't be opened, or is not regular,
lziprecover continues processing the rest of the files. If the dump
fails in one file, lziprecover exits immediately without processing the
rest of the files.
The argument to @samp{--dump} is a colon-separated list of the following
element specifiers; a member list (1,3-6), a reverse member list
(r1,3-6), and the strings "damaged" and "tdata" (which may be shortened
to 'd' and 't' respectively). A member list selects the members (or
gaps) listed, whose numbers coincide with those shown by @samp{--list}.
A reverse member list selects the members listed counting from the last
member in the file (r1). Negated versions of both kinds of lists exist
(^1,3-6:r^1,3-6) which selects all the members except those in the list.
The strings "damaged" and "tdata" select the damaged members and the
trailing data respectively. If the same member is selected more than
once, for example by @code{1:r1} in a single-member file, it is dumped
just once. See the following examples:
@multitable {@code{3,12:damaged:tdata}} {members 3, 12, damaged members, trailing data}
@headitem @code{--dump} argument @tab Elements dumped
@item @code{1,3-6} @tab members 1, 3, 4, 5 and 6
@item @code{r1-3} @tab last 3 members in file
@item @code{^13,15} @tab all but 13th and 15th members in file
@item @code{r^1} @tab all but last member in file
@item @code{damaged} @tab all damaged members in file
@item @code{tdata} @tab trailing data
@item @code{1-5:r1:tdata} @tab members 1 to 5, last member, trailing data
@item @code{damaged:tdata} @tab damaged members, trailing data
@item @code{3,12:damaged:tdata} @tab members 3, 12, damaged members, trailing data
@end multitable
@item --remove=[@var{member_list}][:damaged][:tdata]
Remove the members listed, the damaged members (if any), or the trailing
data (if any) from regular multimember files in place. The date of each
file is preserved if possible. If all members in a file are selected to
be removed, the file is left unchanged and the exit status is set to 2.
If a file does not exist, can't be opened, is not regular, or is left
unchanged, lziprecover continues processing the rest of the files. In
case of I/O error, lziprecover exits immediately without processing the
rest of the files. See @samp{--dump} above for a description of the
argument.
This option may be dangerous even if only the trailing data is being
removed because the file may be corrupt or the trailing data may contain
a forbidden combination of characters. @xref{Trailing data}. It is
advisable to make a backup before attempting the removal. At least
verify that @w{@samp{lzip -cd file.lz | wc -c}} and the uncompressed
size shown by @w{@samp{lzip -l file.lz}} match before attempting the
removal of trailing data.
@item --strip=[@var{member_list}][:damaged][:tdata]
Copy one or more regular multimember files to standard output (or to a
file if the @samp{--output} option is used), stripping the members
listed, the damaged members (if any), or the trailing data (if any) from
each file. If all members in a file are selected to be stripped, the
trailing data (if any) are also stripped even if @samp{tdata} is not
specified. If more than one file is given, the files are concatenated.
In this case the trailing data are also stripped from all but the last
file even if @samp{tdata} is not specified. If a file does not exist,
can't be opened, or is not regular, lziprecover continues processing the
rest of the files. If a file fails to copy, lziprecover exits
immediately without processing the rest of the files. See @samp{--dump}
above for a description of the argument.
@end table
Numbers given as arguments to options may be followed by a multiplier
@ -521,6 +584,80 @@ than the number of corrupt bytes (3104) because contiguous corrupt bytes
are counted as a single multibyte error.
@node Tarlz
@chapter Options supporting the tar.lz format
@cindex tarlz
@uref{http://www.nongnu.org/lzip/manual/tarlz_manual.html,,Tarlz} is an
implementation of the tar archiver which by default creates archives
compressed with lzip on a per file basis. Tarlz can append files to the
end of such compressed archives because each tar member is compressed in
its own lzip member, as well as the end-of-file blocks. Thus tarlz
archives are multimember lzip files, which has some safety advantages
over solidly compressed tar.lz archives. For example, in case of
corruption, tarlz can extract all the undamaged members from the tar.lz
archive, skipping over the damaged members, just like the standard
(uncompressed) tar. In this chapter we'll explain the ways in which
lziprecover can recover and process multimember tar.lz archives.
@ifnothtml
@xref{Top,tarlz manual,,tarlz}.
@end ifnothtml
@sp 1
@section Recovering damaged multimember tar.lz archives
If you have several copies of the damaged archive, try merging
them first because merging has a high probability of success. If the
command below prints something like
@w{@code{Input files merged successfully.}} you are done and
@code{archive.tar.lz} now contains the recovered archive:
@example
lziprecover -m -v -o archive.tar.lz a/archive.tar.lz b/archive.tar.lz
@end example
If you only have one copy of the damaged archive, you may try to repair
the archive, but this has a lower probability of success. If the command
below prints something like
@w{@code{Copy of input file repaired successfully.}} you are done and
@code{archive_fixed.tar.lz} now contains the recovered archive:
@example
lziprecover -v -R archive.tar.lz
@end example
If all the above fails, you may save the damaged members for later and
then copy the good members to another archive. If the two commands below
succeed, @code{bad_members.tar.lz} will contain all the damaged members
and @code{archive_cleaned.tar.lz} will contain a good archive with the
damaged members removed:
@example
lziprecover -v --dump=damaged -o bad_members.tar.lz archive.tar.lz
lziprecover -v --strip=damaged -o archive_cleaned.tar.lz archive.tar.lz
@end example
You can then use @code{tarlz --keep-damaged} to recover as much data as
possible from each damaged member in @samp{bad_members.tar.lz}:
@example
mkdir tmp
cd tmp
tarlz --keep-damaged -xvf ../bad_members.tar.lz
@end example
@sp 1
@section Processing multimember tar.lz archives
Lziprecover is able to copy a list of members from a file to another.
For example the command
@w{@code{lziprecover --dump=1-10:r1:tdata archive.tar.lz > subarch.tar.lz}}
creates a subset archive containing the first ten members, the
end-of-file blocks, and the trailing data (if any) of
@code{archive.tar.lz}. The @code{r1} part selects the last member, which
in an appendable tar.lz archive contains the end-of-file blocks.
@node File names
@chapter Names of the files produced by lziprecover
@cindex file names
@ -581,10 +718,10 @@ Just in case something needs to be modified in the future. 1 for now.
@item DS (coded dictionary size, 1 byte)
The dictionary size is calculated by taking a power of 2 (the base size)
and substracting from it a fraction between 0/16 and 7/16 of the base
and subtracting from it a fraction between 0/16 and 7/16 of the base
size.@*
Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@*
Bits 7-5 contain the numerator of the fraction (0 to 7) to substract
Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract
from the base size to obtain the dictionary size.@*
Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
Valid values for dictionary size range from 4 KiB to 512 MiB.
@ -675,11 +812,11 @@ Example 1: Add a comment or description to a compressed file.
# First append the comment as trailing data to a lzip file
echo 'This file contains this and that' >> file.lz
# This command prints the comment to standard output
lziprecover --dump-tdata file.lz
lziprecover --dump=tdata file.lz
# This command outputs file.lz without the comment
lziprecover --strip-tdata file.lz
lziprecover --strip=tdata file.lz
# This command removes the comment from file.lz
lziprecover --remove-tdata file.lz
lziprecover --remove=tdata file.lz
@end example
@sp 1
@ -690,8 +827,8 @@ to guarantee that both file and hash have not been maliciously replaced).
@example
sha256sum < file.lz >> file.lz
lziprecover --strip-tdata file.lz | sha256sum -c \
<(lziprecover --dump-tdata file.lz)
lziprecover --strip=tdata file.lz | sha256sum -c \
<(lziprecover --dump=tdata file.lz)
@end example
@ -727,6 +864,10 @@ Don't do this
cat file1.lz file2.lz file3.lz | lziprecover -d
Do this instead
lziprecover -cd file1.lz file2.lz file3.lz
You may also concatenate the compressed files like this
lziprecover --strip=tdata file1.lz file2.lz file3.lz > file123.lz
Or keeping the trailing data of the last file like this
lziprecover --strip=damaged file1.lz file2.lz file3.lz > file123.lz
@end example
@sp 1
@ -777,13 +918,13 @@ lziprecover -tv rec*file.lz
Example 8: Recover a compressed backup from two copies on CD-ROM with
error-checked merging of copies.
@ifnothtml
(@xref{Top,GNU ddrescue manual,,ddrescue},
@xref{Top,GNU ddrescue manual,,ddrescue},
@end ifnothtml
@ifhtml
(See the
See the
@uref{http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html,,ddrescue manual}
@end ifhtml
for details about ddrescue).
for details about ddrescue.
@example
ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1
@ -862,15 +1003,15 @@ after the last byte of a member, producing a shorter but valid
compressed file. Except in this latter case, please, report any false
negative as a bug.
In order to compare the outputs, unzcrash needs a @samp{zcmp} program
able to understand the format being tested. For example the one provided
In order to compare the outputs, unzcrash needs a @samp{zcmp} program able
to understand the format being tested. For example the @samp{zcmp} provided
by @samp{zutils}.
@ifnothtml
@xref{Zcmp,,,zutils},
@xref{Zcmp,,,zutils}.
@end ifnothtml
@ifhtml
See
@uref{http://www.nongnu.org/zutils/manual/zutils_manual.html#Zcmp,,zcmp}
@uref{http://www.nongnu.org/zutils/manual/zutils_manual.html#Zcmp,,zcmp}.
@end ifhtml
The format for running unzcrash is:
@ -879,6 +1020,10 @@ The format for running unzcrash is:
unzcrash [@var{options}] 'lzip -t' @var{file}.lz
@end example
@noindent
@var{file}.lz must not contain errors and must be correctly decompressed
by the decompressor being tested for the comparisons to work.
unzcrash supports the following options:
@table @code
@ -889,6 +1034,7 @@ Print an informative help message describing the options and exit.
@item -V
@itemx --version
Print the version number of unzcrash on the standard output and exit.
This version number should be included in all bug reports.
@item -b @var{range}
@itemx --bits=@var{range}
@ -966,7 +1112,10 @@ Verbose mode.
@item -z
@itemx --zcmp=<command>
Set zcmp command name and options. Defaults to @code{zcmp}. Use
@code{--zcmp=false} to disable comparisons.
@code{--zcmp=false} to disable comparisons. If testing a decompressor
different from the one used by default by zcmp, it is needed to force
unzcrash and zcmp to use the same decompressor with a command like
@w{@code{unzcrash --zcmp='zcmp --lz=plzip' 'plzip -t' @var{file}.lz}}
@end table