menuimage

Description

Usage

Options

Why_timedicer-verify

Dependencies

License

Changelog

Download

Donation

My_Other_Sites

My_Programs

Comments

timedicer-verify v2.82 [14 Mar 2024] by Dominic

Description

Verifies integrity of rdiff-backup repositories at /home/*/[here] and /home/*/*/[here]. Unless options -f or -d are used it just verifies the most recent session in each repository (this session is held in the clear in the repository). With -f or -d options it may take a long time!

If run by root (or with sudo) timedicer-verify.sh checks repositories for all users (as restricted by -u, -n and -x options), otherwise it checks only the current user's repositories. A non-zero exit code is returned if one or more problems were found.

Each successful verification for a repository session is saved in that repository's rdiff-backup-data/verified.log file; on a subsequent run that session is skipped for verification unless -r option is used. This file therefore provides a linear record of successful verifications in the form [datetime-of-original-session],[datetime-when-verified],[rdiff-backup exit code].

You can override problem detection (if you have a known reported problem which is not in fact such or you do not consider fatal) by adding the error text to /opt/timedicer-verify.sh-ignore.txt, any whole line matches for such text will be ignored on future runs. Lines in this ignore file where the first non-whitespace is a hash (#) are disregarded i.e. will not trigger an ignoral and can therefore be used for comments.

Normally timedicer-verify.sh will exit immediately if rdiff-backup reports an error (unless the error text is matched in /opt/timedicer-verify.sh-ignore.txt), but you can override this behaviour with -i option.

Also with -b option (deprecated) it fixes ownership of restore.log files (which Rdiffweb may have created and set to root ownership, thus causing warning errors upon user trying to retrieve files).

timedicer-verify.sh is part of the TimeDicer Server software suite.

Usage

timedicer-verify.sh [options]

Options

-a [datetime] - stop (abort) verifications at datetime (with exit code 0) *
-b - fix ownership of any Rdiffweb-created restore.log files
-d [datetime] - verify backup sessions back to session on, or next before, datetime *
-e - debug mode (unexpected things may happen)
-f - verify all sessions in each repository (slowest)
-h - show this help and quit
-i - continue after rdiff-backup reports an error (aka 'ignore')
-l - show changelog and quit
-m - create and use lvm snapshot of source data (discarded at end)
-n [name] - check only repositories containing text 'name' (egrep-type regex) in their pathname (cf. -x)
-q - quiet (text output only on error)
-r - don't skip any repository sessions previously verified as ok
-s - list backup sessions and date/time of the corresponding previous verification session (if any), then quit
-t [tmpdir] - use tmpdir for temporary storage
-u [user] - only for named user (ignored unless run by root)
-v - verbose output
-x [name] - skip any repositories containing text 'name' (egrep-type regex) in their pathname (cf. -n)
-z [name] - process any repositories with path containing text 'name' (sed-type regex) last
-1 - verify only the earliest session for each repository that falls within the specification (cf. -4,-5)
-2 - skip verification of the most recent session for each repository
-3 - skip any verifications that are in progress elsewhere, normally in a different session on the same machine, but with an additional script can also detect sessions in progress on another (i.e. mirror) machine
-4 - skip all verifications that would be performed for each repository except the earliest and instead mark them in that repo's verified.log as having been performed but with a special 'fake' flag. Sessions thus marked will be skipped on any subsequent run of timedicer-verify.sh for this repository unless -6 or -r is specified. In effect this is the same as running timedicer-verify.sh -1, or running rdiff-backup --verify for the earliest verification date, but it affects future runs of timedicer-verify.sh. Marking of fake sessions in verified.log only occurs if and after successful verification of the earliest session, or confirmation of a previous successful verification of the earliest session.
-5 [datetime] - as -4 but perform an actual verification for the earliest session even if it was previously successfully verified but such verification was performed before [datetime] *
-6 - don't skip over 'fake' former verifications (see -4,-5)

* datetime can be but does not have to be in the format used by rdiff-backup; it just needs to be understood by the 'date' command e.g. yesterday, "3 July 2022", "2 weeks ago", "2022-05-03 14:26". See 'man date' for more details.

Why timedicer-verify

rdiff-backup provides the 'verify --at' (formerly '--verify-at-time') option to verify a single repository session. However this does not record the successful verification nor, more importantly, does it verify any intermediate sessions. For instance, a file that did *not* exist at the time of the earlier session, *did* exist at the time of an intermediate session, but has subsequently been deleted, could prove irrecoverable even when the earlier session had been verified. You cannot even be entirely confident that a later version of a file that has been verified at an earlier date is OK (although it is highly likely).

The only way to be confident about all repository sessions is to use rdiff-backup's 'verify --at' option to verify each session individually. This is what timedicer-verify.sh accomplishes, and although it may be a slow process, by keeping a record of successful previous verifications and not (unless required by -r or -5 options) rechecking them, it becomes manageable. A suggested use is as a weekly cron job with -d "one month ago" option (sessions earlier than a week ago will normally be skipped automatically because they were verified previously).

Dependencies

awk bash column coreutils diff findutils grep lvm(optional) lvm-delete-snapshot.sh(optional) rdiff-backup sed

License

Copyright © 2024 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

2.82 [14 Mar 2024] - bug fixes
2.81 [08 Mar 2024] - ensure non-breaking report lines from rdiff-backup, add -w option, deprecate -b option
2.80 [21 Feb 2024] - better shellcheck compliance, compatibility added for rdiff-backup v2.2, removed for v1.x and earlier
2.71 [01 Aug 2022] - allow -n and -x options to be combined
2.70 [14 Apr 2022] - further tweaks, try verification again in case of segmentation fault
2.69 [07 Apr 2022] - further tweaks
2.68 [31 Mar 2022] - more bugfixing, remove concurrency option (as it was not reliable)
2.67 [10 Mar 2022] - more concurrency bugfixing (i.e. -c option), was not spotting when a verification failed, mark concurrency as alpha status
2.66 [25 Jan 2022] - concurrency bugfixing (i.e. -c option)
2.65 [29 Jul 2020] - fix for identifying existing instances of timedicer-verify.sh
2.64 [26 Jul 2020] - fix for identification of rdiff-backup version and of some repos latest date
2.63 [05 Jul 2020] - small fix for verification by v2+ of pre-v2 DOS repositories
2.62 [24 Jun 2020] - don't attempt to patch compare.py for rdiff-backup v2+, but warn for v2.0.0
2.61 [03 Jan 2019] - small tweak to help text
2.60 [24 May 2017] - serious bugfix - was not patching rdiff_backup/compare.py
2.54 [29 Mar 2016] - serious bugfix - v2.25-2.53 were failing to detect many verification failures
2.53 [25 Feb 2016] - bugfix unnecessary re-verification of single session repos with -d
2.52 [12 Feb 2016] - bugfix -5 option
2.51 [29 Jan 2016] - add -5 option (renumber previous -5 as -6)
2.50 [13 Jan 2016] - add -4 and -5 options ('fake' verifications)
2.49 [17 Dec 2015] - add automatic patching of rdiff_backup/compare.py
2.48 [16 Dec 2015] - bugfix -z option
2.47 [15 Dec 2015] - add -z option
2.46 [15 Dec 2015] - bugfix -3 option
2.45 [14 Dec 2015] - add -3 option
2.44 [12 Dec 2015] - bugfix -a option
2.43 [09 Dec 2015] - add -a option
2.42 [30 Nov 2015] - bugfix for error detection, depends on modified compare.py, bugfix for -1 option
2.41 [19 Nov 2015] - bugfix for -1 option
2.40 [16 Nov 2015] - treat non-zero exitcode <> 2 from rdiff-backup as fatal (fix bug whereby killing/aborting process - exit code 1 - set a success 'flag' for the, in fact incomplete, verification action)
2.30 [08 Nov 2015] - add -1 option
2.26 [29 Oct 2015] - default concurrent sessions = 1
2.25 [21 Oct 2015] - add -e debug mode, bugfix use of /opt/timedicer-verify.sh-ignore.txt
2.24 [04 Oct 2015] - abort if an instance of timedicer-verify.sh is already running as a different process
2.23 [14 Jan 2015] - don't touch verified.log file(s) unless need to be updated
2.22 [08 Dec 2014] - bugfix, was not using exit code >0 when error found in repository (except the last)
2.21 [11 Oct 2014] - bugfix, was not counting errors that occurred with rdiff-backup --list-increments
2.20 [27 Mar 2014] - use rdiff-backup --list-increments to obtain list - avoid checking null increments
2.15 [10 Oct 2013] - tweak snapshot settings to allow concurrent runs of timedicer-verify.sh
2.14 [09 Oct 2013] - changes to text output, reassign -o option to -x
2.13 [19 Sep 2013] - add (-o ->) -x option
2.12 [19 Feb 2013] - web help layout fix, reassign -w to -b
2.10 [09 Jul 2012] - first public release
2.04 [10 Jun 2012] - improved concurrency
2.03 [09 Jun 2012] - add -c, -m, -n, -t options
2.02 [06 Jun 2012] - bugfixes
2.0105 [05 Jan 2012] - hide lvremove non-error text on removing snapshot
1.1230 [30 Dec 2011] - (create and) use /home/tmp as TMPDIR, add -l option to show this changelog
1.0818 [18 Aug 2011] - add use of LVM snapshot if available (for root only), add -u and -d options
1.0817 [17 Aug 2011] - add non-root usage, add -s option

Download timedicer-verify.sh

Donation

I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!

My Other Sites

My Programs

Here is a selection of some (other) programs I have written, most of which run under GNU/Linux from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written and tested on an x86-based Linux server, they should run on a Raspberry Pi, and many can run under Windows using Windows Subsystem for Linux (WSL) or Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

Debian/Ubuntu kernel and LVM Utilities

Miscellaneous Programs

Comments

This section is closed. If you have a question, please submit it by email, thank you.

No comments yet for '/programs/help/timedicer-verify.sh.php'