timedicer-mirror v6.56.3 [15 Feb 2024] by Dominic

Description

timedicer-mirror.sh mirrors (synchronises) critical data from a local (source) machine to a remote (destination) machine using rsync and other tools. It is part of the TimeDicer suite (https://www.timedicer.co.uk); its purpose is to update the rdiff-backup repositories, and other TimeDicer-specific settings, of a Mirror TimeDicer to be the same as those on a Primary TimeDicer Server. It is always run on the sending (i.e. source) machine.

On the remote (destination) machine timedicer-mirror.sh creates any missing Linux users (and their associated groups) that are local TimeDicer users on the source machine. It also updates the remote's Rdiffweb users and the directories /opt and /home to match those on the source. (It does not mirror the operating system.) Copies of /etc/crontab and /etc/rc.local, with filenames suffixed by a dot and the machine's BaseID (see below), are placed in source machine's /opt before it is mirrored to remote /opt.

The drawn-out part of the task is mirroring /home. timedicer-mirror.sh ensures that all new /home data is successfully transferred to the remote machine before any of the old data there is replaced. Only after successful transfer of all new data is the old /home data on the remote machine replaced with the new. Failed sessions, say caused by internet connection problems, may delay the updating of the remote machine but should not corrupt its data.

Tested on recent Ubuntu Server versions up to 22.04LTS, and designed for use as a nightly cron job with TimeDicer Server.

Usage

timedicer-mirror.sh [options] dest_address

Example

sudo /opt/timedicer-mirror.sh -v 192.168.100.130

Options

-a: active progress mode - shows file transfer progress, implies verbose
-c: determine if file backup is needed by checksum rather than comparing file date, time & size, and also compare usage of /home on source and destination (much slower)
-e: force no snapshot - backup directly from /home not from a snapshot created using btrfs or LVM, even if such is possible
-f: fast mode - don't check space on destination machine and don't backup /home (see -o)
-h: show help and exit
-i: check space on destination machine as normal, but ignore results (see also -o)
-k num: specify remote WOL UDP port 'num' (default 9; see also -m; requires wakeonlan)
-l: show changelog and exit
-m macid: specify MAC address of destination machine - if asleep a WOL 'magic packet' will be sent to wake it (see also -k; requires wakeonlan)
-n: poweroff destination machine at end ('night night')
-o: don't check space on destination machine - faster (see also -i, -f)
-p num: specify destination SSH TCP port 'num' (default 22)
-q: quiet mode - no output unless there is a problem
-r: restart i.e. stop any prior instance, then continue
-s: stop any prior running instance of this program
-t: test mode - do not make changes on destination (see also -f), repeat -t to run rsync with --dry-run rather than skipping altogether
-u: use pre-existing mounted source snapshot (e.g. as left behind after a previous incomplete run of timedicer-mirror.sh)
-v: verbose mode
-x "uids": excludes the home directories of the space-separated list of user ids from backup to destination
-z: use compression when transferring files (rsync -zz is used if rsync version >=3.1.1)

Details

Before timedicer-mirror.sh can work, you must have added the root public key of source machine (/root/.ssh/id_rsa.pub) to destination machine in file /root/.ssh/authorized_keys.

If a prior instance of timedicer-mirror.sh is already running, a new instance will abort unless called with -r.

If source /home is on a btrfs or LVM volume, timedicer-mirror.sh will create and use a temporary snapshot of such volume for the operation - this can be overridden with -e option.

Further information about how to use timedicer-mirror.sh can be found at https://www.timedicer.co.uk/index.php#Mirror_Server_Setup.

TimeDicer BaseID

Every Timedicer Server machine has a BaseID - a single digit which is normally 1. This default value is overridden based on the last character (digit) in file /opt/baseid or, if there is no such file, the last character (digit) of the machine's hostname. timedicer-mirror.sh will only backup /home subdirectories for users with uids in the range (1000xBaseID)+1 to (1000xBaseID)+999 - default 1001-1999 - these are considered the 'local TimeDicer users'.

Users created through the TimeDicer web interface will have uids and gids which conform to this specification, and it is required for mutual mirroring (bi-directional) or if you run a TimeDicer Server Pool. If you are setting up more than one TimeDicer Server and you might one day want to extend this to have more than two such servers - or where the second machine might have its own local TimeDicer users and you might want to run timedicer-mirror.sh back from there to the first, as well as vice-versa - you should ensure that they have different BaseIDs e.g. by naming the first 'timedicer1', the second 'timedicer2', the third 'timedicer3' etc.

Warning

Because of the major changes which it makes to the destination machine, you should not run timedicer-mirror.sh to a machine which has any other purpose than being a TimeDicer Server, either as a mirror for your source machine or with its own local TimeDicer users in which case it must be configured with a different BaseID to the source machine (see above). timedicer-mirror.sh may alter users and groups on the destination machine as well as overwriting some /home subdirectories and /opt. You have been warned!

Dependencies

[m/g]awk bash coreutils diffutils grep net-tools(hostname) iputils(ping) openssh ps rsync sed util-linux [wakeonlan]

License

Copyright © 2024 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

6.56 [03 Feb 2024] - repeating -t (test mode) causes it to do rsync dry-run, not skip altogether
6.55 [22 Dec 2023] - change method of calculating snapshot LV size, limit to 10G max (initially)
6.54 [25 Jul 2023] - exclude source /home/backup and /home/lost+found from being mirrored
6.53 [22 Jul 2023] - bugfix for updating rdiffweb database users on destination
6.52 [13 Feb 2022] - save progress details to /var/tmp/timedicer-mirror.sh_rsync.txt, show date on main rsync run
6.51 [06 Dec 2021] - minor changes to method of calculating/comparing used space in /home
6.50 [03 Jun 2021] - internal bugfix (RSYNCADDOPTIONS now as array)
6.47 [07 Feb 2020] - exclude any swap files located on /home from being mirrored
6.46 [04 Mar 2019] - fix error retrieving prior contents of destination /home
6.45 [04 Nov 2018] - fix wrong diagnosis of rsync failure
6.44 [26 Dec 2017] - increase ssh timeout/retry settings
6.43 [02 Dec 2017] - use more reliable way to determine primary IPv4 address (same as timedicer-server-setup.sh)
6.42 [11 Aug 2017] - set temporary backup folder on destination (/home/backup) to be accessible only by root
6.41 [28 May 2017] - fix a bug causing failure if Mirror Server is a new 16.04 installation
6.4 [17 May 2017] - minor bugfix to prevent calling apt-get unnecessarily
6.3 [20 Apr 2017] - add -x option, some other fixes
6.2 [23 Nov 2016] - install sqlite3 if required and missing
6.1 [16 Nov 2016] - add error handling on retrieving destination users/groups
6.0825 [25 Aug 2016] - allow more time to umount snapshot; update help text
6.0808 [08 Aug 2016] - small tweaks
6.0525 [25 May 2016] - small text and code tweaks removing source LVM snapshot
6.0430 [30 Apr 2016] - few small code tweaks
6.0427 [27 Apr 2016] - make source/destination user-checking compatible with TimeDicer Server Pool concept, remove redundant -b option
6.0426 [26 Apr 2016] - update destination rdiffweb database in consistent way for TimeDicer Server Pool (kudos: Grant Emsley), also allocate 80% of free VG space for snapshot instead of fixed 4 Gibibytes
6.0425 [25 Apr 2016] - fix timeout error creating snapshot ('special device does not exist')
6.0424 [24 Apr 2016] - really fix fatal error (since 6.0422)
6.0423 [23 Apr 2016] - bugfix fatal error
6.0422 [22 Apr 2016] - add -b switch, only mirror data for local timedicer users based upon the source machine's BaseID
6.0115 [15 Jan 2016] - if rsync operation fails, check whether source snapshot has failed and abort if it has
5.1226 [26 Dec 2015] - bugfix copying of /etc/rdiffweb/rdw.db if destination directory doesn't exist
5.1210 [10 Dec 2015] - bugfix naming of LV (kudos bug report: Alex Racov)
5.1123 [23 Nov 2015] - increase timeout from 2 mins to 30 mins, use new rsync compression (-zz), use --append-verify (later removed)
5.0922 [22 Sep 2015] - fix bug identifying LV
5.0908 [08 Sep 2015] - updated help info
5.0813 [13 Aug 2015] - do not delete extraneous files from destination's /opt
5.0513 [13 May 2015] - minor improvement to LV identification
5.0331 [31 Mar 2015] - improved stop/restart code
5.0314 [14 Mar 2015] - speed optimisation (combined some remote ssh calls)
5.0313 [13 Mar 2015] - make compatible with btrfs and/or non-LVM-mapped filesystems, add -e (force-no-snapshot) option
5.0217 [17 Feb 2015] - minor tweaks
5.0130 [30 Jan 2015] - minor text changes and code tidying
5.0127 [27 Jan 2015] - minor text changes and bugfix
5.0123 [23 Jan 2015] - minor text changes and code tidying
5.0119 [19 Jan 2015] - move mac address to new -m option
5.0118 [18 Jan 2015] - fix bugs in file exclusions
5.0113 [13 Jan 2015] - code tidying
4.1230 [30 Dec 2014] - abort on error when obtaining destination backup space
4.1224 [24 Dec 2014] - remove compression if retrying after timeout (rsync bug 7757)
4.1220 [20 Dec 2014] - add -u option, change -n option to put destination to sleep, add 120s timeout to rsync, fix a bug where it failed to detect rsync failure in verbose or active progress modes
4.1211 [11 Dec 2014] - faster -f fast mode
4.1121 [21 Nov 2014] - add -i option and change -o option
4.0822 [22 Aug 2014] - improved/bugfixed comparing source and destination users
4.0814 [14 Aug 2014] - more bugfixes for large source and destination sizes
4.0813 [13 Aug 2014] - bugfixes for root LVM and large destination size, allow MAC addresses with dashes
4.0808 [08 Aug 2014] - test destination /var/lib/sudo/timedicer/0 is accessible
4.0730 [30 Jul 2014] - bugfix counting space used by destination /home/backup
4.0409 [09 Apr 2014] - resolve destination address explicitly (uses ping)
4.0324 [24 Mar 2014] - bugfix comparing users on source and destination
4.0123 [23 Jan 2014] - use specified SSH port for rsync too
4.0115 [15 Jan 2014] - re-open remote machine if powered off during -c checking
3.1011 [10 Oct 2013] - quit if any ongoing rdiff-backup verification processes
3.0918 [18 Sep 2013] - minor text changes
3.0916 [16 Sep 2013] - bugfix calculation of destination /home/backup space
3.0828 [28 Aug 2013] - test that destination /home is writeable
3.0803 [03 Aug 2013] - bugfix calculation of destination space
3.0530 [30 May 2013] - minor text changes
3.0523 [23 May 2013] - remove dg834g_src option
3.0514 [14 May 2013] - correct help text description of -o option
2.1117 [17 Nov 2012] - change option -o to check for enough space on destination instead of *not* checking (default behaviour is now *not* to check)
2.1107 [07 Nov 2012] - fix bug preventing creation of users and groups on destination
2.0929 [29 Sep 2012] - remove erroneous fail message on local ping
2.0821 [21 Aug 2012] - local ping test informational instead of critical, use /tmp instead of /var/tmp, deprecate dg834g_src, change previous -P (ssh port) switch to -p and previous -p switch ('live progress') to -a
2.0818 [18 Aug 2012] - add -k option to specify remote wakeonlan port
2.0721 [21 Jul 2012] - add -P option to specify remote SSH port, remove warnings about RSA host key not in list of known hosts
2.0715 [15 Jul 2012] - add -o option to skip remote machine space checking
2.0627 [27 Jun 2012] - fix for machine with hyphen in name and re-enable backup of /opt (n/w since 2.0317)
2.0607 [07 Jun 2012] - allow concurrent rdiff-backup verify/restore operations
2.0514 [14 May 2012] - improved help text
2.0412 [12 Apr 2012] - add code (for checksum mode) comparing usage of /home
2.0404 [04 Apr 2012] - further minor changes and bugfixes
2.0330 [30 Mar 2012] - bug fix for recording size of files
2.0325 [25 Mar 2012] - add recording of size of files transferred so that on subsequent runs the required space on destination can be better estimated
2.0317 [17 Mar 2012] - add explicit -z compression option, transfers no longer compressed by default
2.0313 [13 Mar 2012] - add -l 'changelog' option, change previous -l (i.e. 'live') option to be -p (i.e. 'progress')
2.0308 [08 Mar 2012] - renamed program from 'timedicer-mirror' to 'timedicer-mirror.sh', improved help
1.1023 [23 Oct 2011] - exclude /home/tmp/*
1.1012a [12 Oct 2011] - copy .ssh and other hidden folders in user directories on fast copy (disabled in 4.1211)
1.1011 [11 Oct 2011] - further fixes and variable name changes
1.1010 [10 Oct 2011] - further fixes for base filesystem search
1.1009 [09 Oct 2011] - modify search for base filesystem
1.0818 [18 Aug 2011] - remove temporary files on successful completion

Download timedicer-mirror.sh

Donation

I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!

My Other Sites

TimeDicer - Onsite/offsite data backup for Windows (uses rdiff-backup)
Finding a 4D Backup Solution
Web Scraping How To - extracting data from web sites

My Programs

Here is a selection of some (other) programs I have written, most of which run under GNU/Linux from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written and tested on an x86-based Linux server, they should run on a Raspberry Pi, and many can run under Windows using Windows Subsystem for Linux (WSL) or Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

rdiff-backup-regress - GNU/Linux script to regress an rdiff-backup archive. [ GNU/Linux: 2012-24 ]

Debian/Ubuntu kernel and LVM Utilities

kernel-remove - GNU/Linux script to list the installed GNU/Linux kernels in a Debian-based distro (e.g. Ubuntu), and can be used to remove an unwanted kernel and related packages, updating grub appropriately. [ GNU/Linux-Debian/Ubuntu: 2010-24 ]
lvm-usage - GNU/Linux script to show available disk space and how it is used; run as cron job to warn if usage is above a set percentage. Provides additional information if LVM is in use. [ GNU/Linux: 2012-24 ]
lvm-delete-snapshot - GNU/Linux script to remove LVM snapshot that has been left over by another process. [ GNU/Linux: 2012-21 ]
netnames - GNU/Linux script shows current name, biosdevname and 'predictable name' of network device - helps with network device name scheme migration. [ GNU/Linux-Debian/Ubuntu: 2020-20 ]
lv-convert2cache - GNU/Linux script to convert an existing LV into a cache LV using a smaller faster device as a cache. [ GNU/Linux: 2022-24 ]

Miscellaneous Programs

sleepwalker - Windows© program which can be run from a remote machine to 'wake up' a Windows© machine behind a router, wait for it to start and then initiate Remote Desktop session. [MS Windows©: 2008-22]
numliststat - GNU/Linux program giving statistical value(s) for a piped-in list of numbers. [ GNU/Linux: 2022-24 ]
relay-enforcer - GNU/Linux program enabling a postfix-based mail server relaying to Gmail to act on reports from Gmail about blocked emails. [ GNU/Linux: 2016-24 ]
pdf-compress - GNU/Linux program to create smaller b/w pdf file from an original large pdf file, especially when original resulted from scanning. [ GNU/Linux: 2016-23 ]
tiny-device-monitor - GNU/Linux program to test webpages (including password-protected) or machines to check they are live; use as a cron job for your own websites, for hardware presenting a webpage, or for any machines with a presence on your local LAN or on the internet. [ GNU/Linux: 2009-24 ]
form-extractor - GNU/Linux program to extract form tags from a web page or downloaded file. [ GNU/Linux: 2012-24 ]
mythic-dns-sync - GNU/Linux program to update DNS record at mythic-beasts.com to match local external ip. [ GNU/Linux: 2016-23 ]
saynoto0870 - For UK, a GNU/Linux script which performs automated lookup of the www.saynoto0870.com database, finding cheap or free geographic number replacements for expensive non-geographic (087* or 084*) numbers. [ GNU/Linux: 2012-12 ]
bind9-resolved-switch - GNU/Linux program for switching permanently between using bind9 or systemd-resolved as the system DNS resolver. [ GNU/Linux: 2016-24 ]
unlock - GNU/Linux remote program for easy entering of decrypt passphrase on a remote machine which has root dm-crypt+LUKS. [ GNU/Linux: 2017-18 ]
wifi-updown - GNU/Linux program to take down wifi interface if there is a working wired interface (or restore wifi if not). [ GNU/Linux: 2018-23 ]
routefix - GNU/Linux program to restore a default ip traffic route if there is none such (e.g. after running wifi-updown). [ GNU/Linux: 2018-23 ]
dutree - GNU/Linux program to show a tree-style list of files and directories at the specified location which are greater than the specified size (default 1GB). [ GNU/Linux: 2012-24 ]
Rents Program - Residential lettings/landlord front office program, with many special features for UK market [MS Windows©: 1991-2024]

Comments

This section is closed. If you have a question, please submit it by email, thank you.

No comments yet for '/programs/help/timedicer-mirror.sh.php'