Automatic USB backup

TOC

Here is a memo over automatic USB backup.

This is a follow-up of “Backup and snapshot”.

(This uses the latest batch and gather subcommand of bss)

udev(7) under “RUN” clearly states “This can only be used for very short-running foreground tasks.”.

We still see old tips such as Automatic backups with UDEV (2009), which abuse “RUN” in udev rule to perform backup. These tips shouldn’t be followed.

Under modern system managed by systemd, it seems we need to use its systemd service file with “wants” mechanism. (2015-2020)

Find out systemd service name of auto mounted USB disk

Let’s suppose my backup USB drive is (auto)mounted as /media/osamu/BKUP_USB. Then, I can use systemctl list-units -t mount|grep BKUP_USB to find systemd service name corresponding to /media/osamu/BKUP_USB. For example:

$ systemctl list-units -t mount|grep USB
  media-osamu-BKUP_USB.mount      loaded active mounted /media/osamu/BKUP_USB
$ systemctl status media-osamu-BKUP_USB.mount
● media-osamu-BKUP_USB.mount - /media/osamu/BKUP_USB
     Loaded: loaded (/proc/self/mountinfo)
     Active: active (mounted) since Wed 2024-01-03 20:32:41 JST; 14min ago
      Where: /media/osamu/BKUP_USB
       What: /dev/sda1

Backup upon each mount event

Create ~/.config/systemd/user/bss-BKUP_USB.service as:

[Unit]
Description=USB Disk backup
Requires=media-penguin-BKUP_USB.mount
After=media-penguin-BKUP_USB.mount

[Service]
ExecStart=bss --may --type usb batch BKUP_USB

[Install]
WantedBy=media-penguin-BKUP_USB.mount

Create ~/.config/bss/BKUP_USB

########################################################################
# make new backup copy (path are relative from $HOME)
# * source is a btrfs subvolume at ~/SRC_SUBVOL
#   * always ignore error from gather
# * destination is auto-mountable partition on a USB drive:
#   * formatted as btrfs and
#   * labeled as 'DST_LABEL'
########################################################################
bss_usb_backup () {
  # $1: SRC_SUBVOL
  # $2: DST_LABEL
  bss gather "$1" || true
  bss copy   "$1" "/media/$(id -un)/$2/$1" || $BSS_MAY
  bss snap        "/media/$(id -un)/$2/$1" || $BSS_MAY
  bss process     "/media/$(id -un)/$2/$1" || $BSS_MAY
}

# Backup to BKUP_USB (high frequency backup SSD device)
MSGID=$(notify-send -p "bss: BKUP_USB" "backup in progress ...")
bss_usb_backup Documents BKUP_USB
notify-send -r "$MSGID" "bss: BKUP_USB" "backup finished!"

# vim:se sw=2 ts=2 sts=2 et ai tw=78:

Please note use of notify-send in this script.

Then, activate this service unit as:

 $ systemctl --user enable bss-BKUP_USB.service

Somehow, I get the following unexpected spurious yellow warning.

Unit /home/penguin/.config/systemd/user/bss-BKUP_USB.service is added as a dependency to a non-existent unit media-penguin-BKUP_USB.mount.

Despite this warning, USB storage seems to be getting backup data when my workstation is powered up with USB drive plugged-in or when it get plugged-in.

bss gather ...

The bss gather subcommand was introduced as an alternative tool to enable secure remote backups when luksimg was dropped. In March 2024, bss gather changed its configuration file format. Now it uses “FILTER RULES” of rsync(1).

For gather configuration file .gather.rc, bss gather executes rsync with the following (excluding details):

 $ rsync -a --filter=". .gather.rc" $srcdir $destdir

The syntax of gather configuration file is defined in the “FILTER RULES” in rsync(1) manpage.

Although this gather configuration file can be written in any order, I usually organize this file in the following order.

• Partial path exclusion match rules to be forced
  • file name match rule (e.g. “- *~” without “/”)
  • dir name match rule (e.g., “- .git/” with only one “/” at the end)
• Full path inclusion match rules (pattern starts with “/”)
  • the parent3 dir inclusion match rule (e.g., “+ /a/” with “/” at the end)
  • the parent2 dir inclusion match rule (e.g., “+ /a/b/” with “/” at the end)
  • the parent1 dir inclusion match rule (e.g., “+ /a/b/c/” with “/” at the end)
  • the target inclusion match rule (no “/” at the end)
    • inclusion match rule for individual target file (e.g., “+ /a/b/c/targetfile”)
    • recursive inclusion match rule for files under target dir (e.g., “+ /a/b/c/targetdir/***”)
• Full path fall-back all-exclusion match rule (e.g., “- /***”)

For example, configuration for home directory looks like:

# Force exclude always for partial match (dir)
- .git/
- .bss.d/
# Force exclude always for partial match (file)
- *~
- core
# Pull path match for files in the base path
+ /.bashrc
+ /.devscripts
+ /.editorconfig
+ /.face
+ /.face.icon
+ /.gbp.conf
+ /.gitconfig
+ /.profile
+ /.pycodestyle
+ /.reportbugrc
+ /.sbuildrc
+ /.shellcheckrc
+ /.vimrc
# Pull path match for recursive include -- match parent dir path first
+ /.config/
+ /.config/autostart/***
+ /.config/black/***
+ /.config/bss/***
+ /.config/flake8/***
+ /.config/kitty/***
+ /.config/pylint/***
+ /.config/systemd/***
+ /.config/nvim/***
+ /.gnupg/***
+ /.ssh/***
+ /.local/
+ /.local/share/
+ /.local/share/keyrings/***
+ /.sbuildrc.d/***
+ /.bashrc.d/***
+ /bin/***
# Drop files and directory not included explicitly in the above
- /***

The use of *** (available for recent rsync) makes it easy to write rule.

The above execution is actually over-simplified. In reality, details are taken care to avoid gathering snapshot files under the .bss.d/ directory as:

 $ rsync -aHS --delete-excluded --mkpath --filter="- .bss.d/" \
              --filter=". .gather.rc" $srcdir $destdir

For more, see https://github.com/osamuaoki/bss