14 KiB
btrbk
- https://digint.ch/btrbk/doc/readme.html
- https://digint.ch/btrbk/doc/btrbk.1.html
- https://digint.ch/btrbk/doc/btrbk.conf.5.html
btrbk - backup tool for btrfs subvolumes
- Take snapshots (on same disk)
- Create multiple backups (on different disks) over SSH
Further resources:
Snapshots and Backups
subvolume:
snapshot: .[]
timestamp: Can be configured using the timestamp_format option.
N: If multiple snapshots are created at the same time, the increasing counter N differentiates them.
Actions and Informative Commands
run [filter...]
In multiple steps, perform snapshots and backups.
- Read Data: Perform sanity checks
- Create Snapshots
- Create Backups
- Delete Backups
- Delete Snapshots
clean [filter...]
Delete incomplete backups (network error, kill signal while send/receive was ongoing).
stats [filter...]
Print statistics of snapshot and backup subvolumes.
sudo btrbk -c /etc/btrbk/local-backup-hdd.conf stats
SOURCE_SUBVOLUME SNAPSHOT_SUBVOLUME TARGET_SUBVOLUME SNAPSHOTS BACKUPS
/mnt/data/jc-data/cloud.fykml.de /mnt/data/snap2/cloud.fykml.de.* /mnt/backup/snap2/cloud.fykml.de.* 77 17
/mnt/data/jc-data/cloud.privacy1st.de /mnt/data/snap2/cloud.privacy1st.de.* /mnt/backup/snap2/cloud.privacy1st.de.* 77 17
Total:
154 snapshots
34 backups (34 correlated)
diff
List the modified files since generation (transid) of subvolume in subvolume .
Columns:
SIZE file was modified for a total of SIZE bytes
COUNT file was modified in COUNT generations
FLAGS "+" file accessed at offset 0 (at least once)
"c" COMPRESS flag is set (at least once)
"i" INLINE flag is set (at least once)
sudo btrbk -c /etc/btrbk/local-backup-hdd.conf diff cloud.privacy1st.de.20240222T2300 cloud.privacy1st.de.20240223T1400
FLAGS COUNT SIZE FILE
+c. 1 2.36 MiB data/appdata_oc2uzw0ocw65/appstore/apps.json
+.. 1 12.00 KiB data/appdata_oc2uzw0ocw65/preview/a/3/f/c/b/b/7/5105712/256-256-crop.jpg
+.. 1 24.00 KiB data/appdata_oc2uzw0ocw65/preview/a/3/f/c/b/b/7/5105712/384-512-max.jpg
+.. 1 8.00 KiB data/appdata_oc2uzw0ocw65/preview/d/8/4/f/d/0/7/5105717/256-256-crop.jpg
+.. 1 20.00 KiB data/appdata_oc2uzw0ocw65/preview/d/8/4/f/d/0/7/5105717/384-512-max.jpg
+c. 1 208.00 KiB data/appdata_oc2uzw0ocw65/suspicious_login/models/527
... 18 324.00 KiB data/nextcloud.log
... 257 40.21 MiB db/binlog.000119
... 290 48.11 MiB db/ib_logfile0
+c. 5 5.59 MiB db/ibdata1
.c. 4 112.00 KiB db/mysql/innodb_index_stats.ibd
.c. 1 32.00 KiB db/mysql/innodb_table_stats.ibd
+.. 3 264.00 KiB db/nextcloud/oc_activity.ibd
.c. 1 32.00 KiB db/nextcloud/oc_activity_mq.ibd
... 2 16.00 KiB db/nextcloud/oc_appconfig.ibd
+.. 3 88.00 KiB db/nextcloud/oc_authtoken.ibd
... 4 40.00 KiB db/nextcloud/oc_calendar_reminders.ibd
... 3 40.00 KiB db/nextcloud/oc_calendarchanges.ibd
+.. 6 184.00 KiB db/nextcloud/oc_calendarobjects.ibd
+.. 4 232.00 KiB db/nextcloud/oc_calendarobjects_props.ibd
.c. 1 8.00 KiB db/nextcloud/oc_calendarsubscriptions.ibd
... 8 1.18 MiB db/nextcloud/oc_filecache.ibd
... 2 48.00 KiB db/nextcloud/oc_filecache_extended.ibd
... 1 32.00 KiB db/nextcloud/oc_gpodder_episode_action.ibd
... 2 48.00 KiB db/nextcloud/oc_jobs.ibd
+c. 1 48.00 KiB db/nextcloud/oc_login_address.ibd
... 3 64.00 KiB db/nextcloud/oc_login_ips_aggregated.ibd
... 1 40.00 KiB db/nextcloud/oc_maps_photos.ibd
... 1 24.00 KiB db/nextcloud/oc_news_feeds.ibd
+.. 9 4.98 MiB db/nextcloud/oc_news_items.ibd
... 4 64.00 KiB db/nextcloud/oc_notifications.ibd
.c. 2 24.00 KiB db/nextcloud/oc_notifications_settings.ibd
... 1 16.00 KiB db/nextcloud/oc_preferences.ibd
... 1 16.00 KiB db/nextcloud/oc_preview_generation.ibd
... 1 16.00 KiB db/nextcloud/oc_suspicious_login_model.ibd
Total size: 104.47 MiB
Cleanup
First, perform a dryrun. Double-check the output. Then, replace dryrun with run.
Adjustments: Change 2d to something else or keep only latest by setting it to no.
sudo btrbk -c /etc/btrbk/local-backup-ssd.conf \
--override=snapshot_preserve=2d \
--override=snapshot_preserve_min=latest \
--override=snapshot_create=no \
--override=target_preserve=2d \
--override=target_preserve_min=latest \
--print-schedule \
dryrun
SNAPSHOT SCHEDULE
-----------------
ACTION SUBVOLUME SCHEME REASON
delete /mnt/data/snap2/cloud.privacy1st.de.20230927T1827 2d (sunday, 00:00) -
...
delete /mnt/data/snap2/cloud.privacy1st.de.20240220T0000 2d (sunday, 00:00) -
- /mnt/data/snap2/cloud.privacy1st.de.20240221T0000 2d (sunday, 00:00) preserve daily: first of day, 2 days ago, at 00:00
delete /mnt/data/snap2/cloud.privacy1st.de.20240221T0100 2d (sunday, 00:00) -
...
delete /mnt/data/snap2/cloud.privacy1st.de.20240221T2300 2d (sunday, 00:00) -
- /mnt/data/snap2/cloud.privacy1st.de.20240222T0000 2d (sunday, 00:00) preserve daily: first of day, 1 days ago, at 00:00
delete /mnt/data/snap2/cloud.privacy1st.de.20240222T0100 2d (sunday, 00:00) -
...
delete /mnt/data/snap2/cloud.privacy1st.de.20240222T2300 2d (sunday, 00:00) -
- /mnt/data/snap2/cloud.privacy1st.de.20240223T0000 2d (sunday, 00:00) preserve daily: first of day, 0 days ago, at 00:00
delete /mnt/data/snap2/cloud.privacy1st.de.20240223T0100 2d (sunday, 00:00) -
...
delete /mnt/data/snap2/cloud.privacy1st.de.20240223T1500 2d (sunday, 00:00) -
- /mnt/data/snap2/cloud.privacy1st.de.20240223T1600 2d (sunday, 00:00) preserve min: latest
BACKUP SCHEDULE
---------------
ACTION SUBVOLUME SCHEME REASON
delete /mnt/backup/snap2/cloud.privacy1st.de.20230927T1827 2d (sunday, 00:00) -
...
delete /mnt/backup/snap2/cloud.privacy1st.de.20240220T0000 2d (sunday, 00:00) -
- /mnt/backup/snap2/cloud.privacy1st.de.20240221T0000 2d (sunday, 00:00) preserve daily: first of day, 2 days ago, at 00:00
- /mnt/backup/snap2/cloud.privacy1st.de.20240222T0000 2d (sunday, 00:00) preserve daily: first of day, 1 days ago, at 00:00
- /mnt/backup/snap2/cloud.privacy1st.de.20240223T0000 2d (sunday, 00:00) preserve daily: first of day, 0 days ago, at 00:00
- /mnt/backup/snap2/cloud.privacy1st.de.20240223T1600 2d (sunday, 00:00) preserve min: latest
Configuration
General
volume <volume-directory|<ulr>: Usually the mount point of a btrfs filesystem mounted with subvolid=5.
subvolume <subvolume-name|<url>: Subvolume to be backed up, relative to <volume-directory>.
snapshot_dir <directory>: Directory in which the btrfs snapshots are created, relative to <volume-directory>. Hast to be created manually!
target <target-directory>|<url>: Target directory where the backup subvolumes are to be created. Multiple target sections are allowed.
url: Accepted formats are:
ssh://<hostname>[:<port>]/<directory><hostname>:<directory>
hostname: Either a host name, IPv4 or IPv6 address
timestamp_format short|long|long-iso: This becomes relevant for setups with multiple btrbk instances, e.g. many snapshot-only instances (spread around the world), and a fetch-only instance on the backup server. Make sure to run btrbk with the same time zone on every host, e.g. by setting the TZ environment variable (see tzset(3)).
Retention
snapshot_preserve <retention_policy>
snapshot_preserve_min all|latest|<number>{h,d,w,m,y}: Preserve all snapshots for a minimum amount of hours (h), days (d), weeks (w), months (m) or years (y).
target_preserve <retention_policy>
target_preserve_min all|latest|no|<number>{h,d,w,m,y}: Preserve all snapshots for a minimum amount of hours (h), days (d), weeks (w), months (m) or years (y). If set to "no", only the backups following the target_preserve policy are created.
<retention_policy>: [<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]. Defines how many backups of each period should be preserved.
SSH and Data Stream Options
ssh_compression yes|no
stream_compress <compress_command>: Compress the btrfs send stream before transferring it from/to remote locations.
<compress_command>: gzip, pigz, bzip2, pbzip2, bzip3, xz, lzo, lz4, zstd.
stream_buffer <size>|no: This can give a speed improvement (measured up to 20%) on both local or remote operations, but also increases system load. Make sure that the mbuffer command is available. Leave this option disabled if your main concern is a stable backup process: while recent versions of mbuffer have proven reliable, it is often desirable to keep things simple rather than adding an additional, multi-threaded process to the command pipe.
rate_limit <rate>|no: Limit the read rate of the btrfs send stream to <rate> bytes per second. Implies the stream_buffer option.
System Options
transaction_syslog <facility>|no: Log all transactions to syslog.
lockfile <file>|no
Example: Local Snapshots and Backup
timestamp_format long
snapshot_preserve_min 2d
snapshot_preserve 24h 7d 4w 6m
target_preserve_min no
target_preserve 7d 4w 6m
volume /mnt/12tb1-top-level
snapshot_dir @snap
target /mnt/12tb2-top-level/@snap
subvolume @home
subvolume @
Mount top-level subvolume:
mount -o subvolid=5 /dev/mapper/some-disk /mnt/12tb1-top-level
Manually create snapshot directory:
btrfs subvolume create /mnt/12tb1-top-level/@snap
btrfs subvolume create /mnt/12tb2-top-level/@snap
Snapshots only:
btrbk snapshot
Backup only:
btrbk resume
Create an additional snapshot of @home:
btrbk snapshot @home
Testing on Local Host
sudo mkdir -p /mnt/top-level
ls /mnt/top-level/@ || sudo mount -o subvolid=5 /dev/mapper/luks-ea7099e3-320d-4eb3-a4c3-9910a9af817b /mnt/top-level
ls /mnt/top-level/@snap || sudo btrfs subvolume create /mnt/top-level/@snap
ls /mnt/top-level/@foo || {
sudo btrfs subvolume create /mnt/top-level/@foo
sudo touch /mnt/top-level/@foo/bar
}
ls ~/btrbk-usb.img || {
fallocate -l 1G ~/btrbk-usb.img
sudo mkfs.btrfs ~/btrbk-usb.img
# Mount and create subvolume @
}
sudo mkdir -p /mnt/usb-top-level
ls /mnt/usb-top-level/@ || sudo mount -o subvolid=5 ~/btrbk-usb.img /mnt/usb-top-level
ls /mnt/usb-top-level/@snap || sudo btrfs subvolume create /mnt/usb-top-level/@snap
}
cat > ~/btrbk.cfg <<'EOF'
timestamp_format long
snapshot_preserve_min 2d
snapshot_preserve 24h 7d 4w 6m
target_preserve_min no
target_preserve 7d 4w 6m
volume /mnt/top-level
snapshot_dir @snap
target /mnt/usb-top-level/@snap
subvolume @foo
EOF
# Check configuration.
sudo btrbk -c ~/btrbk.cfg config print
sudo btrbk -c ~/btrbk.cfg config list
# Dry run.
sudo btrbk -c ~/btrbk.cfg run -n
# Create first snapshot.
sudo btrbk -c ~/btrbk.cfg run
# Print schedule.
sudo btrbk -c ~/btrbk.cfg run -n -S
sleep 5m
sudo btrbk -c ~/btrbk.cfg run
# List snapshots
sudo btrbk -c ~/btrbk.cfg list
NixOS Config
{ config, pkgs, ... }:
{
services.btrbk = {
extraPackages = [ pkgs.lz4 ];
# Lowest scheduling priority.
niceness = 19;
# Set of btrbk instances. The instance named btrbk is the default one.
instances = {
# This is the configuration equivalent to
# README.md section "Testing on Local Host".
"testing-on-local-host" = {
# The timer is disabled. The service has to be invoked manually.
onCalendar = null;
#onCalendar = "hourly";
#onCalendar = "*:0/15"; # Every 15min
# Configuration options for btrbk. Nested attrsets translate to subsections.
settings = {
timestamp_format = "long";
stream_compress = "lz4";
snapshot_preserve_min = "2d";
snapshot_preserve = "24h 7d 4w 6m";
target_preserve_min = "no";
target_preserve = "7d 4w 6m";
volume."/mnt/top-level" = {
snapshot_dir = "@snap";
target = "/mnt/usb-top-level/@snap";
subvolume = "@foo";
};
};
};
};
};
}
pi3bplus Host Config
A local btrbk instance manages snapshots and local backups.
Another remote backup server periodically pulls snapshots to create remote backups. Here is its config:
cat > ~/btrbk.cfg <<'EOF'
timestamp_format long
stream_compress lz4
ssh_identity /mnt/backup/rootNas_ed25519
# Create backups.
target_preserve_min no
# Daily backups -> @daily cron entry
target_preserve 7d 4w 6m
# Hourly backups -> @hourly cron entry
#target_preserve 24h 7d 4w 6m
# Don't create or delete snapshots on remote server "rootNas".
snapshot_preserve_min all
snapshot_create no
volume ssh://rootNas/jc-data
snapshot_dir /snap
target /mnt/backup/snap
subvolume arch.p1st.de
subvolume blogger.privacy1st.de
subvolume changedetection.p1st.de
subvolume cloud.privacy1st.de
subvolume git.privacy1st.de
subvolume mastodon-toot-follower.privacy1st.de
subvolume money.p1st.de
subvolume music.privacy1st.de
subvolume paste.p1st.de
subvolume proxy
subvolume recipe.privacy1st.de
subvolume traggo.privacy1st.de
volume ssh://rootNas/mnt/data/jc-data
snapshot_dir /mnt/data/snap2
target /mnt/backup/snap2
subvolume cloud.privacy1st.de
subvolume cloud.fykml.de
EOF