Skip to content

Instantly share code, notes, and snippets.

@phretor
Last active January 4, 2021 10:04
Show Gist options
  • Save phretor/3a11b75063d58196670e9ae6341d7656 to your computer and use it in GitHub Desktop.
Save phretor/3a11b75063d58196670e9ae6341d7656 to your computer and use it in GitHub Desktop.
Periodic off-site replication with FreeBSD, duplicacy and B2

Periodic off-site replication with FreeBSD, duplicacy and B2

I've recently discovered duplicacy, and I've decided to use it with an B2 backend to keep a mirrored offsite replication of the main directories of my home NAS.

I use the periodic system, which offers a nice abstraction over crontab for periodic tasks, and duplicacy to perform the actual backup. To setup duplicacy, I've followed the guide. In simple words, for each directory that I want to replicate, I setup duplicacy in it, which in turns means that you'll have to have a .duplicacy subdirectory in it.

FreeBSD's Periodic

First off, I've added the following to /etc/defaults/periodic.conf. The prefix, 666, is on purpose.

# 666.backup
monthly_backup_enabled="YES"                            # Replicate offsite

The monthly job is invoked every month, and more specifically as defined in the /etc/crontab:

$ grep monthly /etc/crontab
# Perform daily/weekly/monthly maintenance.
30      5       1       *       *       root    periodic monthly

The backup job

The above variable enables the 666.backup job, defined in /etc/periodic/monthly/666.backup (make sure the executable bit is set).

#!/bin/sh
#

# If there is a global system configuration file, suck it in.
#
if [ -r /etc/defaults/periodic.conf ]
then
    . /etc/defaults/periodic.conf
    source_periodic_confs
fi

case "$monthly_backup_enable" in
    [Yy][Ee][Ss])
        echo ""
        echo "Backup:"

        periodic /etc/periodic/backup || rc=3;;

    *)  rc=0;;
esac

exit $rc

So, to turn it on/off, just change the monthly_backup_enabled variable accordingly.

The backup periodic task triggers the execution of any executable script, in lexicographic order, found in the /etc/periodic/backup directory. In there, I've defined two scripts:

  • /etc/periodic/backup/100.duplicacy - that is the actual backup
  • /etc/periodic/backup/101.duplicacy.prune - that prunes any old snapshot (see below)

/etc/periodic/backup/100.duplicacy

#!/bin/sh -
#

# If there is a global system configuration file, suck it in.
#
if [ -r /etc/defaults/periodic.conf ]
then
    . /etc/defaults/periodic.conf
    source_periodic_confs
fi

rc=0

. $backup_duplicacy_pre

for target in $backup_duplicacy_targets
do
        path=`echo $target | cut -d\@ -f 1`
        cd "$path"
        echo "[`date`] Backing up $path"
        $backup_duplicacy -log backup -stats
        sleep 1
done

. $backup_duplicacy_post

rc=$?

exit $rc

The main variable is $backup_duplicacy, which points to the duplicacy executable, which is defined in the /etc/defaults/duplicacy.conf file:

# 100.duplicacy
# use <path>@<prune n:m> syntax - https://github.com/gilbertchen/duplicacy/blob/master/GUIDE.md#prune
backup_duplicacy_targets="/etc@0:120 /data/home@0:31 /data/tm@0:31 /data/freezer@0:31 /data/dropbox@0:31"
backup_duplicacy_pre="/root/bin/dup_pre.sh"             # keep this safe!
backup_duplicacy_post="/root/bin/dup_post.sh"
backup_duplicacy="/usr/local/bin/duplicacy"

Cleanup tasks

Notice the $backup_duplicacy_pre variable, which point to a script that define environment variables to hold the duplicacy secrets (that's why you wanna keep them safe). In my case:

#/root/bin/dup_pre.sh

export DUPLICACY_PASSWORD="***"     # duplicacy backup encryption password
export DUPLICACY_B2_ID="***"        # B2 client ID
export DUPLICACY_B2_KEY="***"       # B2 client secret

The $backup_duplicacy_post script takes care of unsetting these variables after duplicacy is done.

#/root/bin/dup_pre.sh

unset DUPLICACY_PASSWORD
unset DUPLICACY_B2_ID
unset DUPLICACY_B2_KEY

Backup target and retain policy definition

The backup targets (i.e., list of directories to be backed up) is defined in the $backup_duplicacy_targets variable, which in my case is:

backup_duplicacy_targets="/etc@0:120 /data/home@0:31 /data/tm@0:31 /data/freezer@0:31 /data/dropbox@0:31"

So, I backup:

  • /etc and retain no snapshot older than 120 days
  • /data/home and retain no snapshot older than 31 days
  • /data/tm and retain no snapshots older than 31 days
  • /data/freezer and retain no snapshots older than 31 days
  • /data/dropbox and retain no snapshots older than 31 days

Given that we run the job every month:

  • I'll keep the last 4 snapshots of /etc
  • I'll keep the last 1-2 snapshots of /data/home, /data/tm, /data/freezer, and /data/dropbox

In other words, this is almost a plain mirror of the above directories, except for /etc/, which is backed up with some versioning (up to 4 versions).

The N:M numbers are as defined in the Prune section of the guide.

/etc/periodic/backup/101.duplicacy.prune

#!/bin/sh -
#

# If there is a global system configuration file, suck it in.
#
if [ -r /etc/defaults/periodic.conf ]
then
    . /etc/defaults/periodic.conf
    source_periodic_confs
fi

rc=0

. $backup_duplicacy_pre

for target in $backup_duplicacy_targets
do
        path=`echo $target | cut -d\@ -f 1`
        keep=`echo $target | cut -d\@ -f 2`
        cd "$path"
        echo "[`date`] Pruning $path $keep"
        $backup_duplicacy -log prune $backup_duplicacy_prune_opts -keep $keep
        sleep 1
done

. $backup_duplicacy_post

rc=$?

exit $rc

Summary

The cron daemon checks /etc/crontab, which triggers the periodic monthly every month. This defines the backup job, which comprises 100.duplicacy and 101.duplicacy.prune tasks. Any output is automatically delivered via email (if set).

@tophee
Copy link

tophee commented Jul 15, 2018

Duplicacy now has a forum at forum.duplicacy.com. Feel free to mention your setup there.

@dr-diem
Copy link

dr-diem commented Dec 27, 2020

Thanks so much for this howto. I've come across two errors:

  • I could not see how the two periodic scripts see the content of /etc/defaults/duplicacy.conf? In my tests I got an error when executing 100.duplicacy on reaching the line '. $backup_duplicacy_pre'. Consequently I added a line '. /etc/defaults/duplicacy.conf' to both scripts in order suck in the contents of the file. How did you solve this in your implementation?
  • your addition to /etc/defaults/periodic.conf uses a var named $monthly_backup_enable, yet the script /etc/periodic/monthly/666.backup tests a var named $monthly_backup_enabled

@phretor
Copy link
Author

phretor commented Jan 4, 2021

Hi @dr-diem!

Although I'm not using this method anymore (since I switched to Duplicacy Web), I think it all boils down to if [ -r /etc/defaults/periodic.conf ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment