Skip to content

Instantly share code, notes, and snippets.

@timwhitlock
Last active May 16, 2019 15:43
Show Gist options
  • Save timwhitlock/e6a60b7ad9dcf3042df8ca44d60a19d6 to your computer and use it in GitHub Desktop.
Save timwhitlock/e6a60b7ad9dcf3042df8ca44d60a19d6 to your computer and use it in GitHub Desktop.
Disk backup rotation script
#!/bin/bash
# Rotates disk image backups inside working directory
# Maintains a rolling window of 4 hourly snapshots (uncompressed)
# Plus 7xdays and 6xmonths of older compressed archives
# config
HOURS=4
DAYS=7
MONTHS=6
# current timestamp (seconds)
NOW=$(date +%s);
# move uncompressed snapshots into daily rotation
for file in hourly-*.img; do
if [ -f $file ]; then
MOD=$( stat -c %Y $file );
AGE=$((NOW - MOD));
MAX=$((HOURS*3600));
if test $AGE -gt $MAX; then
echo "$file is over $HOURS hours old"
DAY=$( date -d "@$MOD" +'%Y%m%d' );
if [ -f "daily-$DAY.img.gz" ]; then
echo "Removing hourly snapshot, already have daily ($DAY)"
rm -v $file
else
echo "Compressing daily backup (this can take a long time)"
mv -v $file "daily-$DAY.img";
gzip -v "daily-$DAY.img"
fi
fi
fi
done
# move daily snapshots over (DAYS) old into monthly rotation
for file in daily-*.img.gz; do
if [ -f $file ]; then
MOD=$( stat -c %Y $file );
AGE=$((NOW - MOD));
MAX=$((DAYS*86400));
if test $AGE -gt $MAX; then
echo "$file is over $DAYS days old"
MTH=$( date -d "@$MOD" +'%Y%m' );
if [ -f "monthly-$MTH.img.gz" ]; then
echo "Removing daily snapshot, already have monthly ($MTH)"
rm -v $file
else
echo "Preserving monthly backup"
mv -v $file "monthly-$MTH.img.gz"
fi
fi
fi
done
# remove monthly snapshots over (MONTHS) old
for file in monthly-*; do
if [ -f $file ]; then
MOD=$( stat -c %Y $file );
AGE=$((NOW - MOD));
MAX=$((MONTHS*2678400));
if test $AGE -gt $MAX; then
echo "$file is over $MONTHS months old, removing permanently"
rm -v file
fi
fi
done
@timwhitlock
Copy link
Author

timwhitlock commented May 16, 2019

I use this to rotate disk images saved to attached storage on one of my Linode servers. (Long story for why).

The actual backup (snapshot) process is not in this script, but is based on this awesome method (I use for MongoDB). That process writes a new disk image every hour to a file named like "hourly-%Y%m%d%H%M%S.img". The rotate script runs beforehand in order to clear space, but could run any time I suppose.

It maintains a rolling window of 4 hourly backups. (This is just to save disk space, change it to 24 if you have room). These images are uncompressed, because compressing and uncompressing is SLOW. I don't want to wait 15 minutes if I have to restore in a hurry.

When an hourly backup becomes over 4 hours old, it is compressed to a daily archive unless one already exists. Then similarly into monthly rotation after a week. I keep six months, but probably only need one. Change that to whatever.

Notes:

  • File names are not parsed for timestamps. Actual modification times are used.
  • For a 5GB disk image (gzips to 1.5G), this full rotation fills a 40G disk
  • Running hourly with HOURS=4: The daily archive compression would occur between 4-5am from the midnight snapshot
  • Months are rounded to 31 days :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment