Using Borg and Google Cloud Storage for backing up personal projects

borg-with-gs.md

Why do you prefer Borg over git-lfs/git-annex?

• GitHub LFS has a 2GB limit on file size. This never works with large projects like video productions.
• git-annex uses symlinks to manage annexed files. This doesn't work well on Windows.

So my conclusion at the moment is that Borg is the best backup software for mid/large-sized personal projects.

Weren't you using rsync.net? Why did you switch to Google Cloud?

rsync.net is quite handy for using with Borg, but it's slow from Japan. The server is located in Hong Kong, and the connection speed varies through the day. It might be a good solution for people in North America, Western Europe and China, but unfortunately it doesn't work well from my workplaces.

I should admit that Google Cloud Storage is super fast and stable. It also provides long-term/low-cost storage classes (nearline/coldline/archive) that are cost beneficial for backup use.

Is Borg flexible enough for version control?

No. Although Borg is useful for taking project snapshots, it doesn't provide any version control features like git or hg. When I need to track file versions, I create a git repository in a subdirectory and make it contained in the Borg repository.

Installing gsutil

• Linux and Windows (WSL): Install pip sudo apt install python3-pip, then install gsutil sudo pip3 install gsutil.
• MacOS: Install pip sudo easy_install -U pip, then install gsutil pip install gsutil --user. The user flag is needed to avoid a package conflict problem with six.

After installing gsutil, authenticate it with gsutil config.

Creating a Google Cloud Storage bucket

Go to Google Cloud Console and create a bucket.

Create Borg folder in the bucket. The Borg repository files will be copied into this folder.

Installing Borg

• Linux: sudo apt install borgbackup
• Windows (WSL): Install the borgbackup apt package from PPA. Don't use the Ubuntu package. Some issues on WSL are only fixed in the very recent versions.
• MacOS: brew cask install borgbackup.

Then create a borg repository:

borg init --encryption=none Borg

Borg related files

Download borg.sh from this gist or run the following command line.

curl -L https://git.io/Jv5k0 > borg.sh

Open borg.sh with a text editor and modify BUCKET_NAME to the Google Storage bucket name. Then put it in the project root directory, and make it executable using chmod.

Run the following command line to initialize the Borg repository.

./borg.sh init

It's recommended to upload borg.sh to the Google Storage bucket because it'll be needed to restore the repository in another environment.

Taking a project snapshot

./borg.sh push

borg.sh

#!/bin/sh

REPO="Borg"
BUCKET="gs://BUCKET-NAME/Borg"
EXCLUDE="--exclude-from Borg/exclude.txt"

if [ -z $1 ]; then
  echo "Usage: $0 [init|push|pull|list|tar]"
elif [ $1 = "push" ]; then
  borg create -s --list --filter=AME --progress $EXCLUDE "$REPO"::'{now}' .
  borg delete "$REPO"::latest
  borg create $EXCLUDE "$REPO"::latest .
  gsutil -m rsync -r "$REPO" "$BUCKET"
elif [ $1 = "pull" ]; then
  test -d "$REPO" || mkdir "$REPO"
  gsutil -m rsync -r "$BUCKET" "$REPO"
  borg extract --list --progress "$REPO"::latest
elif [ $1 = "list" ]; then
  borg list "$REPO"
elif [ $1 = "tar" ]; then
  borg export-tar --tar-filter="pigz" "$REPO"::latest borg-latest.tar.gz
elif [ $1 = "init" ]; then
  borg init --encryption=none Borg
  curl -s -L 'https://git.io/Jv5kB' > Borg/exclude.txt
fi

exclude.txt

Ableton/Backup
Borg
Mastering/Adobe*
Mastering/*_AME
Mastering/*.mov
Unity/Library
Unity/Logs
Unity/Recordings
Unity/Temp