backup_to_box.sh

#!/bin/bash

# source Lmod if it's not already sourced - cron starts in clean environment
if [ -z "$LMOD_VERSION" ]; then
  source /etc/profile.d/chpc.sh
fi

ml rclone
# --links needs rclone >= 1.46
rclone -v sync /home/mcuma ubox:tallboy/mcuma --exclude-from /home/mcuma/bin/scripts/exclude.txt --links >& /home/mcuma/bin/scripts/backup.log

exclude.txt

zhdanov_pdata/**
zhdanov_ser/**
zhdanov_par/**
zhdanov_data/**
mp3/**
Downloads/**
RCLONE/**
.cache/**
.lmod.d/**
.star*/**
.mozilla/**
.config/**
.matlab/**
.allinea/**
.nv/**
.idl/**
.thumbnails/**
.adobe/**
.totalview/**
.local/**
.nx/**
.gimp*/**
.pip/**
.java/**
.macromedia/**
.udocker/**
.docker/**
temp/**
MYTF/**
jupyter-virtualenv/**
crap/**
bin/scripts/backup.log
*~
.ccache/**

instructions.md

Instructions

Set up rclone to link to UBox

ml rclone
rclone config

choose the following options

• n) New remote
• name> ubox
• Storage> box
• client_id> (blank)
• client_secret> (blank)
• box_config_file> (blank)
• box_sub_type> (enter) - default "user"
• Edit advanced config? (y/n) y/n> y
• root_folder_id> (enter) - default "0"
• upload_cutoff> (enter) - default "50M"
• commit_retries> (enter) - default "100"
• encoding> (enter) - default "Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot"
• Use auto config? y/n> y
• Now browser window will open prompting for authentication at UBox (unless you have authenticated earlier and the browser cookie is still active) Got code [ubox] type = box token = {your token}
• y) Yes this is OK (default) y/e/d> y
• q) Quit config e/n/d/r/c/s/q> q

Set up the backup

• Create a script that runs the rclone sync, called backup_to_box.sh (referenced below).

• Create exclusion file

• Create directory on the gdrive where this backup will go

rclone mkdir ubox:tallboy/mcuma

• Do an initial sync, either by running our script or by explicit command:

rclone -v sync /home/mcuma ubox:tallboy/mcuma --exclude-from /home/mcuma/bin/scripts/exclude.txt --links >& /home/mcuma/bin/scripts/backup.log

• Set up a cron job

crontab -e
0 2 * * * /home/mcuma/bin/scripts/backup_to_box.sh

This will run the backup daily at 2am.

Gotchas

• Box file names are case insensitive. Which means that if files in a directory only differ by case, only one of them will be saved. Potential solution: https://forum.rclone.org/t/box-duplicates-due-to-case-insensitive/5400, but it requires client side encryption.
• There appears to be a limit either with a number of transactions or with bandwidth, since after copying about 50 GB worth of files I started getting 2022/02/15 23:06:42 ERROR : marastik/doc: error reading destination directory: couldn't list files: http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug="" Increasing the number of retries and limiting bandwidth and number of transactions helped some: rclone -v sync /home/mcuma ubox:tallboy/mcuma --exclude-from /home/mcuma/bin/scripts/exclude.txt --links --bwlimit 1M --ignore-case-sync --retries 10 --tpslimit 10