Skip to content

Instantly share code, notes, and snippets.

@WaleedMortaja
Forked from dawid-czarnecki/firefly-iii-backuper.sh
Last active December 30, 2023 17:23
Show Gist options
  • Save WaleedMortaja/ddc87fc4dc7dcb73178e971c299d7e78 to your computer and use it in GitHub Desktop.
Save WaleedMortaja/ddc87fc4dc7dcb73178e971c299d7e78 to your computer and use it in GitHub Desktop.
Script to backup Firefly III database, uploads and config files installed with docker-compose
#!/bin/bash
files_to_backup=(.env .db.env .fidi.env docker-compose.yml )
info() { echo -e "\\033[1;36m[INFO]\\033[0m \\033[36m$*\\033[0m" >&2; }
warn() { echo -e "\\033[1;33m[WARNING]\\033[0m \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m \\033[31m$*\\033[0m" >&2; exit 1; }
intro () {
echo " ====================================================="
echo " Backup & Restore docker based FireFly III v1.4-1 "
echo " ====================================================="
echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
echo " Requirements:"
echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
echo " Note: The destination directory is created if it does not exist"
}
usage () {
echo "Usage: $0 backup|restore destination [no_files]"
echo "- backup|restore : Action you want to execute"
echo "- destination : The path for your backup file (including the file name)"
echo "- (optional) no_files : When passed, exclude the docker and environment files. Only backup or restore volumes"
echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
echo "To backup once per day you can add something like this to your cron:"
echo "1 01 * * * bash \"$(realpath $0)\" backup /home/backup/\$(date '+%F').tar.gz"
}
backup () {
script_path="$1"
dest_path="$(dirname $path)"
dest_file="$(basename $path)"
upload_volume="$3"
no_files=$4
to_backup=()
if [ ! -d "$dest_path" ]; then
info "Creating destination directory: $dest_path"
mkdir -p "$dest_path"
fi
if [ -f "$path" ]; then
warn "Provided file path already exists: $path."
read -p "Do you want to overwrite? (yes/no) " yn
case $yn in
yes) warn overwriting... ;;
no ) info exiting... ; exit;;
* ) fatal invalid response ;;
esac
fi
# Create temporary directory
if [ ! -d "$dest_path/tmp" ]; then
mkdir "$dest_path/tmp"
fi
# Files backup
if [ $no_files = "false" ]; then
not_found=()
for f in "${files_to_backup[@]}"; do
if [ ! -f "$script_path/$f" ]; then
not_found+=("$f")
else
cp "$script_path/$f" "$dest_path/tmp/"
to_backup+=("$f")
fi
done
if ((${#not_found[@]})); then
warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
fi
if ((${#to_backup[@]})); then
info "Backing up the following files in $script_path: ${to_backup[@]}"
fi
fi
# Version
app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
info 'Backing up App & database version numbers.'
echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
to_backup+=(version.txt)
# DB container
db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
if [ -z $db_container ]; then
warn "db container is not running. Not backing up."
else
info 'Backing up database'
docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
to_backup+=("firefly_db.sql")
fi
# Upload Volume
if [ -z $upload_volume ]; then
warn "upload volume does NOT exist. Not backing up."
else
info 'Backing up upload volume'
docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
to_backup+=("firefly_upload.tar.gz")
fi
# Compress
tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")
# Clean up
for file in "${to_backup[@]}"; do
rm -f "$dest_path/tmp/$file"
done
rmdir "$dest_path/tmp"
}
restore () {
script_path="$1"
src_path="$(dirname $path)"
backup_file="$(basename $path)"
upload_volume="$3"
no_files=$4
if [ ! -f "$path" ]; then
fatal "Provided backup file does not exist: $path"
fi
# Create temporary directory
if [ ! -d "$src_path/tmp" ]; then
mkdir "$src_path/tmp"
fi
# Files restore
if [ $no_files = "false" ]; then
tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
#readarray -t <<<$(tar -tf "$src_path/$backup_file")
# restored=(${MAPFILE[*]})
not_found=()
restored=()
for f in "${files_to_backup[@]}"; do
if [ ! -f "$script_path/$f" ]; then
not_found+=("$f")
else
cp "$src_path/tmp/$f" .
restored+=("$f")
fi
done
if ((${#not_found[@]})); then
warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
fi
if ((${#restored[@]})); then
info "Restoring the following files: ${restored[@]}"
fi
else
tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz version.txt
#restored=(firefly_db.sql firefly_upload.tar.gz)
fi
if [ ! -z $upload_volume ]; then
warn "The upload volume exists."
read -p "Do you want to overwrite? (yes/no) " yn
case $yn in
yes)
warn overwriting...
docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
restored+=(firefly_upload.tar.gz)
;;
no )
info upload volume is not restored;
rm -f "$src_path/tmp/firefly_upload.tar.gz"
;;
* ) fatal invalid response ;;
esac
fi
db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
if [ -z $db_container ]; then
warn "The db container is not running. Not restoring."
else
info 'Restoring database'
cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
restored+=(firefly_db.sql)
fi
cp "$src_path/tmp/version.txt" .
restored+=(version.txt)
# Clean up
for file in "${restored[@]}"; do
rm -f "$src_path/tmp/$file"
done
rmdir "$src_path/tmp"
}
main () {
intro
if [ $# -lt 2 ]; then
fatal "Not enough parameters.\n$(usage)"
fi
current_dir="$(dirname $0)"
action=$1
path="$2"
if [ -z "$3" ]; then
no_files=false
else
no_files=true
fi
if [ -d "$path" ]; then
fatal "Path is an existing directory. It has to be a file path"
fi
upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"
if [ "$action" == 'backup' ]; then
backup "$current_dir" "$path" "$upload_volume" "$no_files"
elif [ "$action" == 'restore' ]; then
restore "$current_dir" "$path" "$upload_volume" "$no_files"
else
fatal "Unrecognized action $action\n$(usage)"
fi
}
main "$@"
@WaleedMortaja
Copy link
Author

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux.
Linux: Print the resolved absolute file name; all but the last component must exist
MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

@Pindol83
Copy link

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux. Linux: Print the resolved absolute file name; all but the last component must exist MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

I'm not a programmer, so I wouldn't know exactly how to be of help, but by trying with ChatGPT to find a solution, it suggested that it might be possible to replace realpath with dest_file. Could that be a way?

@WaleedMortaja
Copy link
Author

@Pindol83 I found realpath did not seem so required. I removed it and provided some other improvements.
Please check the current version of the script and provide feedback.

@Pindol83
Copy link

Pindol83 commented Oct 11, 2023

Testing the new script gives me new errors, but it does create a backup. However, the backup isn't as large as the one I'm currently doing with a cron job. Could this be due to the fact that my database volume is named "firefly_firefly_iii_db," so it sees "firefly_iii_db" as an unused volume in the container?

[INFO]  Backing up the following files in /Users/macmini2018/docker/firefly: .env .db.env docker-compose.yml
grep: empty (sub)expression
Error response from daemon: No such container: grep
Error response from daemon: No such container: grep
[INFO]  Backing up App & database version numbers.
grep: empty (sub)expression
[WARNING]  db container is not running. Not backing up.
[INFO]  Backing up upload volume```

@Pindol83
Copy link

Pindol83 commented Oct 28, 2023

I found the problem for me, it's in line 90. I had to change this:

db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

to this:

db_container=$(docker ps | grep -E 'firefly_iii_db' | cut -d ' ' -f 1)

It seems that this syntax'firefly[-_](iii|)[_-]?db'doesn't work on MacOS, so I preferred to directly input the container name, even though it's not an elegant solution.

Now the backup runs without any problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment