Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.env .db.env .fidi.env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.4 "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Warning: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore /tmp/backup/destination/dir [no_files]"
    echo "- backup|restore - Action you want to execute"
    echo "- destination path of your backup file including file name"
    echo "- optionally backup or restore volumns only when no_files parameter is passed"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash /home/myname/backuper.sh backup /home/backup/\$(date '+%F').tar.gz"
}

backup () {
    script_path="$1"

    if [ ! -d "$(dirname $2)" ]; then
        info "Creating destination directory: $(dirname $2)"
        mkdir -p "$(dirname $2)"
    fi

    full_path=$(realpath $2)
    dest_path="$(dirname $full_path)"
    dest_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ -f "$full_path" ]; then
        warn "Provided file path already exists: $full_path. Overwriting"
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$script_path/$f" "$dest_path/tmp/"
                to_backup+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    full_path=$(realpath $2)
    src_path="$(dirname $full_path)"
    backup_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$src_path/$backup_file" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz
        restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists. Overwriting."
    fi

    docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
    restored+=(firefly_upload.tar.gz)

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    restored+=(version.txt)
    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    current_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$current_dir" "$path" "$upload_volume" $no_files
    elif [ "$action" == 'restore' ]; then
        restore "$current_dir" "$path" "$upload_volume" $no_files
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

@Pindol83 check my fork. It contains some bug fixes. For more details, see my latest comment. If problems still happens, add more details of the issues you encounter.

@WaleedMortaja Huge thanks for your bug fixes and updates!
I applied all of them and changed the container name regular expressions to catch all that were mentioned here.
It should work both on 5.x and 6.x versions of Firefly.

@jbhammon Thanks for your comments. Should work on your container names now.

Let me know if there are any problems.

Restore

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=hugo fireflyiii'

Hello @hjhp
I have tested your restore and it does not work for me. It's printing many errors, i.e. relations etc. already exist.
Am I missing something here? Have you tested the restore in the meantime?

All I did for the restore was replacing

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'

with (my database name is test and my username aswell)

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=test test'

After that I registered a user and made a backup (working with your suggestions).
Then I removed all containers & volumes and started/created it all up. Once running I ran the restore script.

Thank you very much in advance :)

@MuellerNicolas I moved to GnuCash in 2022-11, principally (if memory serves) because I didn't enjoy the way Firefly-III handled debts. I never ended up testing a restore. Some observations though: my restore line is not vastly different from the original by @dawid-czarnecki and @WaleedMortaja's fork, save that they use /usr/bin/mariadb and I use /usr/bin/psql.

I don't have much experience with Postgres; from my understanding the way all these backups work is they generate a file with SQL statements that can act as a restore file by feeding its contents into a SQL prompt, and that this general approach is the same across many databases (Postgres, MySQL, MariaDB…).

Without seeing your error messages exactly, it won't be easy to suggest many ideas:

• If the "relations already exist", then my first thought is that somehow you're not actually starting with a clean slate: could something have been omitted during your deletion process?
• If I read your comment correctly, you followed my script (i.e. Postgres) rather than the others' (using MariaDB). What happens if you try a backup-and-restore on a MariaDB setup?

Restore with postgresql container

Problem was that the container needs to be running for the restore. But when you start the db container it automatically creates a new database.
Solution was to drop the database, create a fresh one and recover the dump. The following is working for me:

info 'Step 1: Drop existing database'
docker exec -i $db_container dropdb -U username databasename
info 'Step 2: Create fresh database'
docker exec -i $db_container createdb -U username databasename
info 'Step 3: Restore dump'
cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container psql -U username -d databasename

It might be easier to comment if you share your Docker command (or docker-compose.yml if that's what you're using).

• At first glance my uncaffeinated brain wants to suggest that whatever image you're using just doesn't have bash ("bash": executable file not found), which is then causing commands like cat not to work, but if you're running fireflyiii/core:latest then this feels improbable (though I haven't tried running Firefly at all in recent times).
• If cat doesn't work, do things like less work? Can you run ls? Do any of the typical commands work?
• Have you verified that the contents of firefly_db.sql makes sense? (This is probably unrelated to any of the errors, just covering all bases.)

I would also note though that you seem to have 2 errors here: "docker: invalid spec" and "OCI runtime exec failed".

Somehow the tar file only gets a date in my case.
2014-01-18.tar
So no version or anything (this part '+%F' doesn't work).
If anyone knows how I can get that fixed..... :)

Other than that the save and restore work fine!

@MuellerNicolas Unfortunately I don't tested it with PostgreSQL so can't help much. With mariadb the restore works just fine.
@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

The date part already worked. It's the Firefly version that is not included in the filename. But looking at your script again I think I expected something that is not there :). You only include "firefly" in the filename so not also the actual version of Firefly. Right?
So this is in your script as example: firefly-2022-01-01.tar.gz
But mine come out as 2022-01-01.tar
In that case let's ignore it.

Thanks by the way for your script! It's very helpful!

@PuckStar Yes, that's correct. You can still find the firefly and database version inside the archive in version.txt file.
You are welcome! I'm happy it helps :)

Thank you for the script.
Suggestion : Rename ".fidi.env" file to ".importer.env", as its actual name on Firefly-III documetation : https://docs.firefly-iii.org/how-to/data-importer/installation/docker/#:~:text=save%20it%20as-,.importer.env,-next%20to%20the