Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.env .db.env .fidi.env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.4 "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Warning: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore /tmp/backup/destination/dir [no_files]"
    echo "- backup|restore - Action you want to execute"
    echo "- destination path of your backup file including file name"
    echo "- optionally backup or restore volumns only when no_files parameter is passed"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash /home/myname/backuper.sh backup /home/backup/\$(date '+%F').tar.gz"
}

backup () {
    script_path="$1"

    if [ ! -d "$(dirname $2)" ]; then
        info "Creating destination directory: $(dirname $2)"
        mkdir -p "$(dirname $2)"
    fi

    full_path=$(realpath $2)
    dest_path="$(dirname $full_path)"
    dest_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ -f "$full_path" ]; then
        warn "Provided file path already exists: $full_path. Overwriting"
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$script_path/$f" "$dest_path/tmp/"
                to_backup+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    full_path=$(realpath $2)
    src_path="$(dirname $full_path)"
    backup_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$src_path/$backup_file" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz
        restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists. Overwriting."
    fi

    docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
    restored+=(firefly_upload.tar.gz)

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    restored+=(version.txt)
    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    current_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$current_dir" "$path" "$upload_volume" $no_files
    elif [ "$action" == 'restore' ]; then
        restore "$current_dir" "$path" "$upload_volume" $no_files
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

Thanks for spotting this @william-1066. I fixed it.

Thank you for this script!

I made a few notes along the way for my install, which uses PostgreSQL.

full_path failing

I noticed that your full_path was failing on my Ubuntu 18 LTS (Digital Ocean VPS):
Suppose my command is ./fireflyiii_backup.sh backup /home/hugo/fireflyiii/backup/fireflyiii_$(date '+%F').tar AND the folder /home/hugo/fireflyiii/backup/ does not exist.
Then running realpath /home/hugo/fireflyiii/backup/fireflyiii_$(date '+%F').tar returns realpath: /home/hugo/fireflyiii/backup/fireflyiii_2022-08-29.tar: No such file or directory.

This was causing dest_path and dest_file to fail in a potentially dangerous way: because full_path was blank (mkdir: cannot create directory ‘’: No such file or directory), the temporary folder was defined as /tmp (the system tmp folder), which means the rmdir "$dest_path/tmp" command under "Clean up" tried to remove /tmp.
I was not running as root, so nothing bad happened, but the script failed.

Solution: bypass full_path entirely

dest_path="$(dirname $2)"
dest_file="$(basename $2)"

This seems to work because dirname ~/fireflyiii/dir_that_does_not_exist/test.txt returns /home/hugo/fireflyiii/dir_that_does_not_exist.

Modifications for PostgreSQL

• I have not tested a restore yet.
• This assumes passwordless access, which is especially important if this is to be a cron job (see pg_dump's -w flag). This may entail setting up a .pgaccess file.

Backup

docker exec $db_container bash -c '/usr/bin/pg_dump --username=hugo -w fireflyiii' > "$dest_path/tmp/firefly_db.sql"

Restore

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=hugo fireflyiii'

Thanks @hjhp for spotting the bug. I applied realpath at the beginning because I remember having some issues without it when running the script form other locations. For that reason I prefer to create a directory in case it does not exist.
Feel free to implement a logic to detect the type of the database. Keep in mind, you can use credentials saved in the environmental variables inside db container the same way it's used for MySQL right now:

docker exec $db_container bash -c '/usr/bin/mysqldump -u firefly --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"

Thank you for sharing this! It works really well.

Thanks for sharing your script ! It saved me a lot of time

When I tried to run it, it gave me an error "db container is not running. Not backing up."
I found out that the container names have change from firefly_* to firefly3_*. Thus the script was not able to identify the container ID.
updated the code as follows:

db_container=$(docker ps | fgrep firefly3_db | cut -d ' ' -f 1) (lines 85 and 163)

in backup() and restore()

also updated the warning to
warn "The db container is not running. Not restoring." (line 166)
in restore()

@williamgurzoni I'm happy it works for you :)
@mephisto20 Thanks for the feedback. I updated your suggested lines in a way to support older and newer versions. Let me know if that works.

Thanks for the script!
I found a point that might be improved.
In my docker-compose setup, the container names did not have the _ but - in the name (like firefly-app instead of firefly_app).
I would therefore suggest using:
L77
app_container=$(docker ps | grep -E 'firefly[_-]app' | cut -d ' ' -f 1)
L85
db_container=$(docker ps | grep -E 'firefly[0-9]?[_-]db' | cut -d ' ' -f 1)
L163
db_container=$(docker ps | grep -E 'firefly[0-9]?[_-]db' | cut -d ' ' -f 1)

Instead of edition the file as previous comment suggests, I added container_name: new_name fore each service, setting a name that works with the script.

Thanks for the script!
I found a point that might be improved.

MySQL user should be a variable，not fixed string: firefly

docker exec $db_container bash -c '/usr/bin/mysqldump -u "$MYSQL_USER" --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"

@woernsn @lightxue Glad that it helped you and thanks for the feedback.
I updated the script to include your improvements.

Thank you for this script! Really appreciate the work you shared here. I am starting to experiment with firefly-iii, and I ran into a couple issues with this script.

First, I noticed all my containers were being named firefly-iii-*, rather than fireflyiii-*, and the grep statements in the script weren't finding them. I couldn't find a recent commit in the firefly repo that changed the container names, so maybe it's just something I'm not understanding with regex or how docker names things. Changing to grep -E 'firefly-iii[_-]app' seemed to solve the problem.

Second, I see the examples are running the script using a dynamic name for the backup file so that the filename has the day's date in it. My understanding is that you could run that example every day and create a series of backup files, a new one for each day. However, the script uses the realpath command to set the full_path and dest_path variables in the backup function, and realpath fails if the file doesn't currently exist. Am I misunderstanding the example? Should the backup file already exist and we just overwrite it with the latest backup?

First, I noticed all my containers were being named firefly-iii-*, rather than fireflyiii-*, and the grep statements in the script weren't finding them. I couldn't find a recent commit in the firefly repo that changed the container names, so maybe it's just something I'm not understanding with regex or how docker names things. Changing to grep -E 'firefly-iii[_-]app' seemed to solve the problem.

I think, that the base name in front of the container names (app, db) comes from the folder you are in. Is your directory named firefly-iii by chance?
Edit: Normally firefly is used, not fireflyiii or firefly-iii.

Is your directory named firefly-iii by chance? Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It sure is, so that explains that question, appreciate it!

Second, I see the examples are running the script using a dynamic name for the backup file so that the filename has the day's date in it. My understanding is that you could run that example every day and create a series of backup files, a new one for each day.

Right.

However, the script uses the realpath command to set the full_path and dest_path variables in the backup function, and realpath fails if the file doesn't currently exist. Am I misunderstanding the example? Should the backup file already exist and we just overwrite it with the latest backup?

No. The backup file does not need to be existing. If it does, it will be overwritten (and a warning will be shown, but without asking for confirmation).

In the main function line 196, the full path of the backup file is tested. If it is an existing directory, the command would fail. Otherwise (I.e., it does not exist, or it is a file path) it will continue.

The main function later calls the backup function passing the path. At the beginning of the backup function, all the parent directories of the path are created if they do not exist just before calling realpath.

According to realpath's man page:

Print the resolved absolute file name; all but the last component must exist

So, It is not required for the backup file to be already existing.

Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It actually happens that I have firefly-iii also as my directory name. It maybe not be as unusual as you might think. I think the documentation should suggest a name to make the docker container name more consistent. I have filed an issue for that

@dawid-czarnecki
Thank you for the script!
I have some notes (see my fork):

• mysql and mysqldump were deprecated and are now removed from the docker image (reference)
• Compressed tar files (-z flag) usually have .tar.gz file extension
• Creating directories in line 46 is duplicated since it should have been already created in line 34
• I think checking dest_path in line 48 would check the file name against the current working directory, instead of the backup destination directory
• Using fgrep gives a warning of fgrep: warning: fgrep is obsolescent; using grep -F
• Use alpine instead of ubuntu image as it is smaller, but does the job
• Some typos

Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It actually happens that I have firefly-iii also as my directory name. It maybe not be as unusual as you might think. I think the documentation should suggest a name to make the docker container name more consistent. I have filed an issue for that

As per the issue, container names have change to: firefly_iii_core and firefly_iii_db (commit). This will break the current grep in the script. However, the script would be already half-broken for anyone who updates (and pull) his docker file because of "mysql and mysqldump were deprecated and are now removed from the docker image (reference)"

I have updated my fork accordingly.

hi guys, i use Firefly III with Docker on an old Mac Mini that serves as my server. How can I modify the script so that it also works with macOS

What have you identified that doesn't work?

@Pindol83 check my fork. It contains some bug fixes. For more details, see my latest comment. If problems still happens, add more details of the issues you encounter.

@WaleedMortaja Huge thanks for your bug fixes and updates!
I applied all of them and changed the container name regular expressions to catch all that were mentioned here.
It should work both on 5.x and 6.x versions of Firefly.

@jbhammon Thanks for your comments. Should work on your container names now.

Let me know if there are any problems.

Restore

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=hugo fireflyiii'

Hello @hjhp
I have tested your restore and it does not work for me. It's printing many errors, i.e. relations etc. already exist.
Am I missing something here? Have you tested the restore in the meantime?

All I did for the restore was replacing

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'

with (my database name is test and my username aswell)

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=test test'

After that I registered a user and made a backup (working with your suggestions).
Then I removed all containers & volumes and started/created it all up. Once running I ran the restore script.

Thank you very much in advance :)

@MuellerNicolas I moved to GnuCash in 2022-11, principally (if memory serves) because I didn't enjoy the way Firefly-III handled debts. I never ended up testing a restore. Some observations though: my restore line is not vastly different from the original by @dawid-czarnecki and @WaleedMortaja's fork, save that they use /usr/bin/mariadb and I use /usr/bin/psql.

I don't have much experience with Postgres; from my understanding the way all these backups work is they generate a file with SQL statements that can act as a restore file by feeding its contents into a SQL prompt, and that this general approach is the same across many databases (Postgres, MySQL, MariaDB…).

Without seeing your error messages exactly, it won't be easy to suggest many ideas:

• If the "relations already exist", then my first thought is that somehow you're not actually starting with a clean slate: could something have been omitted during your deletion process?
• If I read your comment correctly, you followed my script (i.e. Postgres) rather than the others' (using MariaDB). What happens if you try a backup-and-restore on a MariaDB setup?

Restore with postgresql container

Problem was that the container needs to be running for the restore. But when you start the db container it automatically creates a new database.
Solution was to drop the database, create a fresh one and recover the dump. The following is working for me:

info 'Step 1: Drop existing database'
docker exec -i $db_container dropdb -U username databasename
info 'Step 2: Create fresh database'
docker exec -i $db_container createdb -U username databasename
info 'Step 3: Restore dump'
cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container psql -U username -d databasename

It might be easier to comment if you share your Docker command (or docker-compose.yml if that's what you're using).

• At first glance my uncaffeinated brain wants to suggest that whatever image you're using just doesn't have bash ("bash": executable file not found), which is then causing commands like cat not to work, but if you're running fireflyiii/core:latest then this feels improbable (though I haven't tried running Firefly at all in recent times).
• If cat doesn't work, do things like less work? Can you run ls? Do any of the typical commands work?
• Have you verified that the contents of firefly_db.sql makes sense? (This is probably unrelated to any of the errors, just covering all bases.)

I would also note though that you seem to have 2 errors here: "docker: invalid spec" and "OCI runtime exec failed".

Somehow the tar file only gets a date in my case.
2014-01-18.tar
So no version or anything (this part '+%F' doesn't work).
If anyone knows how I can get that fixed..... :)

Other than that the save and restore work fine!

@MuellerNicolas Unfortunately I don't tested it with PostgreSQL so can't help much. With mariadb the restore works just fine.
@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

The date part already worked. It's the Firefly version that is not included in the filename. But looking at your script again I think I expected something that is not there :). You only include "firefly" in the filename so not also the actual version of Firefly. Right?
So this is in your script as example: firefly-2022-01-01.tar.gz
But mine come out as 2022-01-01.tar
In that case let's ignore it.

Thanks by the way for your script! It's very helpful!

@PuckStar Yes, that's correct. You can still find the firefly and database version inside the archive in version.txt file.
You are welcome! I'm happy it helps :)

Thank you for the script.
Suggestion : Rename ".fidi.env" file to ".importer.env", as its actual name on Firefly-III documetation : https://docs.firefly-iii.org/how-to/data-importer/installation/docker/#:~:text=save%20it%20as-,.importer.env,-next%20to%20the