A quick "how to" on what you need to do to both setup AND recover a single-server PostgreSQL database using WAL-E
- WAL-E: https://github.com/wal-e/wal-e
- Assuming Ubuntu 12.04 LTS ("Precise")
- We'll be using S3. Make sure you have an IAM in a group with
GetObject
,ListBucket
andPutObject
on the bucket you want to use (and that it's not public).
- These packages:
$ sudo apt-get install daemontools libevent-dev python-all-dev lzop pv
$ sudo easy_install pip
$ sudo pip install wal-e
Notes: daemontools provides us with the envdir
program, which lets us (safely) store sensitive keys as environmental variables. We'll use it to store our S3 credentials.
- An S3 bucket (i.e.
/myservice/pg-backups
) - An S3 user that can (at a minimum) GetObject, ListBucket and PutObject. I'd suggest turning on versioning on the bucket and adding GetObjectVersion to that list too. That'll make sure server compromise won't allow someone to overwrite the backups you made[3]
- Set up your access keys:
$ mkdir -p /etc/wal-e.d/env
$ echo "YOUR_AWS_ACCESS_KEY" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
$ echo "YOUR_AWS_SECRET" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
$ echo 's3://myservice/pg-backups' > /etc/wal-e.d/env/WALE_S3_PREFIX
$ chown -R root:postgres /etc/wal-e.d
- Uncomment and modify these lines in your
postgresql.conf
file under/etc/postgresql/9.3/main/
wal_level = archive
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env /usr/local/bin/wal-e wal-push %p'
archive_timeout = 60
Note: Make sure to use the full path to wal-e so that PostgreSQL can find it.
Now we're ready to make our first backup. You can also set the PGDATA
environmental variable to the location of your PostgreSQL data dir (/var/lib/postgresql/9.3/main
on Ubuntu/Debian) either as a shell variable or using envdir
like we did for the AWS keys.
- Make sure you're the
postgres
user (the default user PostgreSQL runs as) withsudo su - postgres
- Create our first full backup, which is the reference point for our WAL logs with the following:
envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-push /var/lib/postgresql/9.3/main
- We can check the list of full backups with
envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-list
You can also delete older backups with wal-e delete [--confirm] before <base_XXXX>
where <base_XXX> is from backup-list
. Make sure to double check the dates on the backups.
We'll assume worst-case here: you have a blank machine with everything as it was except your data. Postgres has been re-installed as before and you're now wanting to recover your database.
Note: Make sure to read the WAL-E docs on user-created tablespaces and how that interacts with fetches first.
- Make sure PostgreSQL isn't running and change to the
postgres
user. - Delete the default data directory as the restore process will re-create it.
$ rm -r /var/lib/postgresql/9.3/main # (under Ubuntu/Debian)
- Fetch the latest backup. You can replace
LATEST
with the name of a specific backup as identified frombackup-list
:
$ envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-fetch /var/lib/postgresql/9.3/main LATEST
- Ensure the permissions are correct:
$ chown -R postgres:postgres /var/lib/postgresql/9.3/main
$ chmod 0700 /var/lib/postgresql/9.3/main
- Create a
recovery.conf
within/var/lib/postgresql/9.3/main/
with the following contents:
restore_command = '/usr/bin/envdir /etc/wal-e.d/env /usr/local/bin/wal-e wal-fetch "%f" "%p"'
- Start your PostgreSQL server. It might take a little while (it will replay the the WAL files on top of your specified base backup). The
recovery.conf
file will be renamed torecovery.done
once the process is complete. - Get back to work (read: check your data is there!)
@james4k - I've not needed to install those on my system(s), but it may be that something else earlier in my deployment is covering that off.
I'll check the requirements for the packages and update the article if so!
PS: It's a shame that Gist's don't notify the author of comments!