A quick "how to" on what you need to do to both setup AND recover a single-server PostgreSQL database using WAL-E
- WAL-E: https://github.com/wal-e/wal-e
- Assuming Ubuntu 12.04 LTS ("Precise")
- We'll be using S3. Make sure you have an IAM in a group with
GetObject
,ListBucket
andPutObject
on the bucket you want to use (and that it's not public).
- These packages:
$ sudo apt-get install daemontools libevent-dev python-all-dev lzop pv
$ sudo easy_install pip
$ sudo pip install wal-e
Notes: daemontools provides us with the envdir
program, which lets us (safely) store sensitive keys as environmental variables. We'll use it to store our S3 credentials.
- An S3 bucket (i.e.
/myservice/pg-backups
) - An S3 user that can (at a minimum) GetObject, ListBucket and PutObject. I'd suggest turning on versioning on the bucket and adding GetObjectVersion to that list too. That'll make sure server compromise won't allow someone to overwrite the backups you made[3]
- Set up your access keys:
$ mkdir -p /etc/wal-e.d/env
$ echo "YOUR_AWS_ACCESS_KEY" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
$ echo "YOUR_AWS_SECRET" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
$ echo 's3://myservice/pg-backups' > /etc/wal-e.d/env/WALE_S3_PREFIX
$ chown -R root:postgres /etc/wal-e.d
- Uncomment and modify these lines in your
postgresql.conf
file under/etc/postgresql/9.3/main/
wal_level = archive
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env /usr/local/bin/wal-e wal-push %p'
archive_timeout = 60
Note: Make sure to use the full path to wal-e so that PostgreSQL can find it.
Now we're ready to make our first backup. You can also set the PGDATA
environmental variable to the location of your PostgreSQL data dir (/var/lib/postgresql/9.3/main
on Ubuntu/Debian) either as a shell variable or using envdir
like we did for the AWS keys.
- Make sure you're the
postgres
user (the default user PostgreSQL runs as) withsudo su - postgres
- Create our first full backup, which is the reference point for our WAL logs with the following:
envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-push /var/lib/postgresql/9.3/main
- We can check the list of full backups with
envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-list
You can also delete older backups with wal-e delete [--confirm] before <base_XXXX>
where <base_XXX> is from backup-list
. Make sure to double check the dates on the backups.
We'll assume worst-case here: you have a blank machine with everything as it was except your data. Postgres has been re-installed as before and you're now wanting to recover your database.
Note: Make sure to read the WAL-E docs on user-created tablespaces and how that interacts with fetches first.
- Make sure PostgreSQL isn't running and change to the
postgres
user. - Delete the default data directory as the restore process will re-create it.
$ rm -r /var/lib/postgresql/9.3/main # (under Ubuntu/Debian)
- Fetch the latest backup. You can replace
LATEST
with the name of a specific backup as identified frombackup-list
:
$ envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-fetch /var/lib/postgresql/9.3/main LATEST
- Ensure the permissions are correct:
$ chown -R postgres:postgres /var/lib/postgresql/9.3/main
$ chmod 0700 /var/lib/postgresql/9.3/main
- Create a
recovery.conf
within/var/lib/postgresql/9.3/main/
with the following contents:
restore_command = '/usr/bin/envdir /etc/wal-e.d/env /usr/local/bin/wal-e wal-fetch "%f" "%p"'
- Start your PostgreSQL server. It might take a little while (it will replay the the WAL files on top of your specified base backup). The
recovery.conf
file will be renamed torecovery.done
once the process is complete. - Get back to work (read: check your data is there!)
I have created an S3 bucket and pushed a backup to the bucket successfully.
-bash-4.2$ envdir /etc/wal-e.d/env wal-e backup-list
wal_e.main INFO MSG: starting WAL-E
DETAIL: The subcommand is "backup-list".
STRUCTURED: time=2018-12-07T19:41:30.441387-00 pid=10841
name last_modified expanded_size_bytes wal_segment_backup_start wal_segment_offset_backup_start wal_segment_backup_stop wal_segment_offset_backup_stop
base_000000010000000000000002_00000032 2018-12-05T21:35:58.000Z 000000010000000000000002 00000032
base_000000010000000000000004_00000032 2018-12-05T21:40:14.000Z 000000010000000000000004 00000032
base_000000010000000300000022_00000032 2018-12-07T18:52:13.000Z 000000010000000300000022 00000032
-bash-4.2$
-bash-4.2$
-bash-4.2$ ls
-bash-4.2$
-bash-4.2$ envdir /etc/wal-e.d/env wal-e backup-fetch /var/lib/pgsql/backups/ LATEST
wal_e.main INFO MSG: starting WAL-E
DETAIL: The subcommand is "backup-fetch".
STRUCTURED: time=2018-12-07T19:42:08.827661-00 pid=10855
wal_e.worker.s3.s3_worker INFO MSG: beginning partition download
DETAIL: The partition being downloaded is part_00000000.tar.lzo.
HINT: The absolute S3 key is test/basebackups_005/base_000000010000000300000022_00000032/tar_partitions/part_00000000.tar.lzo.
STRUCTURED: time=2018-12-07T19:42:09.869089-00 pid=10855
-bash-4.2$
-bash-4.2$ ls
-bash-4.2$
-bash-4.2$
-bash-4.2$ pwd
/var/lib/pgsql/backups
-bash-4.2$
As you can see there is no proper output or no backup downloaded from S3. I am not sure where else to look.
Postgres version: 9.2.24
Wal-e version: 0.9.0
Any help will be greatly appreciated.