Upgrading PostgreSQL from 9.5 to 9.6 on Ubuntu 16.04

upgrade-postgres-9.5-to-9.6.md

TL;DR

Install Postgres 9.6, and then:

sudo pg_dropcluster 9.6 main --stop
sudo pg_upgradecluster 9.5 main
sudo pg_dropcluster 9.5 main

---

Specifically for using the WAL-E enabled Docker image, make sure WAL-E is turned off:

sudo pkill backupcron

---

Install PostgreSQL:

sudo echo 'deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main' > /etc/apt/sources.list.d/pgdg.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
sudo apt-key add -
  
sudo apt update

sudo apt install postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6

Use dpkg -l | grep postgresql to check which versions of postgres are installed:

i   postgresql                                                        - object-relational SQL database (supported version)
i A postgresql-9.5                                                    - object-relational SQL database, version 9.5 server
i A postgresql-9.6                                                    - object-relational SQL database, version 9.6 server
i A postgresql-client-9.5                                             - front-end programs for PostgreSQL 9.5
i A postgresql-client-9.6                                             - front-end programs for PostgreSQL 9.6
i A postgresql-contrib-9.5                                            - additional facilities for PostgreSQL
i A postgresql-contrib-9.6                                            - additional facilities for PostgreSQL

Run pg_lsclusters, your 9.5 and 9.6 main clusters should be "online".

pg_lsclusters 
Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5432 online postgres /var/lib/postgresql/9.3/main /var/log/postgresql/postgresql-9.5-main.log
9.6 main    5433 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.6-main.log

There already is a cluster "main" for 9.6 (since this is created by default on package installation). This is done so that a fresh installation works out of the box without the need to create a cluster first, but of course it clashes when you try to upgrade 9.5/main when 9.6/main also exists. The recommended procedure is to remove the 9.6 cluster with pg_dropcluster and then upgrade with pg_upgradecluster.

Stop the 9.6 cluster and drop it.

sudo pg_dropcluster 9.6 main --stop

Stop and Upgrade the 9.5 cluster to the latest version.

sudo service postgresql stop
sudo pg_upgradecluster 9.5 main

Note that the upgrade can take a long time for large databases. Eg a 15GB database can take hours (from personal experience). In this case you might want to run the pg_upgrade command manually with the -k -j8 switches. -k creates symlinks to the original database and works directly on that, rather than copying it over. SO THIS IS A DESTRUCTIVE OPERATION - ONLY DO IF YOU HAVE BACKUPS. The -j switch specifies how many CPU cores to use in the upgrade. Use 8 if you have an 8-core CPU.

sudo service postgresql stop
sudo -H -u postgres /usr/lib/postgresql/9.6/bin/pg_upgrade \
   -b /usr/lib/postgresql/9.5/bin \
   -B /usr/lib/postgresql/9.6/bin \
   -d /var/lib/postgresql/9.5/main \
   -D /var/lib/postgresql/9.6/main \
   -o ' -c config_file=/etc/postgresql/9.5/main/postgresql.conf' \
   -O ' -c config_file=/etc/postgresql/9.6/main/postgresql.conf' \
   -k -j8

Your 9.5 cluster should now be "down".

pg_lsclusters 
Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5433 down   postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
9.6 main    5432 down postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log

Check that the upgraded cluster works, then remove the 9.5 cluster.

sudo service postgresql start
sudo pg_dropcluster 9.5 main

If seeing file permission errors upon startup:

 * The PostgreSQL server failed to start. Please check the log output:
2017-03-14 04:06:51 UTC FATAL:  private key file "/var/lib/postgresql/ssl/ssl-cert-snakeoil.key" has group or world access
2017-03-14 04:06:51 UTC DETAIL:  File must have permissions u=rw (0600) or less if owned by the database user, or permissions u=rw,g=r (0640) or less if owned by root.
2017-03-14 04:06:51 UTC LOG:  database system is shut down
   ...fail!

Then simply fix the permissions:

sudo chmod 600 /var/lib/postgresql/ssl/*.key

If using WAL-E, push a new base backup:

sudo -u postgres envdir /etc/wal-e.d/env wal-e backup-push

Lastly update config (if required):

sudo vi /etc/postgresql/9.6/main/postgresql.conf

Change cluster_name = '9.5/main' to cluster_name = '9.6/main'.