Skip to content

Instantly share code, notes, and snippets.

@noelruault

noelruault/potgresql-upgrade.md

Last active June 10, 2024 14:12
Show Gist options
  • Save noelruault/c62c4113d0d40a4502c1a27f4abff405 to your computer and use it in GitHub Desktop.
Save noelruault/c62c4113d0d40a4502c1a27f4abff405 to your computer and use it in GitHub Desktop.
Download ZIP
Upgrade PostgreSQL to v14 (Ubuntu)
Raw
potgresql-upgrade.md

Instructions to Upgrade PostgreSQL to v14 (Ubuntu)

Migration plan

Phase one: Develop automation in a isolated environment

  • Develop a playbook and tests on a similar PostgreSQL environment (created using a back-up from staging) for these tests.
  • Create an image of staging in your cloud provider.
  • Initialise a new VM/Docker/? based on this image. (use a backup from staging to get the upgrade project in contact with the environment)
  • Run the pertinent commands to develop a migration note/script.

Phase two: Ensure the integrity of the migration and make a copy of the database

  • Iterate and test the integration craft a set of end-to-end tests. Validate the crafted end-to-end tests in staging.
  • Execute a snapshot from the database disk that could be used in a restore scenario.

Phase three: Freeze the system and migrate

  • Execute any pre-checks for the project.
  • Stop the traffic and applications that access the database.
  • Do not receive traffic.
  • Stop the proxy.
  • Stop the middleware.
  • Execute the migration (note/script) to carry out the PostgreSQL upgrade.

Phase four: Test the upgrade end-to-end in staging-production

  • Verify the integrity and status of the database. Execute the end-to-end tests.
  • Start the applications connected to the DB and execute tests suites. Execute local unit tests on the upgraded database.

Rollback plan

  • Stop the cluster with PostgreSQL 14.
  • Restore the configuration to PostgreSQL 10.
  • Initialize the database in version 10.
  • Start receiving traffic

Playbook

The next commands were executed in an Ubuntu 18.04.4 LTS.

1. Backup

First create a backup of all the databases for that (You can continue from B if you dont need a backup)

  1. Log in as postgres user sudo su postgres
  2. Create a backup .sql file for all the data you have in all the databases pg_dumpall > backup.sql

2. Pull repos and install PostgreSQL 14

  1. Download ca-certificates

    sudo apt update
sudo apt-get install wget ca-certificates

  2. Add the GPG key and PostgreSQL repository

    wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'

  3. Update apt and get latest postgres version

    sudo apt update
sudo apt -y install postgresql=14\* postgresql-client=14\* postgresql-contrib=14\*

  4. (Optional) Check installed versions dpkg -l 'postgres*' | grep ^i

3. Migrate the data (choose one option)

  1. (Option 1: Manual)

    1. Stop postgreSQL sudo systemctl stop postgresql.service

    2. Make sure you are logged in as postgres user sudo su postgres

    3. And you are running the next commands from a directory that is writable by the postgres user like /var/lib/postgresql

      cd /var/lib/postgresql && ls
# output: 10  14
pg_lsclusters
# Ver Cluster Port Status                Owner    Data directory              Log file
# 10  main    5433 down,binaries_missing postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
# 14  main    5432 online                postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log

    4. Run [pg_upgrade](https://www.postgresql.org/docs/14/pgupgrade.html) to migrate the data

      /usr/lib/postgresql/14/bin/pg_upgrade \
    --old-datadir=/var/lib/postgresql/10/main \
    --new-datadir=/var/lib/postgresql/14/main \
    --old-bindir=/usr/lib/postgresql/10/bin \
    --new-bindir=/usr/lib/postgresql/14/bin \
    --old-options '-c config_file=/etc/postgresql/10/main/postgresql.conf' \
    --new-options '-c config_file=/etc/postgresql/14/main/postgresql.conf'

    5. Edit pg_hba.conf and change peer to trust to allow incoming connections

      sudo vim /etc/postgresql/14/main/pg_hba.conf
# change 'peer' to 'trust' for all/all for both host and local / 127.0.0.1/32

      • Example file result

        # TYPE  DATABASE        USER     ADDRESS                 METHOD

# Database administrative login by Unix domain socket
local   all             postgres                         trust # peer -> trust

# "local" is for Unix domain socket connections only
local   all             all                              trust # peer -> trust  
# IPv4 local connections:
host    all             all      127.0.0.1/32            trust # scram-sha-256 -> trust
# IPv6 local connections:
host    all             all      ::1/128                 trust # scram-sha-256 -> trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                               peer
host    replication     all       127.0.0.1/32            scram-sha-256
host    replication     all       ::1/128                 scram-sha-256

    6. Update anything required by the pg_upgrade command (like update_extensions.sql)

      Note: When upgrading, the executable outputted:

      Your installation contains extensions that should be updated with the ALTER EXTENSION command.
The file update_extensions.sql, when executed by psql by the database superuser will update these extensions.

      Now is the time to do so. 🙂

    7. Switch to regular user exit

    8. Swap the ports and delete the old version.

           sudo vim /etc/postgresql/14/main/postgresql.conf
     # change port to 5432
     sudo vim /etc/postgresql/10/main/postgresql.conf
     # change port to 5433

  2. (Option 2: Automatic. Use at your own risk 😵‍💫)

    1. Delete the PostgreSQL brand new cluster to migrate the old one instead

      sudo pg_dropcluster 14 main --stop # Docs: http://manpages.ubuntu.com/manpages/trusty/man8/pg_dropcluster.8.html

    2. Upgrade / Migrate

      sudo pg_upgradecluster -v 14 10 main # Docs: http://manpages.ubuntu.com/manpages/trusty/man8/pg_upgradecluster.8.html
# pg_upgradecluster [-v newversion] oldversion name [newdatadir]

4. Restart and check version

  1. Start the postgresql service sudo systemctl restart postgresql.service
  2. Log in as postgres user sudo su postgres
  3. Check your new postgres version psql -c "SELECT version();"

5. Cleanup

A.k.a cleanup up the old version's mess (manual version of pg_dropcluster stated in 5.b)

  1. (Option 1: Manual)

    1. Return as a normal(default user) exit

    2. Uninstall postgres packages if present dpkg -l 'postgres*' | grep ^i

      sudo apt-get remove postgresql-10 postgresql-client-10

    3. Remove the old postgresql directory if present sudo rm -rf /etc/postgresql/10/

    4. Login as postgres user sudo su postgres

    5. Go inside psql folder and remove old data (/var/lib/postgresql)

      sudo rm -rf '/var/lib/postgresql/10/main' # by the time this guide was written: equivalent to ./delete_old_cluster.sh
rmdir 10                                  # remove folder 10 if empty
sudo rm delete_old_cluster.sh                  # remove script file, equivalent to the first line of this snippet and not useful anymore

  2. (Option 2: Automatic. Use at your own risk 😵‍💫)

    sudo pg_dropcluster 10 main --stop

Note: If everything works well in 2-3, we dont have to apply the backup as we have already migrated the data from the older version to the newer version, the backup is just in case anything goes wrong.

Sources

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment