openssl-upgrade.md

Upgrading OpenSSL on production

We will run the following on all production servers:

ssh admin@X.housetrip.com
sudo -i
apt-get update
apt-get upgrade

This gets us:

openssl version -b
built on: Sun Feb 28 23:52:01 UTC 2016

Although the version will still report as:

openssl version
OpenSSL 1.0.1e 11 Feb 2013

This is because the patches are backported under Debian - see https://www.debian.org/security/2016/dsa-3500

The full apt-get update is recommended, rather than just updating OpenSSL, so that all system components are upgraded in lock-step.

After running the above under admin, we will need to restart God (under deploy account) with:

cd app/current
god terminate
god load -c config/god/backend.god

For the workers, this will need to be:

cd app/current
god terminate
god load -c config/god/queue.god

For the scheduler, this will need to be:

cd app/current
god terminate
god load -c config/god/scheduler.god

This will need to be staged as a rolling upgrade so we maintain some production servers as the apt-get completes, as it restarts several services.

It is also recommended that we reboot the servers after the apt-get.

Rollout plan

First we will bring up web 15-36 and deploy the latest code to them, so we can use them if the upgrades don't work for the current web instances. We can use ht scale for that.

We will then:

• take web1-6 out of the load balancer
• run apt-get update on them all
• reboot
• run the god commands listed above (these steps should take no more than 2-3 minutes per server)
• put them back into the load balancer
• check the logs for any errors (plus alerts, DD, NewRelic)
• repeat the above process for 11-14
• repeat the above process for 7 and 8 (internal API)
• repeat the above process for 9 and 10 (internal API)

Upgrade workers:

• run the upgrade on worker1-3
• reboot
• run god commands, as above
• check logs
• repeat with worker4-6

Upgrade scheduler

• run the upgrade on scheduler
• reboot
• run god commands, as above
• check logs

Upgrade utility

• run the upgrade on utility
• reboot
• check logs

Repeat the steps above for web instances for web15-36 and then shut them down again.

Rollback plan

In the case of any issues with the initial apt-get update for web1-5, we will add web15-20 into the load balancer and deal with the issues raised from the apt-get upgrade. However, it is not expected that there will be any issues as apt-get contains stable updates only.