Runsheet for rebooting the current production morph.io server

Rebooting morph.io production Linode server.md

Rebooting the production morph.io server

1. Log into Linode and go to the page for morph.io
2. Go to the Remote Access tab and to Console Access at the bottom of that page
3. There’s a command there called ‘Lish via SSH’
4. Copy that to your local shell and run it. You should now see the output of the machine, processes running, dying etc.
5. In another shell session ssh to root@morph.io
6. Try the light touch first, run reboot there
   • If you switch to the lish session, you will see processes start shutting down in the linode console access session
   • after a minute that may become stuck on a command
7. Back on the Linode website, hit the big ‘shutdown’ button on the dashboard for morph.io
   • You may get booted from lish, if so run the ‘Lish via SSH’ command again to log into the lindode console.
   • In the lish session, you should now see all the shutdown commands running
8. You will reach the shutdown state and see the shutdown job finished on the linode dashboard
9. Hit the big “Boot” button in the linode dashboard
   • You will see docker, elasticsearch, mysql etc. booting in the console session
   • morph.io comes back online and I could reach it in the browser.
   • In Sidekiq you will see the queue running, with a few retries with Faraday errors. We suspect this is the elasticsearch index building
10. Kill mitmproxy with bash /var/www/current/provisioning/roles/morph-app/files/iptables-morph-remove because of this problem
11. a few minutes later everything should be running smoothly

Add killing mitmproxy step that always trips me up!