Runsheet for rebooting the current production morph.io server
Rebooting the production morph.io server
- Log into Linode and go to the page for morph.io
- Go to the Remote Access tab and to Console Access at the bottom of that page
- There’s a command there called ‘Lish via SSH’
- Copy that to your local shell and run it. You should now see the output of the machine, processes running, dying etc.
- In another shell session ssh to firstname.lastname@example.org
- Try the light touch first, run
- If you switch to the lish session, you will see processes start shutting down in the linode console access session
- after a minute that may become stuck on a command
- Back on the Linode website, hit the big ‘shutdown’ button on the dashboard for morph.io
- You may get booted from lish, if so run the ‘Lish via SSH’ command again to log into the lindode console.
- In the lish session, you should now see all the shutdown commands running
- You will reach the shutdown state and see the shutdown job finished on the linode dashboard
- Hit the big “Boot” button in the linode dashboard
- You will see docker, elasticsearch, mysql etc. booting in the console session
- morph.io comes back online and I could reach it in the browser.
- In Sidekiq you will see the queue running, with a few retries with Faraday errors.
We suspect this is the elasticsearch index building
- Kill mitmproxy with
bash /var/www/current/provisioning/roles/morph-app/files/iptables-morph-remove because of this problem
- a few minutes later everything should be running smoothly