Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Runsheet for rebooting the current production morph.io server

Rebooting the production morph.io server

  1. Log into Linode and go to the page for morph.io
  2. Go to the Remote Access tab and to Console Access at the bottom of that page
  3. There’s a command there called ‘Lish via SSH’
  4. Copy that to your local shell and run it. You should now see the output of the machine, processes running, dying etc.
  5. In another shell session ssh to root@morph.io
  6. Try the light touch first, run reboot there
    • If you switch to the lish session, you will see processes start shutting down in the linode console access session
    • after a minute that may become stuck on a command
  7. Back on the Linode website, hit the big ‘shutdown’ button on the dashboard for morph.io
    • You may get booted from lish, if so run the ‘Lish via SSH’ command again to log into the lindode console.
    • In the lish session, you should now see all the shutdown commands running
  8. You will reach the shutdown state and see the shutdown job finished on the linode dashboard
  9. Hit the big “Boot” button in the linode dashboard
    • You will see docker, elasticsearch, mysql etc. booting in the console session
    • morph.io comes back online and I could reach it in the browser.
    • In Sidekiq you will see the queue running, with a few retries with Faraday errors. We suspect this is the elasticsearch index building
  10. Kill mitmproxy with bash /var/www/current/provisioning/roles/morph-app/files/iptables-morph-remove because of this problem
  11. a few minutes later everything should be running smoothly
@equivalentideas

This comment has been minimized.

Copy link
Owner Author

@equivalentideas equivalentideas commented Oct 1, 2017

Adding runsheet

@equivalentideas

This comment has been minimized.

Copy link
Owner Author

@equivalentideas equivalentideas commented Oct 1, 2017

Add markdown file extension

@equivalentideas

This comment has been minimized.

Copy link
Owner Author

@equivalentideas equivalentideas commented Oct 2, 2017

Add killing mitmproxy step that always trips me up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment