CodeMotion Rome - Chaos experiment documents

1-before.md

Chaos Planning

1. Schedule it:

   Now!

2. Choose tests:

   Kill Mysql!

3. What do we expect to happen?

   • Everything explodes
   • White screen with 500 error code
   • Maybe a maintenance page
   • Timeout connection
   • We should get an alert that mysql is down

4. What are we going to do if things go horribly wrong?

   Walk away, go to a bar, and get a beer!

5. Share this!

2-experiment.md

Chaos Experiment Notes

Things we saw:

We killed the pod and it respawned quickly so we didn't see anything. So we deleted the mysql deployment. This resulted in an error screen that the database was unavailable.

Things that we didn't expect:

• We didn't get a maintenance page or other user-friendly error screen.
• Alerts were misconfigured, so we didn't receive alerts immediately.
• Alerts didn't include action steps or links to relevant dashboards.

Things we should fix/change:

• We should set up a better outage page to maintain the look/feel of the site and other options for users when the DB is down.
• We need to update the alerts so they are sent sooner when the DB goes down. We also need to add action steps.

3-after.md

Chaos Lessons Learned

1. Make Jira tickets from the "Things we should fix/change"

2. Write a summary

   We killed the pod and it respawned quickly so we didn't see anything. So we deleted the mysql deployment. This resulted in an error screen that the database was unavailable. We learned that the current error screen can be improved and that our existing alerts need updating in window time and messaging.

3. Share This!

4. Celebrate!