Skip to content

Instantly share code, notes, and snippets.

@jyee

jyee/1-plan.md

Last active October 2, 2019 15:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jyee/8a4bc0ea330afc2f7f07166de49880d2 to your computer and use it in GitHub Desktop.
Save jyee/8a4bc0ea330afc2f7f07166de49880d2 to your computer and use it in GitHub Desktop.
Download ZIP
DevOpsDays Hartford Chaos
Raw
1-plan.md

Chaos Planning

  1. Schedule it:

Now!

  1. Choose tests:

Kill MySQL. First we'll delete the pod. Second we'll delete the deployment.

  1. What do we expect to happen?

If we delete the pod, kubernetes should recreate a pod. If we delete the deployment, we'll see a 500 error of some sort. We should get some sort of alerting that mysql is down.

  1. What are we going to do if things go horribly wrong?

Walk off stage, eat cookies.

  1. Share this!
Raw
2-play.md

Chaos Experiment Notes

Things we saw:

We deleted the mysql pod, as it was terminating kubernetes started a new pod. Wordpress frontend still got a 500 error connecting to the DB. The restart was long enough that it triggered an alert in Datadog.

We deleted the deploy, it terminated the pod. The website spun for a while (possibly waiting for timeout) then returned the 500 error.

Things that we didn't expect:

We didnt expect that it would continue to produce 500 errors after the new pod was started.

Things we should fix/change:

We should have a better error message.

Raw
3-report.md

Chaos Lessons Learned

  1. Make Jira tickets from the "Things we should fix/change" We should have a better error message.

  2. Write a summary

We deleted the mysql pod, as it was terminating kubernetes started a new pod. Wordpress frontend still got a 500 error connecting to the DB. The restart was long enough that it triggered an alert in Datadog.

We deleted the deploy, it terminated the pod. The website spun for a while (possibly waiting for timeout) then returned the 500 error.

  1. Share This!

  2. Celebrate!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment