Skip to content

Instantly share code, notes, and snippets.

@charity
Created March 8, 2016 18:13
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save charity/d5255ee30ca22496d561 to your computer and use it in GitHub Desktop.
Save charity/d5255ee30ca22496d561 to your computer and use it in GitHub Desktop.
Introducing a new series: Post-Mortem Book Reports
Dear fellow systems engineers,
Take a moment and think about the past few years in systems outages and public
post mortems.
What were your favorite outages? What are the post-mortems that you read that
stick with you, months or years or years and years later? What did you learn
from them?
If you are in AWS us-east-1, you probably think back to the Christmas Eve
outage of 2013 or the long string of EBS outages. If you were an early user of
mongo sharding, I'm betting the 4sq mongo outage is etched into your brain. If
you run physical data centers and run your own networking or experience lots of
DDOS attempts, GitHub post mortems are probably high on your list.
Why do these outages stand out in our minds? Possibly because the ripple
effects affected your own system, but more likely it's because you *learned
something* from the detailed, painstaking, revealing public post mortem that
the affected system published in the wake out of the outage.
For me, the "best" incident retrospectives are the one that make me cringe and
realize, that could have been me. The best ones are where I come away with an
actionable list of things I have learned and can use to protect my own systems.
With this in mind, we are going to start publishing a series of book reports on
both current and classic post-mortems. These reviews are designed to highlight
and recap some of the critical moments in our industry, so we can preserve the
knowledge and collectively level up on our incident reporting skills. Some
questions we will attempt to address are:
* How did affect you? Why does it stand out for you?
* What did you learn from it?
* What was said (or unsaid) that made the post mortem meaningful to you?
* Was something left unclear or dissatisfying? What do you wish they had disclosed,
or been able to disclose?
We appreciate the effort that everyone puts into post mortems and public
retrospectives. Writing retrospectives is hard! Pitching the level of detail
to the right audience is hard. We hope that providing feedback on how your
retrospectives were meaningful for us will help everyone level up on their
skills.
Yours in triumph and in tragedy,
charity.
@jmooo
Copy link

jmooo commented Oct 12, 2016

Hey Charity, did you ever do anything with this? Secondly, have you seen this repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment