charity

Introducing a new series: Post-Mortem Book Reports

Dear fellow systems engineers,

Take a moment and think about the past few years in systems outages and public post mortems. 

What were your favorite outages?  What are the post-mortems that you read that stick with you, months or years or years and years later?  What did you learn from them?

If you are in AWS us-east-1, you probably think back to the Christmas Eve outage of 2013 or the long string of EBS outages.  If you were an early user of mongo sharding, I'm betting the 4sq mongo outage is etched into your brain.  If you run physical data centers and run your own networking or experience lots of DDOS attempts, GitHub post mortems are probably high on your list.

charity

Failure is Not Optional -- it will happen

http://highscalability.com/blog/2010/10/15/troubles-with-sharding-what-can-we-learn-from-the-foursquare.html
https://web.archive.org/web/20110209190434/http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/

https://www.joyent.com/blog/postmortem-for-outage-of-us-east-1-may-27-2014

https://aws.amazon.com/message/5467D2/

http://perfcap.blogspot.com/2012/11/cloud-outage-reports.html

charity

$ for svc in $(aws list 3>&1 1>&2 2>&3 3>&- | sed -e '1,7d' |sed -e 's/\|//g') ; do aws $svc describe-account-attributes 2>/dev/null || echo "not supported for $svc" ; done

charity

mongodata3:SECONDARY> c.find({ $query: {ppoi_0: { $nearSphere: [ -74.5287, 40.1301 ], $maxDistance: 0.02601798524805497 } , deviceType: "ios", channels: { $in: [ "featured_coupons", "special_sales" ] }, appVersion: { $in: [ "3.0", "3.0.1", "3.1", "3.5" ] } }}).explain()
{
        "cursor" : "BasicCursor",
        "isMultiKey" : false,
        "n" : 0,
        "nscannedObjects" : 6484024,
        "nscanned" : 6484024,
        "nscannedObjectsAllPlans" : 6484024,
        "nscannedAllPlans" : 6484024,
        "scanAndOrder" : false,