allspaw.md

Writing Post Mortems - John Allspaw

This talk is also known as ‘PostMortem Facilitation: Theory and Practice of "New View" Debriefings’ Background reading for attendees

large-scale-design.md

Non-Abstract Large Scale Design Workshop

Essential Reading

• Software Engineering Advice from building Large Scale Systems, Jeff Dean
• Video: Building Software Systems At Google and Lessons Learned
• The Consensus Protocols series by Henry Robinson:
  • Two­phase commit
  • Three­phase commit
  • Paxos
• Bigtable: A Distributed Storage System for Structured Data, by Jeff Dean et al
• The Google File System, by Sanjay Ghemawat et al
• Dynamo: Amazon's Highly Available Key­Value store

Recommended Reading

• Notes on Distributed Systems for Young Bloods by Jeff Hodges
• Fallacies of Distributed Computing Explained by Arnon Rotem-­Gal-­Oz
• 7 Ways to Handle Concurrency in Distributed Systems by Colin Scott
• Latency Trends by Colin Scott
• CAP Twelve Years Later: How the "Rules" Have Changed by Eric Brewer
• Chord: A Scalable Peer­to­Peer Lookup Protocol
• How to build a highly ­available system using Distributed Consensus, Butler Lampson

Further Reading (Optional)

• Distributed Systems and Parallel Computing at research.google.com
• Breaking the Mishap Chain: Human Factors Lessons Learned from Aerospace Accidents and Incidents in Research, Flight Test, and Development by Peter W. Merlin

prometheus.md

Familiarize yourself with Prometheus.

A good starter is the SoundCloud blog post

Next, you should look at the “Week of Monitoring” blog posts by Boxever

Finally, you find comprehensive documentation at http://prometheus.io.

Bring a Linux or Mac laptop. (Optionally, have a running Linux in a VM on a Mac or on any other platform.)

Install the most recent binaries for your platform from https://github.com/prometheus/prometheus/releases (Prometheus server) and https://github.com/prometheus/node_exporter/releases (Node Exporter). If you have Linux running in a VM, install the node exporter there. (The node exporter works much better on Linux.)

An example application to be instrumented will be offered in Go and Python.

If you prefer Python, make sure you have a working Python development environment on your laptop and install the Python client library by running

pip install prometheus_client

If you prefer Go, make sure you have a working Go1.4.x development on your machine and clone https://github.com/prometheus/client_golang into $GOPATH/src/github.com/prometheus/client_golang. Make sure the following works:

cd $GOPATH/src/github.com/prometheus/client_golang/examples/simple
go get
go build