Skip to content

Instantly share code, notes, and snippets.

View jssjr's full-sized avatar

Scott Sanders jssjr

View GitHub Profile

Keybase proof

I hereby claim:

  • I am jssjr on github.
  • I am jssjr (https://keybase.io/jssjr) on keybase.
  • I have a public key whose fingerprint is 2081 E230 3001 6508 8847 1BBF A0A8 DB0F D1EA B94D

To claim this, I am signing this object:

The Lifecycle of an Outage

Scott Sanders -- jssjr@github.com


So we're all here at Monitorama, and it's awesome to see so many incredible people in one place focusing on such an important topic. I'd like to talk a bit about outage lifecycles and how the monitoring and alerting tools we're familiar with can be woven into processes that enable confidence.

When an incident occurs, we typically have an increased risk of an outage. How we structure our initial response, our decision making process, and our communication directly affects the impact that this incident will have. We need to think critically about our ability to quickly resolve any problems and reduce the risk of future incidents.

This file has been truncated, but you can view the full file.
26798 1035 (sshd) access /proc/self/oom_score_adj read: 0 write: 2
26802 1035 (sshd) iotime /proc/self/oom_score_adj time: 3
27062 1035 (sshd) access /etc/ld.so.cache read: 0 write: 0
27099 1035 (sshd) access /lib/x86_64-linux-gnu/libwrap.so.0 read: 832 write: 0
27101 1035 (sshd) iotime /lib/x86_64-linux-gnu/libwrap.so.0 time: 2
27127 1035 (sshd) access /lib/x86_64-linux-gnu/libpam.so.0 read: 832 write: 0
27128 1035 (sshd) iotime /lib/x86_64-linux-gnu/libpam.so.0 time: 1
27155 1035 (sshd) access /lib/x86_64-linux-gnu/libselinux.so.1 read: 832 write: 0
27156 1035 (sshd) iotime /lib/x86_64-linux-gnu/libselinux.so.1 time: 1
27180 1035 (sshd) access /usr/lib/libck-connector.so.0 read: 832 write: 0
@jssjr
jssjr / IMG_4044.jpg
Created June 25, 2015 17:28
Atom Editor!
IMG_4044.jpg
@jssjr
jssjr / notify.sh
Created October 21, 2011 14:54
Enhanced Nagios Email Notification Script
#!/usr/local/bin/bash
NAGIOS_BASEURL="https://mynagios.server.tld"
NAGIOS_CGIURL="${NAGIOS_BASEURL}/nagios/cgi-bin/"
function hostgroup_links {
URL="${NAGIOS_CGIURL}status.cgi?hostgroup="
[ -n "$*" ] && echo $* | tr ',' "\n" | sed -e 's,^, ,' -e "s,^ \([^ ]*\)$, \1 (${URL}\1)," || echo ' NONE';
}
@jssjr
jssjr / cfg---hosts---_groups---class-servers.cfg
Created September 1, 2012 02:48
Nagios inheritance patters
define hostgroup{
hostgroup_name servers
alias Class - Servers
}
@jssjr
jssjr / opscode-cookbook-mirror.sh
Created November 20, 2012 17:22
Mirror the opscode-cookbook repos
#!/usr/bin/env sh
### Mirror all of the opscode-cookbooks repositories
trap "exit 1" INT TERM # exit gracefully if we're in the main loop
repos=$(curl -s -o- https://api.github.com/users/opscode-cookbooks/repos\?per_page\=1000 \
| ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each{|r| puts r["git_url"]}')
for repo in $repos ; do
@jssjr
jssjr / Rakefile
Created January 22, 2013 18:36
Rake tasks for auto testing and uploading Chef cookbooks in Jenkins
require 'rubygems'
require 'chef'
require 'json'
# Load constants from rake config file.
require File.join(File.dirname(__FILE__), 'config', 'rake')
# Detect the version control system and assign to $vcs. Used by the update
# task in chef_repo.rake (below). The install task calls update, so this
# is run whenever the repo is installed.
# arg1 = "String to expand as the run command, $key will be interpretted as the item name"
# arg2..n = items to iterate across
function tmux_windows() {
session="tmux-cmd-$$"
tmux new-session -d -s $session
cmd=$1 ; shift
index=1
while [ ! -z $1 ] ; do
index=$((++index))
export var=$1 ; shift
@jssjr
jssjr / install.sh
Created April 3, 2013 01:43
GreeBSD Veewee/Bento install script
#!/bin/bash
# This is the current stable release to default to, with Omnibus patch level (e.g. 10.12.0-1)
# Note that the chef template downloads 'x.y.z' not 'x.y.z-r' which should be a duplicate of the latest -r
use_shell=0
prerelease="false"
# Check whether a command exists - returns 0 if it does, 1 if it does not
exists() {