gorsuch

Shotgun Services

They ride shotgun to the main app, checking in on it, and taking action when things go south. They know how to check in and how to make sure that things are going okay. They know how to call for help if things go south. Maybe they call in to pagerduty for you automatically after N failures over a period of time. Maybe they just mark the instance unhealthy. That's up to you.

Sidekicks could be configured via standard env vars, being told if alerts are warranted on failure, and if they should page, whom deserves it.

Ideally, you have very few of these, as your apps conform to a common convention.

This aims to eliminate the need for a centralized monitoring system, to make things more composable, and still give you the benefits you are looking for.

gorsuch

{:id=>0, :t=>1414954583, :data=>"ping"}
{:id=>9, :t=>1414954583, :data=>"ping"}
{:id=>4, :t=>1414954584, :data=>"ping"}
{:id=>6, :t=>1414954584, :data=>"ping"}
{:id=>5, :t=>1414954584, :data=>"ping"}
{:id=>7, :t=>1414954584, :data=>"ping"}
{:id=>1, :t=>1414954584, :data=>"ping"}
{:id=>8, :t=>1414954584, :data=>"ping"}
{:id=>3, :t=>1414954584, :data=>"ping"}
{:id=>9, :t=>1414954584, :data=>"ping"}

gorsuch

{:id=>8, :t=>1414954582, :data=>"ping"}
{:id=>5, :t=>1414954582, :data=>"ping"}
{:id=>6, :t=>1414954582, :data=>"ping"}
{:id=>1, :t=>1414954582, :data=>"ping"}
{:id=>4, :t=>1414954582, :data=>"ping"}
{:id=>7, :t=>1414954582, :data=>"ping"}
{:id=>3, :t=>1414954582, :data=>"ping"}
{:id=>0, :t=>1414954582, :data=>"ping"}
{:id=>9, :t=>1414954582, :data=>"ping"}
{:id=>2, :t=>1414954582, :data=>"ping"}

gorsuch

#!/usr/bin/env ruby

# build our pipe to be shared between processes
# remember that pipes are unidirectional
r,w = IO.pipe

if fork
  # if we're here, we are the parent

  # close our copy of the read end because we don't use it

gorsuch

---
- hosts: all
  sudo: true
  vars:
    manifest:
      1:
        librato_user: michael.gorsuch@gmail.com
        librato_token: 12345
        sites:
          1:

gorsuch

This document explores possible output formats for the canaryio/canary tool.

First, naked:

$ canary -u http://www.canary.io
canary 2.0.0 http://www.canary.io sites.canary dev 1413338267 0 200 0.685774 0.455677 0.498353 0.622554 23.235.40.133

Next, logfmt:

gorsuch

package main

import (
	"encoding/xml"
	"io/ioutil"
	"log"
	"net/url"

	"github.com/bmizerany/aws4"
)

gorsuch

json = Net::HTTP.get(uri)

JSON.parse(json).map do |item|
  target = item['target']
  datapoints = item['datapoints'].map do |y,x| 
    { 
      x: x,
      y: y || 0
    }
  end

gorsuch

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

gorsuch

package main

import (
	"fmt"
	"strconv"
	"time"

	"github.com/hashicorp/serf/client"
)