Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?



Edward Muller





AKA System Heuristics Herald

Polls the system in different ways at a given interval,

relaying the collected metrics somewhere

Yes, it's a backronym


Use the same metrics path that heroku uses for apps

but for systems

What is the app metric path?

app ⇢ logs ⇢ logplexl2metlibrato

(anything can drain)

What is the old system metric path?

collectdwrite_http pluginbackstopgraphite | librato

What is new system metrics path?

shh ⇢ logs ⇢ logplexl2metlibrato

(anything can drain)

Notice what's the same?

logs ⇢ logplexl2metlibrato


  1. No real difference between apps & systems (less cognitive overhead)
  2. Not running collectd anymore
  3. Can retire backstop
  4. Single metric sink/source: libratro (at least for now)
  5. Which means no more Graphite to maintain
  6. 12 Factor isn't just for apps.

But Why Not collectd, munin, etc...?

  • I wanted to learn/use Go

  • I believe existing solutions are

    • antiquated
    • hard to manage
    • hard to modify
    • overly complicated

Additional Considerations

I dislike purpose built config files.

shh uses environment variables ONLY.

These are usually encoded via envdir

(some 12 Factor influence here)

How does it work?

Main loop ticks pollers every interval as Goroutines

Pollers send measurements via a channel

Outputters receive measurments from that channel and do what they will with them.


A sample of something at a point in time

type Measurement struct {
  When   time.Time
  Poller string
  What   []string
  Value  interface{}


generate measurements each tick via Poll()

type Poller interface {
  Name() string
  Poll(tick time.Time)

Current Pollers

  • Conntrack
  • CPU
  • Df
  • Disk
  • Listen (simple plugin interface)
  • Load
  • Memory
  • Network Interface
  • Ntpdate
  • Processes
  • Self


type Outputter interface {

Current Outputters

  • Standard Out Raw
  • Standard Out Derivative
  • Librato

Integers are counters

Floats are guages



  1. Go is new(ish).
  2. Taking naive approaches to pollers in many cases.
  3. Possible over use of channels
  4. Linux Only (atm)
  5. .....


  • More Pollers
  • More Outputters
  • More adoption



Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment