SF Metrics Meetup Presentation.md

SHH

Who

Edward Muller

@freeformz

icanhazdowntime.org

What?

SHH

AKA System Heuristics Herald

Polls the system in different ways at a given interval,

relaying the collected metrics somewhere

github.com/freeformz/shh

Yes, it's a backronym

Goal

Use the same metrics path that heroku uses for apps

but for systems

What is the app metric path?

app ⇢ logs ⇢ logplex ⇢ l2met ⇢ librato

(anything can drain)

What is the old system metric path?

collectd ⇢ write_http plugin ⇢ backstop ⇢ graphite | librato

What is new system metrics path?

shh ⇢ logs ⇢ logplex ⇢ l2met ⇢ librato

(anything can drain)

Notice what's the same?

logs ⇢ logplex ⇢ l2met ⇢ librato

Winning!

1. No real difference between apps & systems (less cognitive overhead)
2. Not running collectd anymore
3. Can retire backstop
4. Single metric sink/source: libratro (at least for now)
5. Which means no more Graphite to maintain
6. 12 Factor isn't just for apps.

But Why Not collectd, munin, etc...?

• I wanted to learn/use Go

• I believe existing solutions are

  • antiquated
  • hard to manage
  • hard to modify
  • overly complicated

Additional Considerations

I dislike purpose built config files.

shh uses environment variables ONLY.

These are usually encoded via envdir

(some 12 Factor influence here)

How does it work?

Main loop ticks pollers every interval as Goroutines

Pollers send measurements via a channel

Outputters receive measurments from that channel and do what they will with them.

Measurements

A sample of something at a point in time

type Measurement struct {
  When   time.Time
  Poller string
  What   []string
  Value  interface{}
}

Pollers

generate measurements each tick via Poll()

type Poller interface {
  Name() string
  Exit()
  Poll(tick time.Time)
}

Current Pollers

• Conntrack
• CPU
• Df
• Disk
• Listen (simple plugin interface)
• Load
• Memory
• Network Interface
• Ntpdate
• Processes
• Self

Outputters

type Outputter interface {
  Start()
}

Current Outputters

• Standard Out Raw
• Standard Out Derivative
• Librato

Integers are counters

Floats are guages

Demo

Problems

1. Go is new(ish).
2. Taking naive approaches to pollers in many cases.
3. Possible over use of channels
4. Linux Only (atm)
5. .....

Future

• More Pollers
• More Outputters
• More adoption

See http://github.com/freeformz/shh/issues

Questions