DataDog Based Realtime Chef monitoring

README.md

Chef 11 introduces event dispatching mechanism, which emits event at different milestones in Chef run (aka lifecycle) , and users can add their custom handlers to hook into these events. Chef formatters already uses this API. For a complete list of currently available events check this

Following is a bare minimal example for adding a custom handler:

chef_event_handler_foo.rb

require 'chef/event_dispatch/base'
class Foo < Chef::EventDispatch::Base
  # lets say hello world when chef run ends
  def run_completed(node)
     Chef::Log.info('Several species of small furry creatures...')
  end
end

Now, we need to hook this handler, as of now, i could not find an easy way to hook it via config file, hence i have submitted a patch, till it gets reviewed, you can alter the lib/chef/config.rb file and add

event_handlers []

Once this is done, you can add your handler straight from config file

client.rb

require  'chef_event_handler_foo'
event_handlers << Foo.new

DataDog integration

DataDog agent by default runs a statsd daemon locally, what can be more awesome than pushing all these chef events there, thankfully datadog also maintain a ruby client for the statsd api. So, lets modify the handler to push metric in DD/StatsD

require 'chef/event_dispatch/base'
require 'statsd'

class Foo < Chef::EventDispatch::Base

 def statsd
    @statsd||=Statsd.new('localhost', 8125)
  end 
 
 # lets increment the successfully ran metric
  def run_completed(node)
     statsd.increment('chef.client.successfull_run')
  end
end