Exporting chef-client metrics to Prometheus

README.md

Prometheus metrics from chef_client

About

This shows how to create a metrics file for the textfile collector at the end of a chef-client run, and collect some metrics. The approach is to register a report handler with Chef, which gets executed after everything else.

Status

Consider this a rough sketch – details of how Node Exporter is installed are elided, and we cannot provide any support for this. May it guide you on your way.

Dependencies

This uses the chef_handler cookbook. It is tested with version 1.1.7 of this cookbook. The handler goes into files/default/, the recipe into recipes/. Adapt the recipe to how and where you install Node Exporter.

LICENSE.md

The MIT License (MIT)

Copyright © 2017 SoundCloud Ltd.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

prometheus_handler.rb

require 'chef/handler'
require 'prometheus/client/push'

class PrometheusHandler < Chef::Handler
  attr_reader :textfile, :registry

  JOB = 'chef_client'

  def initialize(textfile)
    @textfile  = textfile
    @registry = Prometheus::Client.registry
  end

  def report
    labels = { zone: node.zone || 'unknown' }

    collect_time_metrics(labels)
    collect_role_metrics(labels)
    exception ? collect_error_metrics(labels) : collect_resource_metrics(labels)

    if textfile
      File.write(textfile, Prometheus::Client::Formats::Text.marshal(registry))
    end
  rescue => ex
    Chef::Log.error("PrometheusHandler: #{ex.inspect}")
  end

  private

  def collect_time_metrics(labels)
    registry
      .gauge(:chef_client_duration_ms,
             'The duration of chef-client run (milliseconds).')
      .set(labels, run_status.elapsed_time * 1e+3)

    registry
     .gauge(:chef_client_last_run_timestamp_seconds,
             'The unix timestamp of the finish of the last chef-client run.')
     .set(labels, run_status.end_time.to_i)
  end

  def collect_error_metrics(labels)
    registry
      .gauge(:chef_client_errors_count, 'The number of chef-client errors.')
      .set(labels, 1)
  end

  def collect_resource_metrics(labels)
    registry
      .gauge(:chef_client_resources_count,
            'The number of all resources in the run context.')
      .set(labels, run_status.all_resources.size)

    registry
      .gauge(:chef_client_updated_resources_count,
            'The number of all updated resources in the run context.')
      .set(labels, run_status.updated_resources.size)
  end

  def collect_role_metrics(labels)
    roles = registry.gauge(
      :chef_client_roles,
      'The list of all roles currently applied to the node.')

    node.roles.each do |role|
      roles.set({ role: role }.merge(labels), 1)
    end
  end
end

recipe.rb

# assuming there is some cookbook that installs Node Exporter …
include_recipe 'node_exporter'

textfile = File.join('/path/to/textfile_metrics.d/', 'chef-client.prom')
include_recipe 'chef_handler'

handler = 'prometheus_handler.rb'
path    = File.join(node['chef_handler']['handler_path'], handler)

chef_gem 'prometheus-client'

# `run_action` causes this to be executed in the first phase of the chef-client
# run, so that the handler is installed as soon as possible.
cookbook_file path do
  source handler
  mode '0755'
  action :nothing
end.run_action(:create)

chef_handler 'PrometheusHandler' do
  source path
  arguments textfile
  action :nothing
end.run_action(:enable)