Logging events from sensu to logstash

gistfile1.md

To log checks from sensu into logstash, I use the following sensu config:

{
  "handlers": {
    "logstash": {
      "type": "tcp",
      "socket": {
        "host": "logstash.example.com",
        "port": 1234
      }
    }
  }
}

Since this is json output, logstash will be able to consume this easily.

On the logstash side, I use a tcp input to receive the messages:

input {
  tcp {
    codec => json {}
    port => 1234
    tags => ["sensu"]
  }
}

and I then use filters to 'tidy up' some of the message contents (mostly throwing away fields I don't want (optional):

filter {
  if ("sensu" in [tags]) {
      date {
        match => ["[check][issued]", "UNIX" ]
      }
      mutate {
        remove_field => [ "host", "[client][handlers]", "[check][handlers]", "[check][history]", "[client][keepalive][handler]", "[client][keepalive][refresh]", "[client][keepalive][thresholds][critical]", "[client][keepalive][thresholds][warning]", "[client][subscriptions]", "[client][address]" ]
      }
   }
}

This will record a log event each time a check fires... however since tcp handlers don't honour the 'occurrences', 'refresh' or 'interval' fields of a check, this could be as often as every 30 seconds!

Since I only care about logging the first and last event (i.e. when did the service break, and when did it resolve) I use the 'throttle' filter to discard messages which 'repeat' the first message:

filter {
  mutate {
    add_field => { "event_id" => "%{[client][name]}_%{[check][name]}_%{[check][status]}" }
    }

  throttle {
    after_count => 1
    period => 86400
    key => "%{event_id}"
    add_tag => "throttled"
  }
}

This config adds a new field, made up of client.name, check.name and check.status, and then tags all subsequent messages with an identical 'event_id field in any 24 hour period, as 'throttled'.

I then only output those messages where this tag isn't set:

output {
  if "throttled" not in [tags]
    elasticsearch { ... }
  }
}

To log metrics from sensu into logstash, you need to process the message string with grok:

filter {
  grok => {"message", "%{DATA:metric}\\t%{DATA:value}\\t%{INT:unixtime}"}
}

Additionally, be sure to use the line codec for the input, since sensu will send multiple metrics in one message, separated by newlines.