A parser for Heroku log messages

README.md

A parser for Heroku log messages

This parser uses the Parslet to define rules for parsing a log message in the simple key=value format used by Heroku, eg:

at=error code=H12 desc="Request timeout" method=GET path="/foo" dyno=web.3 connect=1ms service=30000ms status=503 bytes=0

Given a message in the above format, the parser returns a hash of the key/value pairs:

message = 'at=error code=H12 desc="Request timeout" method=GET...'
Parser.parse(message)
#=> { at: "error", code: "H12", desc: "Request timeout", method: "GET, ... }

Why a custom parser?

Some of the values are quoted strings that contains spaces (desc="Request timeout") so you can't just use message.split(" ") or a regular expression. I was unable to find an existing parser for this format so, execution speed not being a critical factor, I explored Ruby parser generators. Parslet features a very readable DSL and was very easy to work with.

parser.rb

require 'parslet'

class Parser < Parslet::Parser
  # introduce some supporting characters
  rule(:space)  { str(' ') }
  rule(:quote)  { str('"') }
  rule(:equals) { str('=') }

  # handle values with quotes and spaces ("Request timeout")
  rule(:quoted_value) do
    quote >> (match('[^"]')).repeat(1).as(:value) >> quote
  end

  # handle simple values without quotes like (error)
  rule(:unquoted_value) do
    match('[^ "]').repeat(1).as(:value)
  end

  # define a key-value pair
  rule(:key) { match('[a-z_]').repeat(1).as(:key) }
  rule(:value) { quoted_value | unquoted_value }
  rule(:pair) { key >> equals >> value >> space.maybe }

  # the entire log message is comprised of key/value pairs
  rule(:pairs) { pair.repeat }
  root(:pairs)

  def self.parse(message)
    # Given a log message like this:
    #
    #   code=NY name="New York"
    #
    # Parser.new.parse() returns an array of key/value hashes:
    #
    #   [
    #     { key: 'code', value: 'NY'},
    #     { key: 'name', value: 'New York'}
    #   ]
    #
    key_value_pairs = self.new.parse(message)

    # We then transform that array into 1 simple hash, eg: 
    #
    #   { code: 'NY', name: 'New York' }
    #
    key_value_pairs.inject({}) do |hash, pair|
      key = pair[:key].to_s
      value = pair[:value].to_s
      hash.merge(key => value)
    end
  end
end