I had 4GB of logfiles which consisted of mixed line-based content and JSON data. So i combined an evented JSON Parser with a regular linebased search. The logfile was generated by zabbix-server, set to the most verbose loglevel. We used this script to generate a list of all hosts and items that were still sending data to this machine.

searchLogForJSON.rb

require 'yajl'
require 'yajl/json_gem'

@hostlist = {}

def object_parsed(obj)
  puts  Yajl::Encoder.encode(obj)

  obj.each do |item|
    if item['request'] == 'sender data'
      item['data'].each do |data|
        @hostlist[data['host']] ||= []
        @hostlist[data['host']] << data['key'] unless @hostlist[data['host']].include? data['key']
      end
    elsif item['request'] == 'active checks'
        @hostlist[item['host']] ||= []
    else
      STDERR << "New Request Type: #{item['request']}"
    end
  end
end

def skip_to_trapper_input filehandle
  while line = filehandle.readline
    if line["Trapper got "]
      filehandle.pos = filehandle.pos-4
      return
    end
  end
end


begin
  File.open('zabbix_server_condensed.1.log', 'rb:ASCII-8BIT') do |f|
    
    while f.read(1) != ""

      skip_to_trapper_input(f)

      puts "// #{f.lineno} / #{f.pos}"
      @parser = Yajl::Parser.new()
      @parser.on_parse_complete = method(:object_parsed)

      begin  
        @parser.parse(f)
      rescue Yajl::ParseError => dontcare
  #      puts dontcare
      end
    end
  end

rescue EOFError => e
  puts e
end

puts "============================\n"
puts JSON::pretty_generate(@hostlist)