Skip to content

Instantly share code, notes, and snippets.

@xorpaul
Last active December 22, 2015 15:59
Show Gist options
  • Save xorpaul/6496661 to your computer and use it in GitHub Desktop.
Save xorpaul/6496661 to your computer and use it in GitHub Desktop.
logstash 1.2.1 grok parse failure
# java -jar /data/logstash/logstash-1.2.1-flatjar.jar agent -f /etc/logstash/agent/config/rsyslogs2redis-shipper.conf                                                    
Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. 
 For more information on plugin milestones, see http://logstash.net/docs/1.2.1/plugin-milestones {:level=>:warn}
Using milestone 2 output plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.2
.1/plugin-milestones {:level=>:warn}
Grok regexp threw exception {:exception=>"invalid byte sequence in UTF-8", :field=>"message", :grok_pile=>#<Grok::Pile:0x1077a833 @patterns={"NETSCREENSESSIONLOG"=>
[...]
92, 48, 48, 52, 124, 200, 62, 62, 32, 40, 115, 104, 111, 114, 116, 101, 110, 101, 100, 41, 32, 91, 110, 111, 32, 116, 114, 97, 99, 101, 93], :level=>:warn}                                     [8/611]
Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (GeneratorError) source sequence is illegal/malformed utf-8
        at json.ext.GeneratorMethods$RbHash.to_json(json/ext/GeneratorMethods.java:71)
        at LogStash::Event.to_json(file:/data/logstash/logstash-1.2.1-flatjar.jar!/logstash/event.rb:169)
        at LogStash::Outputs::Redis.receive(file:/data/logstash/logstash-1.2.1-flatjar.jar!/logstash/outputs/redis.rb:150)
        at RUBY.initialize((eval):17)
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:255)
        at LogStash::Pipeline.output(file:/data/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:247)
        at RUBY.outputworker(file:/data/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:212)
        at RUBY.start_outputs(file:/data/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:140)

# cat /etc/logstash/agent/config/rsyslogs2redis-shipper.conf 
input {
  syslog {
    port => 5544
  }
}

output {
    #stdout { debug => false debug_format => "json"}
    redis { host => "logstash-redis.domain.tld" data_type => "list" key => "logstash" }
}
@xorpaul
Copy link
Author

xorpaul commented Sep 18, 2013

It has to do with non UTF-8 log messages. The numbers at the end of this log message is a byte array containing the actual log message.
You can use ruby to decode it:

[116, 114, 97, 99, 101, 93].pack("C*")

I did a downgrade to logstash 1.2.0, which still throws errors when it encounters such log events, but doesn't terminate like logstash 1.2.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment