Skip to content

Instantly share code, notes, and snippets.

@pcn
Created May 25, 2012 05:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save pcn/2785976 to your computer and use it in GitHub Desktop.
Save pcn/2785976 to your computer and use it in GitHub Desktop.
input {
file {
path => ["/var/log/knewton/*/*.log"]
type => "knewton_logback"
}
}
filter {
grok {
type => "knewton_logback"
patterns_path => "/usr/local/share/grok/patterns"
pattern => ["%{JAVALOGBACK}"]
named_captures_only => false
}
date {
type => "knewton_logback"
timestamp => "yyyy-MM-dd HH:mm:ss,SSS"
}
mutate { replace => ["@level", "%{level}"] }
multiline {
type => "knewton_logback"
pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2} " # This goes with
negate => true # this
# pattern => "^(\s+|com|org|net|Caused)" # Doesn't work as well
what => "previous"
}
}
output {
gelf {
host => "graylog2.utility.knewton.net"
facility => "Platform"
}
stdout {
debug => true
}
}
@pcn
Copy link
Author

pcn commented May 25, 2012

The result of one of these going to graylog2 looks like this:

Message aHoQ8_pPQ62nVGbE7f48yg
2012-05-25 05:30:48,815 DEBUG [CuratorService STARTING-SendThread(ec2-23-21-69-105.compute-1.amazonaws.com:2181)] o.apache.zookeeper.ClientCnxn : Got ping response for sessionid: 0x237336d5b950811 after 0ms

In which terms was this message broken to?
From: ip-10-28-115-107
Date: Fri May 25 05:30:54 +0000 2012
Severity: Alert
Facility: Platform
File: //var/log/knewton/substrate/substrate.log:151
MINUTE: 30
JAVALOGBACK: 2012-05-25 05:30:48,815 DEBUG
SECOND: 48
timestamp: 2012-05-25 05:30:48,815
MILLISECOND: 815
HOUR: 05
MONTHNUM: 05
MONTHDAY: 25
YEAR: 2012
severity: DEBUG
Full message:
2012-05-25 05:30:48,815 DEBUG [CuratorService STARTING-SendThread(ec2-23-21-69-105.compute-1.amazonaws.com:2181)] o.apache.zookeeper.ClientCnxn : Got ping response for sessionid: 0x237336d5b950811 after 0ms

Filed in following streams:
Platform_Staging
Permalink

@pcn
Copy link
Author

pcn commented May 25, 2012

The log of what's going on in the stdout output:

2012-05-25T05:39:29.93845 {
2012-05-25T05:39:29.93847          "@source" => "file://ip-10-28-115-107//var/log/knewton/substrate/substrate.log",
2012-05-25T05:39:29.93849            "@type" => "knewton_logback",
2012-05-25T05:39:29.93850            "@tags" => [],
2012-05-25T05:39:29.93851          "@fields" => {
2012-05-25T05:39:29.93852         "JAVALOGBACK" => [
2012-05-25T05:39:29.93853             [0] "2012-05-25 05:39:24,546 DEBUG"
2012-05-25T05:39:29.93854         ],
2012-05-25T05:39:29.93855           "timestamp" => [
2012-05-25T05:39:29.93856             [0] "2012-05-25 05:39:24,546"
2012-05-25T05:39:29.93857         ],
2012-05-25T05:39:29.93858                "YEAR" => [
2012-05-25T05:39:29.93859             [0] "2012"
2012-05-25T05:39:29.93860         ],
2012-05-25T05:39:29.93861            "MONTHNUM" => [
2012-05-25T05:39:29.93862             [0] "05"
2012-05-25T05:39:29.93863         ],
2012-05-25T05:39:29.93864            "MONTHDAY" => [
2012-05-25T05:39:29.93865             [0] "25"
2012-05-25T05:39:29.93866         ],
2012-05-25T05:39:29.93867                "HOUR" => [
2012-05-25T05:39:29.93869             [0] "05"
2012-05-25T05:39:29.93870         ],
2012-05-25T05:39:29.93871              "MINUTE" => [
2012-05-25T05:39:29.93872             [0] "39"
2012-05-25T05:39:29.93873         ],
2012-05-25T05:39:29.93874              "SECOND" => [
2012-05-25T05:39:29.93875             [0] "24"
2012-05-25T05:39:29.93876         ],
2012-05-25T05:39:29.93877         "MILLISECOND" => [
2012-05-25T05:39:29.93878             [0] "546"
2012-05-25T05:39:29.93879         ],
2012-05-25T05:39:29.93880            "severity" => [2012-05-25T05:39:29.93881             [0] "DEBUG"
2012-05-25T05:39:29.93882         ]
2012-05-25T05:39:29.93883     },
2012-05-25T05:39:29.93884       "@timestamp" => "2012-05-25T05:39:24.546000Z",
2012-05-25T05:39:29.93885     "@source_host" => "ip-10-28-115-107",
2012-05-25T05:39:29.93886     "@source_path" => "//var/log/knewton/substrate/substrate.log",
2012-05-25T05:39:29.93888         "@message" => "2012-05-25 05:39:24,546 DEBUG [CuratorService STARTING-SendThread(ec2-23-21-69-105.compute-1.amazonaws.com:2181)] o.apache.zookeeper.ClientCnxn : Got ping response for sessionid: 0x237336d5b950811 after 0ms"

@pcn
Copy link
Author

pcn commented May 25, 2012

Using these defintions:

JAVALOGBACKTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND}),%{MILLISECOND}
JAVALOGBACK (%{JAVALOGBACKTIMESTAMP:timestamp}) (?:%{WORD:severity})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment