Skip to content

Instantly share code, notes, and snippets.

@human39
Last active October 17, 2019 14:26
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save human39/8b80d51a75c99e64eb42 to your computer and use it in GitHub Desktop.
Save human39/8b80d51a75c99e64eb42 to your computer and use it in GitHub Desktop.
Logstash split field solution.
Problem: We have a log line that includes a perl class that we want to log the class
and method in their respected fields. An example class and method in perl:
Animal::Dog::bark
In this example, "bark" is the method. "Animal::Dog" is the class.
After some searching and hacking, I found a solution that works with Logstash 1.4.2
Assume the input is "Animal::Dog::bark".
For completeness, I'm going to just add my entire configuration file I used for testing.
Comments are included and should explain what's going on.
input {
stdin { }
}
filter {
grok {
match => { "message" => "%{GREEDYDATA:api_class}" }
}
mutate {
# split the field on ::
split => ["api_class" , "::"]
# save the last element of the array as the api_method.
add_field => ["api_method", "%{[api_class][-1]}" ]
}
ruby {
# Go directly to the array and remove the last element.
code => "event['api_class'].pop()"
}
mutate {
# Join together whats left as the class name.
join => ["api_class", "::"]
}
}
output {
stdout { codec => rubydebug }
}
I tried to use mutate's remove_field to remove the last element of the array but it didn't work.
There are tickets created and possibly even a fix in the new version, however, this should continue
to work as long as the ruby filter is around.
The output:
{
"message" => "Animal::Dog::bark",
"@version" => "1",
"@timestamp" => "2014-12-09T13:38:58.178Z",
"host" => "host.example.com",
"api_class" => "Animal::Dog",
"api_method" => "bark"
}
@andres-perez
Copy link

andres-perez commented Oct 17, 2019

Hi. I came here from a stackoverflow answer: https://stackoverflow.com/a/46031524
I appreciate the Ruby code to understand some useful array operations :-)

I think your use case could be solved with a simpler grok pattern though, the trick is to leverage the end anchor "$" in the regexp

grok { match => { "message" => "^%{GREEDYDATA:api_class}::%{WORD:api_method}$" } }

@human39
Copy link
Author

human39 commented Oct 17, 2019

Hi. I came here from a stackoverflow answer: https://stackoverflow.com/a/46031524
I appreciate the Ruby code to understand some useful array operations :-)

I think your use case could be solved with a simpler grok pattern though, the trick is to leverage the end anchor "$" in the regexp

grok { match => { "message" => "^%{GREEDYDATA:api_class}::%{WORD:api_method}$" } }

I totally forgot about this gist! I'm glad it's helping others and inspiring better solutions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment