Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Logstash ruby filter to flatten nested fields to dotted names. I don't recommend this for many reaons (performance, json parsing pain, elasticsearch indexing problems, etc), but sometimes your data pipeline requires this sort of silliness.
filter {
ruby {
path => 'flatten_nested_field.rb'
script_params => {
"source" => "observer"
"delimiter" => "."
}
}
}
# Flatten one layer of document nesting
def register(params)
@source = params["source"]
@delimiter = params.fetch("delimiter", ".")
end
def filter(event)
if event.get(@source).nil? or !event.get(@source).kind_of?(Hash) or event.get(@source).empty?
event.tag("nested_field_does_not_exist-#{@source}")
return [event]
end
ev_hash = event.to_hash
ev_hash["#{@source}"].each do |key, value|
ev_hash["#{@source}#{@delimiter}#{key}"] = value
end
ev_hash.delete("#{@source}")
event.initialize( ev_hash )
return [event]
end
## validation tests
test "when event has key" do
parameters {{"source" => "observer"}}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"} }}
expect("subfields are flattened") {|events| events.first.get("observer.name") == "nsm"}
end
test "when has custom delimiter" do
parameters {{"source" => "observer", "delimiter" => "|" }}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"} }}
expect("subfields are flattened with delimiter") {|events| events.first.get("observer|name") == "nsm"}
end
test "when event doesn't have key" do
parameters {{"source" => "host"}}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"} }}
expect("tags as not found") {|events| events.first.get("tags").include?("nested_field_does_not_exist-host")}
end
test "when event doesn't have hash at key" do
parameters {{"source" => "host"}}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"}, "host" => "foo" }}
expect("tags as not found") {|events| events.first.get("tags").include?("nested_field_does_not_exist-host")}
end
test "when event has array at key" do
parameters {{"source" => "host"}}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"}, "host" => [] }}
expect("tags as not found") {|events| events.first.get("tags").include?("nested_field_does_not_exist-host")}
end
test "when event has empty hash at key" do
parameters {{"source" => "host"}}
in_event {{ "observer" => {"name" => "nsm", "version" => "1.0.3"}, "host" => {} }}
expect("tags as not found") {|events| events.first.get("tags").include?("nested_field_does_not_exist-host")}
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment