Skip to content

Instantly share code, notes, and snippets.

@andsel
Created July 19, 2021 09:37
Show Gist options
  • Save andsel/209289fa8212a1754c12bb4d31843dfa to your computer and use it in GitHub Desktop.
Save andsel/209289fa8212a1754c12bb4d31843dfa to your computer and use it in GitHub Desktop.
Logstash ruby filter - remove leafs of a path selected by regexp
# input sample: {"name": "John", "outer": {"inner": {"leaf_1": "a leaf 1", "leaf_2": "a leaf 2", "leaf_3": "a leaf 3"}}}
input {
stdin {
codec => json
}
}
filter {
ruby {
path => "${PWD}/regexp_dropper.rb"
script_params => {
source => "[outer][inner]"
field_regexp => "leaf_[1, 2]"
}
}
}
output {
stdout {
codec => rubydebug
}
}
#Usage:
# source: is path tot he parent node where to delete the fields, must be in format [field_parent][field_middle]
# field_regexp: is regexp that has to be matched to remove a field, example field_{1, 2} will remove [field_parent][field_middle][field_1] and [field_parent][field_middle][field_2]
def register(params)
params = params.dup
@source = params.delete("source")
@field_regexp = params.delete("field_regexp")
params.empty? || report_configuration_error("unknown script parameter(s): #{params.keys}.")
end
def filter(event)
source_map = @source.nil? ? event.to_hash : event.get(@source)
return [event] unless source_map
fail('source not a key/value map') unless source_map.kind_of?(Hash)
source_map.keys.each do |key|
if key =~ Regexp.new(@field_regexp)
# remove if match the leaf name
full_path = "#{@source}[#{key}]"
#logger.info("field to remove: #{full_path}")
event.remove(full_path)
end
end
[event]
end
def report_configuration_error(message)
raise LogStash::ConfigurationError, message
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment