Skip to content

Instantly share code, notes, and snippets.

@ijokarumawak
Last active June 15, 2022 09:34
Show Gist options
  • Save ijokarumawak/5a9bab0137856aabd19b314caa86e7fa to your computer and use it in GitHub Desktop.
Save ijokarumawak/5a9bab0137856aabd19b314caa86e7fa to your computer and use it in GitHub Desktop.
Use Logstash ruby filter to rename field name based on value type to avoid Elasticsearch mapping conflict

This is Logstash ruby filter example to rename field names within a specified hash, based on each value type. The original question was asked at this Elastic Discuss.

How to test the filter

With -t, the filter can be tested:

logstash -e "filter { ruby { path => '/{path_of_the_ruby_script}/rename_by_data_type.rb' script_params => { 'field' => 'the_name_of_target_field' } } }" -t

Output:

[2022-06-15T18:14:42,729][INFO ][logstash.filters.ruby.script] Test run complete {:script_path=>"/{path_of_the_ruby_script}/rename_by_data_type.rb", :results=>{:passed=>2, :failed=>0, :errored=>0}}
Configuration OK

Usage

Specify the script file path, and a field name of the target hash field.

filter {
  ruby {
    path => '/{path_of_the_ruby_script}/rename_by_data_type.rb'
    script_params => { 'field' => 'the_name_of_target_field' }
  }
}

Example input and output

The data is shown in JSON format for illustration purpose.

Input:

[
{"values": {"fieldA": "a", "fieldB": "b", "fieldC": "c"}},
{"values": {"fieldA": 1, "fieldB": 2, "fieldC": 3}}
]

Output:

[
{"values": {"fieldA_String": "a", "fieldB_String": "b", "fieldC_String": "c"}},
{"values": {"fieldA_Integer": 1, "fieldB_Integer": 2, "fieldC_Integer": 3}}
]
def register(params)
@field = params["field"]
end
def filter(event)
values = event.get(@field)
renamed = {}
values.each do |key, value|
renamed[key + "_" + value.class.to_s()] = value
end
event.set(@field, renamed)
return [event]
end
test "rename fields by data type - String" do
parameters do
{ "field" => "json_values" }
end
in_event {
{"json_values" => {
"fieldA" => "a",
"fieldB" => "b",
"fieldC" => "c"
}}
}
expect("updated") do |events|
events[0].get("json_values")["fieldA_String"] == "a"
events[0].get("json_values")["fieldB_String"] == "b"
events[0].get("json_values")["fieldC_String"] == "c"
end
end
test "rename fields by data type - Integer" do
parameters do
{ "field" => "json_values" }
end
in_event {
{"json_values" => {
"fieldA" => 1,
"fieldB" => 2,
"fieldC" => 3
}}
}
expect("updated") do |events|
events[0].get("json_values")["fieldA_Integer"] == 1
events[0].get("json_values")["fieldB_Integer"] == 2
events[0].get("json_values")["fieldC_Integer"] == 3
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment