Use Logstash ruby filter to rename field name based on value type to avoid Elasticsearch mapping conflict

0_README.md

This is Logstash ruby filter example to rename field names within a specified hash, based on each value type. The original question was asked at this Elastic Discuss.

How to test the filter

With -t, the filter can be tested:

logstash -e "filter { ruby { path => '/{path_of_the_ruby_script}/rename_by_data_type.rb' script_params => { 'field' => 'the_name_of_target_field' } } }" -t

Output:

[2022-06-15T18:14:42,729][INFO ][logstash.filters.ruby.script] Test run complete {:script_path=>"/{path_of_the_ruby_script}/rename_by_data_type.rb", :results=>{:passed=>2, :failed=>0, :errored=>0}}
Configuration OK

Usage

Specify the script file path, and a field name of the target hash field.

filter {
  ruby {
    path => '/{path_of_the_ruby_script}/rename_by_data_type.rb'
    script_params => { 'field' => 'the_name_of_target_field' }
  }
}

Example input and output

The data is shown in JSON format for illustration purpose.

Input:

[
{"values": {"fieldA": "a", "fieldB": "b", "fieldC": "c"}},
{"values": {"fieldA": 1, "fieldB": 2, "fieldC": 3}}
]

Output:

[
{"values": {"fieldA_String": "a", "fieldB_String": "b", "fieldC_String": "c"}},
{"values": {"fieldA_Integer": 1, "fieldB_Integer": 2, "fieldC_Integer": 3}}
]

rename_by_data_type.rb

def register(params)
  @field = params["field"]
end

def filter(event)
  values = event.get(@field)
  renamed = {}
  values.each do |key, value|
    renamed[key + "_" + value.class.to_s()] = value
  end
  event.set(@field, renamed)
  return [event]
end

test "rename fields by data type - String" do
  parameters do
    { "field" => "json_values" }
  end

  in_event {
    {"json_values" => {
      "fieldA" => "a",
      "fieldB" => "b",
      "fieldC" => "c"
    }}
  }

  expect("updated") do |events|
    events[0].get("json_values")["fieldA_String"] == "a"
    events[0].get("json_values")["fieldB_String"] == "b"
    events[0].get("json_values")["fieldC_String"] == "c"
  end
end

test "rename fields by data type - Integer" do
  parameters do
    { "field" => "json_values" }
  end

  in_event {
    {"json_values" => {
      "fieldA" => 1,
      "fieldB" => 2,
      "fieldC" => 3
    }}
  }

  expect("updated") do |events|
    events[0].get("json_values")["fieldA_Integer"] == 1
    events[0].get("json_values")["fieldB_Integer"] == 2
    events[0].get("json_values")["fieldC_Integer"] == 3
  end
end