metricbeat-ilm-quickstart.txt

1) Prerequisite : Properly configured Hot / Warm Elasticsearch cluster with correct node attributes.

2) Configure Metricbeat to point directly to Elasticsearch and run setup

NOTE: When metricbeat setup is run, it will overwrite the ILM Policy and recreate the bootstrap index with that policy,
this is often confusing. Running setup also creates all the dashboards, index templates etc. so it is very useful. 
As part of the template it will name the ILM policy and rollover_alias in this example metricbeat-7.4.0

./metricbeat setup

Take a look at the template

GET _template/metricbeat-7.4.0

{
  "metricbeat-7.4.0" : {
    "order" : 1,
    "index_patterns" : [
      "metricbeat-7.4.0-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "metricbeat-7.4.0",
          "rollover_alias" : "metricbeat-7.4.0"
        },
..........................
        
Take a look at the default policy

GET _ilm/policy/metricbeat-7.4.0

{
  "metricbeat-7.4.0" : {
    "version" : 27,
    "modified_date" : "2019-11-10T17:18:45.326Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  }
}

Take a look at the bootstrap index and note the default ILM policy is applied.

GET metricbeat-7.4.0-2019.11.10-000001/_ilm/explain

{
  "indices" : {
    "metricbeat-7.4.0-2019.11.10-000001" : {
      "index" : "metricbeat-7.4.0-2019.11.10-000001",
      "managed" : true,
      "policy" : "metricbeat-7.4.0",
      "lifecycle_date_millis" : 1573404721340,
      "age" : "1.68m",
      "phase" : "hot",
      "phase_time_millis" : 1573404721567,
      "action" : "unfollow",
      "action_time_millis" : 1573404721567,
      "step" : "wait-for-follow-shard-tasks",
      "step_time_millis" : 1573404721645,
      "phase_execution" : {
        "policy" : "metricbeat-7.4.0",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 23,
        "modified_date_in_millis" : 1573404720555
      }
    }
  }
}

3) Create and apply the ILM Policy you want. You can do this via the Kibana GUI if you like. 
You can also see what the request would look like. 

NOTE: This is for testing purposes only, and also note since ILM processes are background and opportunistic it will not
fire exactly at 1001 documents it is not what it is designed for, it may take what seems like much longer, but at scale
this % overrun etc will be very small. This example is so that you can observe the ILM work in shorter time frames.

Example 

PUT _ilm/policy/metricbeat-7.4.0
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1h",
            "max_size": "1gb",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "data": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "data": "warm"
            }
          },
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "4h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

4) Delete the bootstrap index 

VERY IMPORTANT NOTE : When an index is created the ILM policy is applied and is immutable for that phase that the index 
is in. (causes confusion) That is why applying a new policy does not take effect on the bootstrap because it is already 
in the “hot” phase. This is why we delete the bootstrap index and recreate it. 

https://www.elastic.co/guide/en/elasticsearch/reference/7.4/update-lifecycle-policy.html

DELETE metricbeat-7.4.0-2019.11.10-000001

5) Recreate the bootstrap index… 

PUT metricbeat-7.4.0-2019.11.10-000001
{
  "aliases": {
    "metricbeat-7.4.0": {
      "is_write_index": true
    }
  }
} 

Take a look at the new ILM explain, 
NOTE: It is now the correct policy

GET metricbeat-7.4.0-2019.11.10-000001/_ilm/explain
{
  "indices" : {
    "metricbeat-7.4.0-2019.11.10-000001" : {
      "index" : "metricbeat-7.4.0-2019.11.10-000001",
      "managed" : true,
      "policy" : "metricbeat-7.4.0",
      "lifecycle_date_millis" : 1573404930326,
      "age" : "3.85s",
      "phase" : "hot",
      "phase_time_millis" : 1573404930534,
      "action" : "unfollow",
      "action_time_millis" : 1573404930534,
      "step" : "wait-for-follow-shard-tasks",
      "step_time_millis" : 1573404930619,
      "phase_execution" : {
        "policy" : "metricbeat-7.4.0",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "5gb",
              "max_age" : "1h",
              "max_docs" : 1000
            }
          }
        },
        "version" : 24,
        "modified_date_in_millis" : 1573404919616
      }
    }
  }
}


Now you can start metricbeat and it will follow the ILM policy you have defined. 

./metricbeat -e 

If you want to run metricbeat through logstash configure metricbeat to output to logstash

metricbeat.yml
...  
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]


A Basic logstash configuration should look like this. 
NOTE: You do NOT need to add all the ilm_* configurations as they are automatically taken care of. 

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://adfasdfasdfsadfsadfasdf.us-west1.gcp.cloud.es.io:9243"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    user => "elastic"
    password => "sfgdsfgdsfgdsfgsdfgdsfg"
  }
}