Elasticsearch Logs Handbook

elasticsearch_logs_handbook.md

Fetch documents with timetamp lower than the defined date (curl format)

curl --request GET \
  --url http://localhost:9200/_search \
  --header 'Content-Type: application/json' \
  --header 'cache-control: no-cache' \
  --data '{\n  "query": {\n    "bool": {\n      "filter": {\n        "range": {\n          "@timestamp": {\n            "lt": "2019-01-01"\n          }\n        }\n      }\n    }\n  }\n}'

Fetch all the indices sorted by size descending (simple http format)

GET /_cat/indices?v&s=store.size:desc

Delete all the indices that contains '2018' in the index name (simple http format)

DELETE /*2018*

Unlock read-only /allow delete (kibana dev tools format)

elaticsearch set this flag to true in all indices when the disc space is low (https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.html)

PUT _settings
{
 "index.blocks.read_only_allow_delete": null
}

Elasticsearch good starting Point for configs

• Number of replicas = ideally equals to the number of replica nodes
  • number of cluster nodes = primary node + replica nodes
• Number of shards = 1,5 <-> 3 factor of Nodes number
  • example: ES cluster with 3 nodes, each indice should have (at most) 9 (3 * 3) shards
  • refs:
    • https://youtu.be/aG6WPu08qBs
    • https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index
• Note that if using a single node strategy, you dont need any replica unless you want the data duplicated (for backups purpose)
• For indices that are rotated daily, meaning 1 indice per day, 1 single shard is enough till +- 2B documents

How save ram:

• close indicies after X (eg. 15) days, obtainable with Curator How save disk space:
• Delete indices after X + Y (eg. 30) days, obtainable with Curator

Note the highest priority matching template will be the chosed one.

POST _index_template/default-single-node
{
  "priority": 1000,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "0",
        "refresh_interval": "10s"
      }
    }
  },
  "index_patterns": [
    "*"
  ],
  "composed_of": []
}

legacy:

POST _template/default-cluster
{
   "index_patterns" : ["*"],
   "order" : 0,
   "settings" : {
       "refresh_interval" : "10s",
       "number_of_shards" : "1",
       "number_of_replicas" : "0"
   }
}

• Index Lifecycle Management as an built-in alternative of curator (external tool) to manage the old indexes. Requirements ES >= 6.8

Another great tutorial: https://bonsai.io/blog/ideal-elasticsearch-cluster.html

Index Naming Conventions

Example from here:

logs-system-{date}
logs-iis-{date}
logs-prometheus-{date}
logs-app-{applicationName}-{date}

simple_elasticsearch_pre_config.md

Basic configurations for a smooth setup

Tested with ELK version 7.9

ILM

PUT _ilm/policy/basic-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "60d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Index template

• config for a single node

PUT _index_template/default
{
  "priority": 1000,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "0",
        "refresh_interval": "10s",
        "lifecycle.name": "basic-logs-policy",
        "lifecycle.rollover_alias": "logs"
      }
    },
    "mappings": {
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ]
    }
  },
  "index_patterns": [
    "*"
  ],
  "composed_of": []
}