Part 3: Index templates

es-mapping-article-examples.org

Part 3: Index templates

Create an index template

First, we create a template matching any index with the name “visitor_logs*”, and set those indices up with a mapping, and with an index alias.

Create a new index template:

curl -XPUT http://localhost:9200/_template/visitor_logs_template_1 -H "content-type: application/json" -d @- <<EOF
{
    "index_patterns": ["visitor_logs*"],
    "mappings" : {
        "_doc" : {
            "dynamic": "strict",
            "properties" : {
                "user-id":    { "type": "keyword" },
                "ip":         { "type": "ip" },
                "session-id": { "type": "keyword" },
                "ts":         { "type": "date" },
                "url":        { "type": "text" },
                "method":     { "type": "keyword" }
            }
        }
    },
    "aliases": {
        "visitor_logs": {}
    }
}
EOF

{"acknowledged":true}

Retrieve that template:

curl -XGET http://localhost:9200/_template/visitor_logs_template_1?pretty

{
  "visitor_logs_template_1" : {
    "order" : 0,
    "index_patterns" : [
      "visitor_logs*"
    ],
    "settings" : { },
    "mappings" : {
      "_doc" : {
        "dynamic" : "strict",
        "properties" : {
          "user-id" : {
            "type" : "keyword"
          },
          "ip" : {
            "type" : "ip"
          },
          "session-id" : {
            "type" : "keyword"
          },
          "ts" : {
            "type" : "date"
          },
          "url" : {
            "type" : "text"
          },
          "method" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : {
      "visitor_logs" : { }
    }
  }
}

Show that nothing has (yet) been created

We still don’t have any indices, nor any aliases defined in the cluster:

curl http://localhost:9200/_cat/indices/visitor_logs*?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

curl -XGET http://localhost:9200/_aliases?pretty

{
}

(monitoring index elided for clarity)

Index some data

Now, simply indexing data into a target index will create the index if it doesn’t exist, before adding the data.

Index two documents, simulating how they would be added over time (note the year in the target index):

curl -XPOST http://localhost:9200/visitor_logs_2017/_doc -H "content-type: application/json" -d @- <<EOF
{
  "user-id": "30c1b62a",
  "ip": "10.76.54.93",
  "session-id": "08298f4a",
  "ts": "2017-12-31T08:52:19Z",
  "url": "https://www.example.com/api/reports/228422",
  "method": "PUT"
}
EOF

echo

curl -XPOST http://localhost:9200/visitor_logs_2018/_doc -H "content-type: application/json" -d @- <<EOF
{
  "user-id": "30c1b62a",
  "ip": "10.76.54.93",
  "session-id": "d8e81b56",
  "ts": "2018-01-01T13:55:01Z",
  "url": "https://www.example.com/api/reports/228422",
  "method": "GET"
}
EOF

{"_index":"visitor_logs_2017","_type":"_doc","_id":"PAruAGIBJPVC1sk2nKAn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
{"_index":"visitor_logs_2018","_type":"_doc","_id":"PQruAGIBJPVC1sk2naA0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

Query the indices using the index alias:

curl http://localhost:9200/visitor_logs/_search?q=30c1b62a | jq .hits.hits

[
  {
    "_index": "visitor_logs_2018",
    "_type": "_doc",
    "_id": "PQruAGIBJPVC1sk2naA0",
    "_score": 0.2876821,
    "_source": {
      "user-id": "30c1b62a",
      "ip": "10.76.54.93",
      "session-id": "d8e81b56",
      "ts": "2018-01-01T13:55:01Z",
      "url": "https://www.example.com/api/reports/228422",
      "method": "GET"
    }
  },
  {
    "_index": "visitor_logs_2017",
    "_type": "_doc",
    "_id": "PAruAGIBJPVC1sk2nKAn",
    "_score": 0.2876821,
    "_source": {
      "user-id": "30c1b62a",
      "ip": "10.76.54.93",
      "session-id": "08298f4a",
      "ts": "2017-12-31T08:52:19Z",
      "url": "https://www.example.com/api/reports/228422",
      "method": "PUT"
    }
  }
]

As expected, we see two hits, one from each index created when the documents themselves were indexed.

Show the index setup

List the attributes of the index. Note: we didn’t ever explicitly create this index, it was created when we indexed a document into a desired target index matching the template wildcard.

curl -XGET http://localhost:9200/visitor_logs_2018?pretty

{
  "visitor_logs_2018" : {
    "aliases" : {
      "visitor_logs" : { }
    },
    "mappings" : {
      "_doc" : {
        "dynamic" : "strict",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "method" : {
            "type" : "keyword"
          },
          "session-id" : {
            "type" : "keyword"
          },
          "ts" : {
            "type" : "date"
          },
          "url" : {
            "type" : "text"
          },
          "user-id" : {
            "type" : "keyword"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1520434060409",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "iMPwQ2YaQMChr-YbZaAPbw",
        "version" : {
          "created" : "6020199"
        },
        "provided_name" : "visitor_logs_2018"
      }
    }
  }
}

Show the aliases:

curl -XGET http://localhost:9200/_aliases?pretty

{
  "visitor_logs_2018" : {
    "aliases" : {
      "visitor_logs" : { }
    }
  },
  "visitor_logs_2017" : {
    "aliases" : {
      "visitor_logs" : { }
    }
  }
}

There isn’t anything surprising either in the resulting index definition, nor in the alias setup. Index templates are unobtrusive, yet incredibly useful.