Skip to content

Instantly share code, notes, and snippets.

@FrankHassanabad
Last active June 7, 2021 17:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save FrankHassanabad/8aeaeb0a7f5f6a4c796d70936cc46232 to your computer and use it in GitHub Desktop.
Save FrankHassanabad/8aeaeb0a7f5f6a4c796d70936cc46232 to your computer and use it in GitHub Desktop.
Elasticsearch runtime fields are very loose and liberal with dots for field names.

Steps are, add a simple static mapping:

# Add our mapping of just a keyword called "test"
DELETE frank-test-delme-5
PUT frank-test-delme-5
{
  "mappings": {
    "dynamic": "false",
    "properties": {
      "test": {
        "type": "keyword"
      }  
    }
  }
}

Add a simple document to compare with

# Works and you are ok
PUT frank-test-delme-5/_doc/1
{
  "test": "hello"
}

Notice you cannot just start a field with a "." in a static mapping

# We get an error here of "field name cannot contain only dots"
PUT frank-test-delme-5/_doc/1
{
  ".": "hello"
}
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "field name cannot contain only dots"
    }
  },
  "status" : 400
}

You also cannot turn a keyword into an object with a static mapping:

# We get an error here trying to force an object ontop of test. Existing mapping for [test] must be of type object but found [keyword]."
PUT frank-test-delme-5/_doc/1
{
  "test.yolo": "hello"
}
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "Could not dynamically add mapping for field [test.yolo]. Existing mapping for [test] must be of type object but found [keyword]."
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Could not dynamically add mapping for field [test.yolo]. Existing mapping for [test] must be of type object but found [keyword]."
  },
  "status" : 400
}

But if you use runtime fields you are allowed to do all of these things, even with the same document.

# But we get no errors here with runtime fields are very liberal about what they allow for field names which includes:
#  * only dots
#  * multiple dots
#  * dots around "test"
# This does gets ambigious as we parse "fields" instead of "_source" when since "fields" is a flattened representation of
# objects except for nested fields.
GET frank-test-delme-5/_search
{
  "runtime_mappings": {
    ".": {
      "type": "keyword",
      "script": {
        "source": """
          emit("field names can now contain dots")
        """
      }
    },
    "..": {
      "type": "keyword",
      "script": {
        "source": """
          emit("field names can now contain two dots")
        """
      }
    },
    "test.": {
      "type": "keyword",
      "script": {
        "source": """
          emit("field names can now conflict with objects")
        """
      }
    },
    ".test.": {
      "type": "keyword",
      "script": {
        "source": """
          emit("field names can now conflict with objects")
        """
      }
    },
    "test.yolo": {
      "type": "keyword",
      "script": {
        "source": """
          emit("Ambigious naming requires we examine mappings or do best effort")
        """
      }
    }
  },
  "fields": [
    "*"
  ]
}

No errors, we are allowed to use all of those cases above.

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "frank-test-delme-5",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "test" : "hello"
        },
        "fields" : {
          ".." : [
            "field names can now contain two dots"
          ],
          "test" : [
            "hello"
          ],
          "test.yolo" : [
            "Ambigious naming requires we examine mappings or do best effort"
          ],
          "." : [
            "field names can now contain dots"
          ],
          "test." : [
            "field names can now conflict with objects"
          ],
          ".test." : [
            "field names can now conflict with objects"
          ]
        }
      }
    ]
  }
}
@FrankHassanabad
Copy link
Author

FrankHassanabad commented Jun 7, 2021

It looks like runtime fields allows you to do things with field names that normal mappings do not allow you to do such as:

  • Having field names with just dots, ".", ".."
  • Having field names that start or end with just dots ".test", "test."
  • Having field names which conflict with primitive/objects such as shown above with "test.yolo" when "test" is a keyword and not an object.

I think you do want to convert/change some data types, but I am wondering how many rules do you want to break about field names and do you want to break those rules in the same response? Right now we cannot determine in the response fields which ones are runtime fields vs. non-runtime fields.

If we have an existing index mapping we can kind of do compares of fields and any not in the mapping could be considered a strong probability of being a runtime field but since runtime fields "shadow" existing data types, that can be tricky as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment