Skip to content

Instantly share code, notes, and snippets.

@FrankHassanabad
Last active June 2, 2021 19:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save FrankHassanabad/f614ec9762d59cd1129b3269f5bae41c to your computer and use it in GitHub Desktop.
Save FrankHassanabad/f614ec9762d59cd1129b3269f5bae41c to your computer and use it in GitHub Desktop.
Mixing doc_valuefields with fields can lead to unexpected results

I want to return @timestamp as a different format such as "YYYY" but am mixing together doc_valuefields with fields. Fields takes priority over doc_valuefields it looks like.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html

We might have a slight bug here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.ts#L109-L125

Where we mix these two. I test this by trying override the date times below but it only works when we remove doc_valuefields and not mix it with fields since fields uses a *.

# Using just doc_valuefields works, you get this returned
# "fields" : {
#   "@timestamp" : [
#     "2021" <--- Matches my format: "YYYY"
#   ]
# }
GET auditbeat*/_search
{
  "size": 1,
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "YYYY"
    }
  ]
}

# This doesn't work and you get the default type since fields is overriding things. You get this returned
# "fields" : {
#   "@timestamp" : [
#     "2021-05-05T16:03:17.037Z" <--- Does not match because fields is overriding
#   ]
# }
GET auditbeat*/_search
{
  "size": 1,
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "YYYY"
    }
  ],
  "fields": [
    {
      "field": "*",
      "include_unmapped": true
    }
  ]
}

# This works because we remove the docvalue_fields and just use fields.
# Ordering DOES matter, and you cannot put @timestamp first in the fields array. You now get this returned:
# "fields" : {
#   "@timestamp" : [
#     "2021" <--- Works as long as @timestamp specifically is last.
#   ]
# }
GET auditbeat*/_search
{
  "size": 1,
  "fields": [
    {
      "field": "*",
      "include_unmapped": true
    },
    {
      "field": "@timestamp",
      "format": "YYYY"
    }
  ]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment