secondary zipkin template for elasticsearch 7

elasticsearch-secondary.md

Zipkin by default does special indexing to deal with limitations around names with dots and to reduce storage costs. Normal elasticsearch queries for tags won't work because of this. However, you can always add a second indexing template with the tags you feel you need to work with other tools like kibana. This shows you how to do that.

Before asking any questions please run exactly as mentioned here. These instructions are Elasticsearch v7 specific. If you do something different like use Elasticsearch 6, or do things out of order, it might not work. In other words, use our docker-compose stuff so to eliminate variance and understand how things work before changing variables.

$ git clone https://github.com/openzipkin/zipkin
$ cd zipkin/docker/examples/

Edit docker-compose-elasticsearch.yml and uncomment out ports. This allows you to access ES directly

-    # ports:
-    #   - 9200:9200
+    ports:
+      - 9200:9200

Startup zipkin with elasticsearch using docker-compose up

$ docker-compose -f docker-compose-elasticsearch.yml -f docker-compose-example.yml up

Hit a url so that it implicitly loads the zipkin index templates

$ curl -s localhost:9411/api/v2/traces

Add a second index template, taking care to do it exactly like ours (do not include_type_name=true). This below adds standard indexing for the tag named http.status_code.

$ curl -X PUT -v -s localhost:9200/_template/zipkin:span_http.status_code_template -H'Content-Type: application/json' -d'{
  "index_patterns": [
    "zipkin:span-*"
  ],
  "order": 0,
  "mappings": {
    "span": {
      "properties": {
        "tags.http.status_code": {
          "norms": false,
          "type": "keyword"
        }
      }
    }
  }
}'

Hit the brave-example app in a way that results in a non-success code. Most instrumentation do not record the http.status_code on success, so that's why you need to hit something that produces an error. The following endpoint doesn't exist on the example frontend, so it will make a 404.

$ curl -s localhost:8081/bobby

Now you can validate Elasticsearch normal indexing applies to that field.

$ curl -s 'localhost:9200/zipkin:span-*/_search?q=tags.http.status_code:404'

An option that I was able to make work if you need to map tags in your index is below. However, there are a couple caveats:

1. If the tag has any dots in it (like in the template in the gist), spans containing tags that are made up of parts of the tag leading up to but not including the last part will fail to save.
   Example: If instead of tag_name below, you had foo.bar, you could not send a tag foo, or if in the template below you had alpha.beta.gamma, you could not send tags alpha or alpha.beta.

2. Similarly, you would not be able to send a span with a tag which is preceded by the tag name in the template followed by a dot and additional text.
   Example: Given tag_name below, you could not send tag_name.foo or tag_name.foo.bar, etc.

Additionally, you should be able to set whatever typical properties on the mapping that you would otherwise, e.g. "norms" or a different "type" like long or boolean, but I have not tried any of these.

{
  "index_patterns": [
    "zipkin-span-*"
  ],
  "order": 1,
  "mappings": {
    "properties": {
      "tags": {
        "dynamic": false,
        "enabled": true,
        "properties": {
          "tag_name": {
            "index": true,
            "type": "keyword"
          }
        }
      }
    }
  }
}

from @xeraa this approach can change from 7.8+ and will need to by 8:

1. There are the legacy / v1 templates that can be merged, but I would be careful (multiple templates with the same order will result in a non-deterministic merging order, merging valid templates might lead to an invalid mapping). Right now they are deprecated and we'll see when we'll remove them — you have at least until 8.0.
2. The new component based approach still allows you to combine multiple pieces, but you don't merge them (to avoid the two problems mentioned above). This feature was only added in 7.8, so I'm not sure what will be the sanest approach for you — support it in 7.x already or switch over for any (upcoming) 8.x cluster.

@eraffel-MDSol indeed this won't work for all tags, which is why we don't use natural indexing. Especially opentracing and things that implement it send "error" and also "error.message" and similar sometimes. The original example here was from netflix who have <=20 tags they ever use so control explicitly. In some environments, they need only 4 so it is possible for some environments to choose what you do. The fact that not everyone can is why this remains a gist :D