Jaeger.md

jaeger-cr.yaml

# setup an elasticsearch with `make es`
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  query:
    image: "pavolloffay/jaeger-query:es-scripts-tls2"
  collector:
    image: "pavolloffay/jaeger-collector:es-scripts-tls2"
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: https://quickstart-es-http.default.svc:9200
        use-aliases: true
        tls.ca: /es/secrets/ca.crt
    #        tls.skip-host-verify: true
    #        username: elastic
    #        password: ql7hbmqfzzkrtn6klcdsh8n5
    secretName: jaeger-secret
    esIndexCleaner:
      image: "pavolloffay/jaeger-es-index-cleaner:es-scripts-tls2"
      schedule: "*/1 * * * *"
    esRollover:
      image: "pavolloffay/jaeger-es-rollover:es-scripts-tls2"
      schedule: "*/2 * * * *"
  volumeMounts:
    - name: secrets
      mountPath: /es/secrets/
      readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: quickstart-es-http-certs-public

Jaeger.md

This readme describes how to configure Jaeger storages and various Jaeger services.

Cassandra

This section describes how to configure Jaeger with Cassandra storage

Run Cassandra with ccm

https://github.com/pcmanus/ccm

# create one node cassandra
ccm create test -v 3.11.0 -n 1 -s
# create two datacenters with two nodes each
ccm create test -v 3.11.0 2:2 -s

# debug, show datacenter name
ccm node1 ring
ccm mode1 nodetool status

# clean
ccm remove test

# initialize Jaeger schema
MODE=test sh plugin/storage/cassandra/schema/create.sh | ccm node1 cqlsh 
MODE=prod DATACENTER=datacenter1 sh plugin/storage/cassandra/schema/create.sh | ccm node1 cqlsh 
 

Run Cassandra with docker

docker run --name=cassandra --rm -it -p 7000:7000 -p 9042:9042 cassandra:3.9
MODE=test ./plugin/storage/cassandra/schema/create.sh | docker exec -i cassandra cqlsh

Then add --cassandra.keyspace=jaeger_v1_test to Jaeger services.

Elasticsearch

This section describes how to configure Jaeger with Elasticsearch storage

Run from docker image

docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "discovery.type=single-node" -e "xpack.security.enabled=false" --name=elasticsearch  docker.elastic.co/elasticsearch/elasticsearch:5.6.10
docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "discovery.type=single-node" --name=elasticsearch  docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.4
docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "discovery.type=single-node" -e "path.data=/data" --name=elasticsearch  -v "/home/ploffay/tmp/es-data:/data" docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.4

Add environmental variable SPAN_STORAGE_TYPE=elasticsearch when running Jaeger services. This storage doesn't require any other initialization.

Debug commands:

• list indices: http://localhost:9200/_cat/indices?v
• list spans: http://localhost:9200/jaeger-span-2017-09-18/_search?pretty&size=9999
• list shards: http://localhost:9200/_cluster/health/?level=shards
• Use scroll API: curl -ivX POST -H "Content-Type: application/json" localhost:9200/jaeger-service-2018-07-26/_search?scroll=1m -d '{"size": 300000}'
• curl POST -H "Content-Type: application/json" localhost:9200/_search/scroll -d '{"scroll": "1m", "scroll_id": ""}

Remove data

curl -X DELETE localhost:9200/jaeger-span-`date +%Y-%m-%d` # only spans
curl -X DELETE localhost:9200/jaeger-* # all Jaeger data

Kibana

docker run --rm -it --link=elasticsearch --name=kibana -p 5601:5601 docker.elastic.co/kibana/kibana:5.6.10

Search examples:

• tag.span@kind:server AND process.tag.ip:10.33.144.223

http://localhost:5601

Run jaeger services

Add env variable SPAN_STORAGE_TYPE=elasticsearch|cassandra to use specific storage type.

SPAN_STORAGE_TYPE=elasticsearch  go run -tags ui cmd/standalone/main.go --collector.queue-size=300000
go run cmd/collector/main.go --cassandra.keyspace=jaeger_v1_test
go run cmd/query/main.go  --query.static-files=jaeger-ui-build/build/ --cassandra.keyspace=jaeger_v1_test

With docker:

docker run -it --rm --name=jaeger-collector --link=cassandra -p 14268:14268 -e CASSANDRA_SERVERS=cassandra -e CASSANDRA_KEYSPACE=jaeger_v1_test -e COLLECTOR_QUEUE_SIZE=300000  jaegertracing/jaeger-collector:latest
docker run -it --rm --name=jaeger-agent --link=jaeger-collector -e COLLECTOR_HOST_PORT=jaeger-collector:14267 -e PROCESSOR.JAEGER_COMPACT_SERVER_QUEUE_SIZE=300000 jaegertracing/jaeger-agent:latest

For performance tunning change this:

• Agent: --processor.jaeger-compact.server-queue-size=N
• Collector: --collector.queue-size=N

Query traces from Jaeger

curl localhost:16686/api/traces?limit=300000\&service=perf

Build locally all-in-one image

make build-all-in-one-linux
docker build  -f cmd/standalone/Dockerfile -t  jaeger .
docker run --network=host -e "COLLECTOR_ZIPKIN_HTTP_PORT=9411" jaeger

Jaeger Java client initialization

  public com.uber.jaeger.Tracer createTracer(String serviceName) {
    return new com.uber.jaeger.Tracer.Builder(serviceName,
        new RemoteReporter(new HttpSender("http://localhost:14268/api/traces", 65000), 1, 100,
            new Metrics(new StatsFactoryImpl(new NullStatsReporter()))), new ConstSampler(true))
        .build();
  }

  public com.uber.jaeger.Configuration createConfiguration(String serviceName) {
    Configuration conf = new Configuration(serviceName, new SamplerConfiguration(ConstSampler.TYPE, 1),
        new ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")));

    return conf;
  }

curl -ivX GET --cacert cacertdecoded  --key adminkeydecoded --cert admincertdecoded https://elasticsearch:9200/jaeger-span-2019-01-29/_search\?pretty

SPAN_STORAGE_TYPE=elasticsearch go run -tags ui cmd/all-in-one/main.go --es.tls.cert /home/ploffay/admincertdecoded --es.tls.key /home/ploffay/adminkeydecoded --es.server-urls=https://elasticsearch:9200 --es.tls=true --es.tls.ca=/home/ploffay/cacertdecoded --es-archive.tls=true --es-archive.tls.cert /home/ploffay/admincertdecoded --es-archive.tls.key /home/ploffay/adminkeydecoded --es-archive.server-urls=https://elasticsearch:9200 --es-archive.tls.ca=/home/ploffay/cacertdecoded