ceph-end2end-tracing.md

setup

note that the following PR introduces end2end tracing (RGW->OSD) for "object PUT"

• start the cluster with tracing enabled:

OSD=1 MON=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -d -n -o jaeger_tracing_enable=true -o jaeger_agent_port=6831

Note that, by default, both OSD and RGW try to send th traces to a different port than the one the agent listen on by default

• start the "all in one" container of jaeger, this will run: the agent, the collector the backend and the UI:

$ podman run \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.42

since no storage is defined here, the traces will be only in memory and will be lost once the container stops. for persistent storage the simplest option is to use the file based badger db. in this case the commands to start the "all in one" container would be:

$ mkdir -p /tmp/badger
$ podman run \                                                                                                                             
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -e SPAN_STORAGE_TYPE=badger \
  -e BADGER_EPHEMERAL=false \
  -e BADGER_DIRECTORY_VALUE=/badger/data \
  -e BADGER_DIRECTORY_KEY=/badger/key \
  -v /tmp/badger:/badger \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.42

test

• create a bucket and upload a file:

$ aws --endpoint-url http://localhost:8000 s3 mb s3://mybucket
$ aws --endpoint-url http://localhost:8000 s3 cp myfile s3://mybucket

• check the web UI to see the traces: http://localhost:16686/search
• there is also a REST API to get traces (note that this API in considered internal and may not be stable btween releases)
• following fetches max 20 end2end traces from the last hour:

curl "http://localhost:16686/api/traces?service=rgw&limit=20&lookback=1h"

• the folowing fetches a specific trace:

curl "http://localhost:16686/api/traces/0f1f3fb4d2192a86a959771064505538"