Grafana Tips.md

• Expressions on different stats - doesn't work due to extra name label which we need to discard:

sum without (name) (kv_ep_db_data_size_bytes{bucket="$bucket"}) / sum without (name) (kv_ep_db_file_size_bytes{bucket="$bucket"})

Heatmaps

Note: Set chart type to 'Heatmap' (top right), Series format to 'Heatmap'

• Heatmap of KV opcode latencies:

sum by(le) (rate(kv_cmd_duration_seconds_bucket{opcode="ADD", bucket="$bucket"}[5m]))

• Heatmap of KV SyncWrite latencies:

sum by(le) (rate(kv_sync_write_commit_duration_seconds_bucket{level="majority", bucket="$bucket"}[10m]))

Percentiles

• p99 of a specific opcode:

histogram_quantile(0.99, rate(kv_cmd_duration_seconds_bucket{opcode="SET", bucket="$bucket"}[10m])) 

Thread Utilization

• Per thread utilization (user+sys summmed):

sum by (thread_name) (rate(kv_thread_cpu_usage_seconds[6m]))

• Thread pool utilisation (absolute):

sum by (thread_pool) (rate(kv_thread_cpu_usage_seconds[6m]))

• How utilised is each KV thread pool (relative to thread pool size)?

sum by (thread_pool) (rate(kv_thread_cpu_usage_seconds[2m])) / count by(thread_pool) (kv_thread_cpu_usage_seconds{domain="user"})