For our services we make sure of course to record the right metrics in Prometheus. Which means creating recording rules for
- 7 services,
- two environments,
- from both a latency - and throughput perspective,
- on service internals like: outward http, circuitbreakers, persistence (mongo, hikari connection pooling), logging, exceptions, and more
- different percentiles and
- intervals and windows
All possible combinations make for a total of 4600 lines of yaml
per environment.