Skip to content

Instantly share code, notes, and snippets.

@debugloop
Created September 30, 2019 14:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save debugloop/d77d35cd5f53f2d462c546586715b7e6 to your computer and use it in GitHub Desktop.
Save debugloop/d77d35cd5f53f2d462c546586715b7e6 to your computer and use it in GitHub Desktop.
promql query to compare to global avg
# take the sum of the rate, leave stuff in irrelevant labels alone
# maybe use delta instead of rate to make it sign-aware
sum without (stuff) (rate($metric[1d]))
>
# ignore the instance on this side of the compare, would be even better without the instance of interest
ignoring (instance, stuff) group_left()
(
avg without(instance, stuff)(rate($metric[1w]))
+
# just give that some more breathing room
2 * stddev without(instance, stuff)(rate($metric[1w]))
)
# maybe fiddle with the 1d/1w part, larger difference in time scope makes this more aggressive, in theory
# Once more, but readable:
sum without (stuff) (rate($metric[1h]))
> ignoring (instance, stuff) group_left()
(
avg without(instance, stuff)(rate($metric[1w]))
+
2 * stddev without(instance, stuff)(rate($metric[1w]))
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment