| #!/bin/bash | |
| name=$RANDOM | |
| url='http://localhost:9093/api/v1/alerts' | |
| echo "firing up alert $name" | |
| # change url o | |
| curl -XPOST $url -d "[{ | |
| \"status\": \"firing\", | |
| \"labels\": { | |
| \"alertname\": \"$name\", | |
| \"service\": \"my-service\", | |
| \"severity\":\"warning\", | |
| \"instance\": \"$name.example.net\" | |
| }, | |
| \"annotations\": { | |
| \"summary\": \"High latency is high!\" | |
| }, | |
| \"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
| }]" | |
| echo "" | |
| echo "press enter to resolve alert" | |
| read | |
| echo "sending resolve" | |
| curl -XPOST $url -d "[{ | |
| \"status\": \"resolved\", | |
| \"labels\": { | |
| \"alertname\": \"$name\", | |
| \"service\": \"my-service\", | |
| \"severity\":\"warning\", | |
| \"instance\": \"$name.example.net\" | |
| }, | |
| \"annotations\": { | |
| \"summary\": \"High latency is high!\" | |
| }, | |
| \"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
| }]" | |
| echo "" |
This is useful! Thanks for posting it.
Very useful. Thanks !
Thanks!
Original doc: https://prometheus.io/docs/alerting/clients/
Thanks!
To save folks the frustration - this script actually doesn't send the resolve alert according to the v1 api spec. The prometheus contributors have point this out prometheus/alertmanager#1306. The alert only resets after a few minutes because alertmanager expects to get a list of currently firing alerts every 30s-3mins. In order to resolve the alert immediately after the curl you have to include the endsAt timestamp:
url='http://localhost:9093/api/v1/alerts'
echo "Firing up alert"
curl -XPOST $url -d '[{"status": "firing","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00"}]'
echo ""
echo "press enter to resolve alert"
read
echo "sending resolve"
curl -XPOST $url -d '[{"status": "resolved","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00","endsAt": "2020-07-23T01:05:38+00:00"}]'
echo ""
Please note you'll have to update the startsAt and endsAt timestamps. I used this site to get RFC3339 compliant timestamps:
https://www.unixtimestamp.com/
So timestamp generation code as per suggestion above
startsAt=`date --iso-8601=seconds`
... # after the `read` command
endsAt=`date --iso-8601=seconds`
You just need to insert these shell variables into the curl command. This works for me. Using the --rfc-3339 instead of --iso-8601 I found T separator missing which may or may not be an issue.
I've revisited this snippet multiple times each time having to make the alterations. Here's a gist incorporating all the changes https://gist.github.com/carinadigital/fd2960fdccd77dbdabc849656c43a070
single quote is your friend..