#!/bin/bash | |
name=$RANDOM | |
url='http://localhost:9093/api/v1/alerts' | |
echo "firing up alert $name" | |
# change url o | |
curl -XPOST $url -d "[{ | |
\"status\": \"firing\", | |
\"labels\": { | |
\"alertname\": \"$name\", | |
\"service\": \"my-service\", | |
\"severity\":\"warning\", | |
\"instance\": \"$name.example.net\" | |
}, | |
\"annotations\": { | |
\"summary\": \"High latency is high!\" | |
}, | |
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
}]" | |
echo "" | |
echo "press enter to resolve alert" | |
read | |
echo "sending resolve" | |
curl -XPOST $url -d "[{ | |
\"status\": \"resolved\", | |
\"labels\": { | |
\"alertname\": \"$name\", | |
\"service\": \"my-service\", | |
\"severity\":\"warning\", | |
\"instance\": \"$name.example.net\" | |
}, | |
\"annotations\": { | |
\"summary\": \"High latency is high!\" | |
}, | |
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
}]" | |
echo "" |
This is useful! Thanks for posting it.
Very useful. Thanks !
Thanks!
Original doc: https://prometheus.io/docs/alerting/clients/
Thanks!
To save folks the frustration - this script actually doesn't send the resolve alert according to the v1 api spec. The prometheus contributors have point this out prometheus/alertmanager#1306. The alert only resets after a few minutes because alertmanager expects to get a list of currently firing alerts every 30s-3mins. In order to resolve the alert immediately after the curl you have to include the endsAt
timestamp:
url='http://localhost:9093/api/v1/alerts'
echo "Firing up alert"
curl -XPOST $url -d '[{"status": "firing","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00"}]'
echo ""
echo "press enter to resolve alert"
read
echo "sending resolve"
curl -XPOST $url -d '[{"status": "resolved","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00","endsAt": "2020-07-23T01:05:38+00:00"}]'
echo ""
Please note you'll have to update the startsAt and endsAt timestamps. I used this site to get RFC3339 compliant timestamps:
https://www.unixtimestamp.com/
So timestamp generation code as per suggestion above
startsAt=`date --iso-8601=seconds`
... # after the `read` command
endsAt=`date --iso-8601=seconds`
You just need to insert these shell variables into the curl command. This works for me. Using the --rfc-3339 instead of --iso-8601 I found T
separator missing which may or may not be an issue.
I've revisited this snippet multiple times each time having to make the alterations. Here's a gist incorporating all the changes https://gist.github.com/carinadigital/fd2960fdccd77dbdabc849656c43a070
Use this if you're on Mac/OSX: $(date -u '+%FT%TZ')
tks
single quote is your friend..