#!/bin/bash | |
name=$RANDOM | |
url='http://localhost:9093/api/v1/alerts' | |
echo "firing up alert $name" | |
# change url o | |
curl -XPOST $url -d "[{ | |
\"status\": \"firing\", | |
\"labels\": { | |
\"alertname\": \"$name\", | |
\"service\": \"my-service\", | |
\"severity\":\"warning\", | |
\"instance\": \"$name.example.net\" | |
}, | |
\"annotations\": { | |
\"summary\": \"High latency is high!\" | |
}, | |
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
}]" | |
echo "" | |
echo "press enter to resolve alert" | |
read | |
echo "sending resolve" | |
curl -XPOST $url -d "[{ | |
\"status\": \"resolved\", | |
\"labels\": { | |
\"alertname\": \"$name\", | |
\"service\": \"my-service\", | |
\"severity\":\"warning\", | |
\"instance\": \"$name.example.net\" | |
}, | |
\"annotations\": { | |
\"summary\": \"High latency is high!\" | |
}, | |
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\" | |
}]" | |
echo "" |
This comment has been minimized.
This comment has been minimized.
This is useful! Thanks for posting it. |
This comment has been minimized.
This comment has been minimized.
Very useful. Thanks ! |
This comment has been minimized.
This comment has been minimized.
Thanks! |
This comment has been minimized.
This comment has been minimized.
Thanks! |
This comment has been minimized.
This comment has been minimized.
To save folks the frustration - this script actually doesn't send the resolve alert according to the v1 api spec. The prometheus contributors have point this out prometheus/alertmanager#1306. The alert only resets after a few minutes because alertmanager expects to get a list of currently firing alerts every 30s-3mins. In order to resolve the alert immediately after the curl you have to include the
Please note you'll have to update the startsAt and endsAt timestamps. I used this site to get RFC3339 compliant timestamps: |
This comment has been minimized.
This comment has been minimized.
So timestamp generation code as per suggestion above
You just need to insert these shell variables into the curl command. This works for me. Using the --rfc-3339 instead of --iso-8601 I found |
This comment has been minimized.
This comment has been minimized.
I've revisited this snippet multiple times each time having to make the alterations. Here's a gist incorporating all the changes https://gist.github.com/carinadigital/fd2960fdccd77dbdabc849656c43a070 |
This comment has been minimized.
single quote is your friend..