Skip to content

Instantly share code, notes, and snippets.

@cherti
Created December 9, 2016 13:47
Show Gist options
  • Save cherti/61ec48deaaab7d288c9fcf17e700853a to your computer and use it in GitHub Desktop.
Save cherti/61ec48deaaab7d288c9fcf17e700853a to your computer and use it in GitHub Desktop.
send a dummy alert to prometheus-alertmanager
#!/bin/bash
name=$RANDOM
url='http://localhost:9093/api/v1/alerts'
echo "firing up alert $name"
# change url o
curl -XPOST $url -d "[{
\"status\": \"firing\",
\"labels\": {
\"alertname\": \"$name\",
\"service\": \"my-service\",
\"severity\":\"warning\",
\"instance\": \"$name.example.net\"
},
\"annotations\": {
\"summary\": \"High latency is high!\"
},
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\"
}]"
echo ""
echo "press enter to resolve alert"
read
echo "sending resolve"
curl -XPOST $url -d "[{
\"status\": \"resolved\",
\"labels\": {
\"alertname\": \"$name\",
\"service\": \"my-service\",
\"severity\":\"warning\",
\"instance\": \"$name.example.net\"
},
\"annotations\": {
\"summary\": \"High latency is high!\"
},
\"generatorURL\": \"http://prometheus.int.example.net/<generating_expression>\"
}]"
echo ""
@jbarr13421
Copy link

single quote is your friend..

-d '{
  "alertname": "'$name'"
}'

@Daniel15
Copy link

This is useful! Thanks for posting it.

@abessifi
Copy link

Very useful. Thanks !

@BarakBD-Globality
Copy link

@msvprogs
Copy link

Thanks! 👍

@vanugrah
Copy link

vanugrah commented Jul 23, 2020

To save folks the frustration - this script actually doesn't send the resolve alert according to the v1 api spec. The prometheus contributors have point this out prometheus/alertmanager#1306. The alert only resets after a few minutes because alertmanager expects to get a list of currently firing alerts every 30s-3mins. In order to resolve the alert immediately after the curl you have to include the endsAt timestamp:

url='http://localhost:9093/api/v1/alerts'
echo "Firing up alert" 
curl -XPOST $url -d '[{"status": "firing","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00"}]'
echo ""

echo "press enter to resolve alert"
read

echo "sending resolve"
curl -XPOST $url -d '[{"status": "resolved","labels": {"alertname": "my_cool_alert","service": "curl","severity": "warning","instance": "0"},"annotations": {"summary": "This is a summary","description": "This is a description."},"generatorURL": "http://prometheus.int.example.net/<generating_expression>","startsAt": "2020-07-23T01:05:36+00:00","endsAt": "2020-07-23T01:05:38+00:00"}]'
echo ""

Please note you'll have to update the startsAt and endsAt timestamps. I used this site to get RFC3339 compliant timestamps:
https://www.unixtimestamp.com/

@madimadi
Copy link

So timestamp generation code as per suggestion above

startsAt=`date --iso-8601=seconds`
... # after the `read` command
endsAt=`date --iso-8601=seconds`

You just need to insert these shell variables into the curl command. This works for me. Using the --rfc-3339 instead of --iso-8601 I found T separator missing which may or may not be an issue.

@carinadigital
Copy link

I've revisited this snippet multiple times each time having to make the alterations. Here's a gist incorporating all the changes https://gist.github.com/carinadigital/fd2960fdccd77dbdabc849656c43a070

@BeyondEvil
Copy link

Use this if you're on Mac/OSX: $(date -u '+%FT%TZ')

@Panboo-leo
Copy link

tks

@nrukavkov
Copy link

nrukavkov commented Dec 11, 2023

You can pass different severity as argument to check different channels, for example:

sh test.sh 'critical'

#!/bin/bash

# Set default values
name=$RANDOM
url='https://alertmanager.local/api/v1/alerts'
summary='Testing summary!'
instance="$name.example.net"
default_severity='warning'

# Function to send alert
send_alert() {
    local status=$1
    local custom_severity=$2
    local current_severity=${custom_severity:-$default_severity}

    curl -XPOST $url -d "[
        {
            \"status\": \"$status\",
            \"labels\": {
                \"alertname\": \"$name\",
                \"service\": \"my-service\",
                \"severity\":\"$current_severity\",
                \"instance\": \"$instance\"
            },
            \"annotations\": {
                \"summary\": \"$summary\"
            },
            \"generatorURL\": \"https://prometheus.local/<generating_expression>\"
        }
    ]"
    echo ""
}

# Main script
echo "Firing up alert $name"
send_alert "firing" "$1"

read -p "Press enter to resolve alert"

echo "Sending resolve"
send_alert "resolved" "$1"

@saarw-opti
Copy link

saarw-opti commented Mar 8, 2024

You can pass different severity as argument to check different channels, for example:

sh test.sh 'critical'

#!/bin/bash

# Set default values
name=$RANDOM
url='https://alertmanager.local/api/v1/alerts'
summary='Testing summary!'
instance="$name.example.net"
default_severity='warning'

# Function to send alert
send_alert() {
    local status=$1
    local custom_severity=$2
    local current_severity=${custom_severity:-$default_severity}

    curl -XPOST $url -d "[
        {
            \"status\": \"$status\",
            \"labels\": {
                \"alertname\": \"$name\",
                \"service\": \"my-service\",
                \"severity\":\"$current_severity\",
                \"instance\": \"$instance\"
            },
            \"annotations\": {
                \"summary\": \"$summary\"
            },
            \"generatorURL\": \"https://prometheus.local/<generating_expression>\"
        }
    ]"
    echo ""
}

# Main script
echo "Firing up alert $name"
send_alert "firing" "$1"

read -p "Press enter to resolve alert"

echo "Sending resolve"
send_alert "resolved" "$1"

should change from v1 to v2 and add an header:

    -H "Content-Type: application/json" \

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment