Skip to content

Instantly share code, notes, and snippets.

@milesbxf
Last active November 25, 2024 17:07
Show Gist options
  • Save milesbxf/e2744fc90e9c41b47aa47925f8ff6512 to your computer and use it in GitHub Desktop.
Save milesbxf/e2744fc90e9c41b47aa47925f8ff6512 to your computer and use it in GitHub Desktop.
Monzo's Alertmanager Slack templates
###################################################
##
## Alertmanager YAML configuration for routing.
##
## Will route alerts with a code_owner label to the slack-code-owners receiver
## configured above, but will continue processing them to send to both a
## central Slack channel (slack-monitoring) and PagerDuty receivers
## (pd-warning and pd-critical)
##
routes:
###################################################
## Duplicate code_owner routes to teams
## These will send alerts to team channels but continue
## processing through the rest of the tree to handled by on-call
- match_re:
code_owner: '.+'
routes:
- match: {severity: info|warning|critical}
continue: true
receiver: slack-code-owners
###################################################
## Standard on-call routes
- match_re:
severity: info|warning|critical
receiver: slack-monitoring
continue: true
###################################################
## Pagerduty routes
- match: {severity: warning}
routes:
- receiver: pd-warning
- match: {severity: critical}
routes:
- receiver: pd-critical
receivers:
###################################################
## Slack Receivers
- name: slack-code-owners
slack_configs:
- channel: '#{{- template "slack.monzo.code_owner_channel" . -}}'
send_resolved: true
title: '{{ template "slack.monzo.title" . }}'
icon_emoji: '{{ template "slack.monzo.icon_emoji" . }}'
color: '{{ template "slack.monzo.color" . }}'
text: '{{ template "slack.monzo.text" . }}'
actions:
- type: button
text: 'Runbook :green_book:'
url: '{{ (index .Alerts 0).Annotations.runbook }}'
- type: button
text: 'Query :mag:'
url: '{{ (index .Alerts 0).GeneratorURL }}'
- type: button
text: 'Dashboard :grafana:'
url: '{{ (index .Alerts 0).Annotations.dashboard }}'
- type: button
text: 'Silence :no_bell:'
url: '{{ template "__alert_silence_link" . }}'
- type: button
text: '{{ template "slack.monzo.link_button_text" . }}'
url: '{{ .CommonAnnotations.link_url }}'
# This builds the silence URL. We exclude the alertname in the range
# to avoid the issue of having trailing comma separator (%2C) at the end
# of the generated URL
{{ define "__alert_silence_link" -}}
{{ .ExternalURL }}/#/silences/new?filter=%7B
{{- range .CommonLabels.SortedPairs -}}
{{- if ne .Name "alertname" -}}
{{- .Name }}%3D"{{- .Value -}}"%2C%20
{{- end -}}
{{- end -}}
alertname%3D"{{ .CommonLabels.alertname }}"%7D
{{- end }}
{{ define "__alert_severity_prefix" -}}
{{ if ne .Status "firing" -}}
:lgtm:
{{- else if eq .Labels.severity "critical" -}}
:fire:
{{- else if eq .Labels.severity "warning" -}}
:warning:
{{- else -}}
:question:
{{- end }}
{{- end }}
{{ define "__alert_severity_prefix_title" -}}
{{ if ne .Status "firing" -}}
:lgtm:
{{- else if eq .CommonLabels.severity "critical" -}}
:fire:
{{- else if eq .CommonLabels.severity "warning" -}}
:warning:
{{- else if eq .CommonLabels.severity "info" -}}
:information_source:
{{- else -}}
:question:
{{- end }}
{{- end }}
{{/* First line of Slack alerts */}}
{{ define "slack.monzo.title" -}}
[{{ .Status | toUpper -}}
{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{- end -}}
] {{ template "__alert_severity_prefix_title" . }} {{ .CommonLabels.alertname }}
{{- end }}
{{/* Color of Slack attachment (appears as line next to alert )*/}}
{{ define "slack.monzo.color" -}}
{{ if eq .Status "firing" -}}
{{ if eq .CommonLabels.severity "warning" -}}
warning
{{- else if eq .CommonLabels.severity "critical" -}}
danger
{{- else -}}
#439FE0
{{- end -}}
{{ else -}}
good
{{- end }}
{{- end }}
{{/* Emoji to display as user icon (custom emoji supported!) */}}
{{ define "slack.monzo.icon_emoji" }}:prometheus:{{ end }}
{{/* The test to display in the alert */}}
{{ define "slack.monzo.text" -}}
{{ range .Alerts }}
{{- if .Annotations.message }}
{{ .Annotations.message }}
{{- end }}
{{- if .Annotations.description }}
{{ .Annotations.description }}
{{- end }}
{{- end }}
{{- end }}
{{- /* If none of the below matches, send to #monitoring-no-owner, and we
can then assign the expected code_owner to the alert or map the code_owner
to the correct channel */ -}}
{{ define "__get_channel_for_code_owner" -}}
{{- if eq . "platform-team" -}}
platform-alerts
{{- else if eq . "security-team" -}}
security-alerts
{{- else -}}
monitoring-no-owner
{{- end -}}
{{- end }}
{{- /* Select the channel based on the code_owner. We only expect to get
into this template function if the code_owners label is present on an alert.
This is to defend against us accidentally breaking the routing logic. */ -}}
{{ define "slack.monzo.code_owner_channel" -}}
{{- if .CommonLabels.code_owner }}
{{ template "__get_channel_for_code_owner" .CommonLabels.code_owner }}
{{- else -}}
monitoring
{{- end }}
{{- end }}
{{ define "slack.monzo.link_button_text" -}}
{{- if .CommonAnnotations.link_text -}}
{{- .CommonAnnotations.link_text -}}
{{- else -}}
Link
{{- end }} :link:
{{- end }}
@max06
Copy link

max06 commented Mar 16, 2022

I've had similar issues with the silence button, sometimes showing, sometimes not. It turned out, the request URL was malformed due to Spaces in the label values like job="Node Exporter" (whoever that bright idea had).

Fix:

{{ define "__alert_silence_link" -}}
    {{ .ExternalURL }}/#/silences/new?filter=%7B
    {{- range .CommonLabels.SortedPairs -}}
        {{- if ne .Name "alertname" -}}
            {{- .Name }}%3D"{{- reReplaceAll " +" "%20" .Value -}}"%2C%20
        {{- end -}}
    {{- end -}}
    alertname%3D"{{- .CommonLabels.alertname -}}"%7D
{{- end }}

Not sure yet why slack still accepted and displayed the broken message, just without that silence button.

@BenjaminS-81
Copy link

Did anyone happen find template examples of PagerDuty for AlertManager if there are any? (like pagerduty.tmpl ?)

@rs16w
Copy link

rs16w commented Apr 18, 2022

Can somebody help me, please
I want to do words with hyperlink, how i can do it?

@benjisho
Copy link

benjisho commented Apr 18, 2022

Hey @rs16w ,

Regarding your question, checkout this page - https://api.slack.com/legacy/interactive-message-field-guide#action_payload
You can create a type: "button" that will have a URL opened once clicked on as follows:

- name: custom-slack-config-name
  slack_configs:
  - api_url: https://hooks.slack.com/services/YOUR-SLACK-CHANNEL
    channel: '#channel-name'
    send_resolved: true
    title: like-in-main-example
    icon_emoji: like-in-main-example
    color: like-in-main-example
    text: like-in-main-example
    actions:
    - type: button
      text: 'Google Button :laughing:'
      url: https:google.com

Or you may try using action like:
- type: interactive_message
Or
- type: select
You should read about their structure also in the same link i have shared above

Hope it helped 😄

@naviat
Copy link

naviat commented Apr 28, 2022

@milesbxf Thanks for your big values, I tried to apply this my way but I cannot see any icon or information about Dashboard URL. Runbook, Query, and Silent work well.

- type: button
  text: "Dashboard :chart_with_upwards_trend:"
  url: "{{ (index .Alerts 0).Annotations.dashboard_url }}"

@bzakrzewski1
Copy link

Thank for the template, I managed to adjust it to our needs.

Is there any way to add a hostname or IP address to the alert?

I've been trying the following method, then use .Labels.ip in the template but it doesn't work.

image

image

The above solution simply prints {{ $ip }} as a text.

It'd be helpful if I could preview the full message payload that's being sent from Alert Manager, but seems like it's not possible.

Any ideas?

@rajnikhil17
Copy link

Can we get extra labels into the slack notification?
I am new to this templating language.
Template will help

@savan4yk
Copy link

savan4yk commented Aug 8, 2022

Thank for the template.
If anybody has issue with "Silence" button on mobile devices (when press Silence on phone nothing happens, button doesn't open Alertmanager silence link), just change quotes (") to (%22) in __alert_silence_link template.
Hope this help.

@ioncoa
Copy link

ioncoa commented Aug 9, 2022

Awesome thx, why second, third alert has indentation?

It's not an indentation, it's censoring the name of their Prometheus instance

Hi @dewey ! ,

I think there is not about censoring , but actually some indentation appears at second , third , x, item of a range .
My example :

SSL certificate expires in less than 30 days for instance A .
           SSL certificate expires in less than 30 days for instance B.
           SSL certificate expires in less than 30 days for instance C .
           SSL certificate expires in less than 30 days for instance D.

No idea , how to get rid of it .
I changed the definition of text to take only description annotations in template but unfortunately it doesn't work .

                {{/* The text to display in the alert */}}
                {{ define "slack.monzo.text" -}}
                    {{ range .Alerts }}
                        {{- if .Annotations.description }}
                            {{ .Annotations.description }}
                        {{- end }}
                    {{- end }}
                {{- end }}

Have anyone ideas ?

@brovoca
Copy link

brovoca commented Aug 25, 2022

@ioncoa Remove the indentation from the go template function. E.g.

{{/* The text to display in the alert */}}
{{ define "slack.monzo.text" -}}
{{ range .Alerts }}
{{- if .Annotations.description }}
{{ .Annotations.description }}
{{- end }}
{{- end }}
{{- end }}

@aorfanos
Copy link

aorfanos commented Oct 1, 2022

Addition for Grafana Alerting integration with AlertManager:

ℹ️ You have to set alertmanger.externalUrl to your Grafana instance external URL

    {{ define "__grafana_alert_silence_link" -}}
        {{ .ExternalURL }}/alerting/silence/new?
        {{- range .CommonLabels.SortedPairs -}}
            matcher={{- .Name }}%3D{{- .Value | urlquery -}}&
        {{- end -}}
    {{- end }}

@audunsolemdal
Copy link

Addition for Grafana Alerting integration with AlertManager:

ℹ️ You have to set alertmanger.externalUrl to your Grafana instance external URL

    {{ define "__grafana_alert_silence_link" -}}
        {{ .ExternalURL }}/alerting/silence/new?
        {{- range .CommonLabels.SortedPairs -}}
            matcher={{- .Name }}%3D{{- .Value | urlquery -}}&
        {{- end -}}
    {{- end }}

Thanks @aorfanos

Have you also found a way to get the query button working towards Grafana? Alternatively, an external Prometheus?

@Slapper
Copy link

Slapper commented Apr 29, 2023

Hello,

Thanks for sharing :)

In case you have built in Grafana Alerting/silence but also using Alertmanager as an additional Data source you have to append alertmanager=Alertmanager at the end of the url .

      {{ define "__grafana_alert_silence_link" -}}
        {{ .ExternalURL }}/alerting/silence/new?
        {{- range .CommonLabels.SortedPairs -}}
            matcher={{- .Name }}%3D{{- .Value | urlquery -}}&
        {{- end -}}
        alertmanager=Alertmanager
        {{- end }}

P.S : tested on alertmanager 0.25 / grafana 9.1.6

Cheers

@arnoldyahad
Copy link

Hey everyone,
I want to share some findings as i tried to implement what is written here and got some insights that might help you:

  1. name field for slack button:
    if you dont specify a name for a slack button, you will only have 1 slack button as they need it to diffrenciate between the buttons in the list. here is where i found it in their documents: https://api.slack.com/legacy/interactive-message-field-guide

  2. if your button links to a url it must contain http:// or https:// i had a button with: grafana.com and it didn't work, only after i made it https://grafana.com it worked.

  3. the advise by @80kk was crucial for the buttons, the value: 'test' field was mandatory for me to make it work.

so this is my configuration for working buttons:

          - type: button
            name: mutebutton
            text: 'Mute :prometheus:'
            value: 'text'
            url: "https://{{ (index .Alerts 0).Labels.alertManagerUrl }}" # just a link for our internal alert manager, with added https://
          - type: button
            name: readmebutton
            text: 'README :green_book:'
            value: 'text'
            url: "{{ (index .Alerts 0).Labels.readmeUrl }}" # just a link for our internal readme, it comes with https:// so no need to add
          - type: button
            name: grafana
            text: 'Grafana :grafana:'
            value: 'text'
            url: 'https://{{ (index .Alerts 0).Labels.grafanaAlertUrl }}' # same as first button

this is the final result:
image

and thanks everyone for all the comments here!

@stefaneacsu147
Copy link

stefaneacsu147 commented May 29, 2023

      actions:
      - type: button
        text: 'README :green_book:'
        url: '{{ (index .Alerts 0).Annotations.runbook_url }}'
      - type: button
        text: 'Query :mag:'
        url: '{{ (index .Alerts 0).GeneratorURL }}'
      - type: button
        text: 'Dashboard :chart_with_upwards_trend:'
        url: '{{ (index .Alerts 0).Annotations.dashboard_url }}'
      - type: button
        text: 'Silence :no_bell:'
        url: '{{ template "__alert_silence_link" . }}'
EOF

I have my buttons set up like this and I can only get in Slack the "Runbook" one to be shown. Any idea why it's not showing the Query, Dashboard or Silence button as well?

I have tried the value: 'text' but it seems not to make a difference.

@arnoldyahad
Copy link

@stefaneacsu147 see my previous answer, you might need to add name: field to each button.

@amirnoshian
Copy link

hi guys how can use this? should you create docs dude

@Zorato
Copy link

Zorato commented Jul 17, 2024

If someone has missing Silence button, then double check your alert name. If alert name has spaces, then your link is malformed.
You can use this template

{{ define "__alert_silence_link" -}}
          {{ .ExternalURL }}/#/silences/new?filter=%7B
          {{- range .CommonLabels.SortedPairs -}}
              {{- if ne .Name "alertname" -}}
                  {{- .Name }}%3D"{{- .Value | urlquery -}}"%2C%20
              {{- end -}}
          {{- end -}}
          alertname%3D"{{- .CommonLabels.alertname | urlquery  -}}"%7D
      {{- end }}

Notice .CommonLabels.alertname | urlquery change in the end.
Buttons are going to disappear if you link is malformed (i.e. non-urlencoded spaces or other chars). The best way to check it is to add the same link to message text and check the result:

  receivers:
    - name: slack-receiver
      slack_configs:
        - channel: "#alerts" # slack channel for notrifications
          send_resolved: true
          color: '{{ template "slack.monzo.color" . }}'
          title: '{{ template "slack.monzo.title" . }}'
          text: '{{ template "slack.monzo.text" . }} LINK: `{{ template "__alert_silence_link" . }}` '

Hope this will help!

@ariefMudahMy
Copy link

My case is there are too many labels value with a space " ",
this is how i solved it.

{{ define "__alert_silence_link" -}}
    {{ .ExternalURL }}/#/silences/new?filter=%7B
    {{- range .CommonLabels.SortedPairs -}}
        {{- if ne .Name "alertname" -}}
            {{- .Name }}%3D"{{- urlquery .Value | reReplaceAll "\\+" "%20" -}}"%2C%20
        {{- end -}}
    {{- end -}}
    alertname%3D"{{- urlquery .CommonLabels.alertname | reReplaceAll "\\+" "%20" -}}"%7D
{{- end }}

Hope this will help solving the same case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment