Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Monzo's Alertmanager Slack templates
###################################################
##
## Alertmanager YAML configuration for routing.
##
## Will route alerts with a code_owner label to the slack-code-owners receiver
## configured above, but will continue processing them to send to both a
## central Slack channel (slack-monitoring) and PagerDuty receivers
## (pd-warning and pd-critical)
##
routes:
###################################################
## Duplicate code_owner routes to teams
## These will send alerts to team channels but continue
## processing through the rest of the tree to handled by on-call
- match_re:
code_owner: '.+'
routes:
- match: {severity: info|warning|critical}
continue: true
receiver: slack-code-owners
###################################################
## Standard on-call routes
- match_re:
severity: info|warning|critical
receiver: slack-monitoring
continue: true
###################################################
## Pagerduty routes
- match: {severity: warning}
routes:
- receiver: pd-warning
- match: {severity: critical}
routes:
- receiver: pd-critical
receivers:
###################################################
## Slack Receivers
- name: slack-code-owners
slack_configs:
- channel: '#{{- template "slack.monzo.code_owner_channel" . -}}'
send_resolved: true
title: '{{ template "slack.monzo.title" . }}'
icon_emoji: '{{ template "slack.monzo.icon_emoji" . }}'
color: '{{ template "slack.monzo.color" . }}'
text: '{{ template "slack.monzo.text" . }}'
actions:
- type: button
text: 'Runbook :green_book:'
url: '{{ (index .Alerts 0).Annotations.runbook }}'
- type: button
text: 'Query :mag:'
url: '{{ (index .Alerts 0).GeneratorURL }}'
- type: button
text: 'Dashboard :grafana:'
url: '{{ (index .Alerts 0).Annotations.dashboard }}'
- type: button
text: 'Silence :no_bell:'
url: '{{ template "__alert_silence_link" . }}'
- type: button
text: '{{ template "slack.monzo.link_button_text" . }}'
url: '{{ .CommonAnnotations.link_url }}'
# This builds the silence URL. We exclude the alertname in the range
# to avoid the issue of having trailing comma separator (%2C) at the end
# of the generated URL
{{ define "__alert_silence_link" -}}
{{ .ExternalURL }}/#/silences/new?filter=%7B
{{- range .CommonLabels.SortedPairs -}}
{{- if ne .Name "alertname" -}}
{{- .Name }}%3D"{{- .Value -}}"%2C%20
{{- end -}}
{{- end -}}
alertname%3D"{{ .CommonLabels.alertname }}"%7D
{{- end }}
{{ define "__alert_severity_prefix" -}}
{{ if ne .Status "firing" -}}
:lgtm:
{{- else if eq .Labels.severity "critical" -}}
:fire:
{{- else if eq .Labels.severity "warning" -}}
:warning:
{{- else -}}
:question:
{{- end }}
{{- end }}
{{ define "__alert_severity_prefix_title" -}}
{{ if ne .Status "firing" -}}
:lgtm:
{{- else if eq .CommonLabels.severity "critical" -}}
:fire:
{{- else if eq .CommonLabels.severity "warning" -}}
:warning:
{{- else if eq .CommonLabels.severity "info" -}}
:information_source:
{{- else -}}
:question:
{{- end }}
{{- end }}
{{/* First line of Slack alerts */}}
{{ define "slack.monzo.title" -}}
[{{ .Status | toUpper -}}
{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{- end -}}
] {{ template "__alert_severity_prefix_title" . }} {{ .CommonLabels.alertname }}
{{- end }}
{{/* Color of Slack attachment (appears as line next to alert )*/}}
{{ define "slack.monzo.color" -}}
{{ if eq .Status "firing" -}}
{{ if eq .CommonLabels.severity "warning" -}}
warning
{{- else if eq .CommonLabels.severity "critical" -}}
danger
{{- else -}}
#439FE0
{{- end -}}
{{ else -}}
good
{{- end }}
{{- end }}
{{/* Emoji to display as user icon (custom emoji supported!) */}}
{{ define "slack.monzo.icon_emoji" }}:prometheus:{{ end }}
{{/* The test to display in the alert */}}
{{ define "slack.monzo.text" -}}
{{ range .Alerts }}
{{- if .Annotations.message }}
{{ .Annotations.message }}
{{- end }}
{{- if .Annotations.description }}
{{ .Annotations.description }}
{{- end }}
{{- end }}
{{- end }}
{{- /* If none of the below matches, send to #monitoring-no-owner, and we
can then assign the expected code_owner to the alert or map the code_owner
to the correct channel */ -}}
{{ define "__get_channel_for_code_owner" -}}
{{- if eq . "platform-team" -}}
platform-alerts
{{- else if eq . "security-team" -}}
security-alerts
{{- else -}}
monitoring-no-owner
{{- end -}}
{{- end }}
{{- /* Select the channel based on the code_owner. We only expect to get
into this template function if the code_owners label is present on an alert.
This is to defend against us accidentally breaking the routing logic. */ -}}
{{ define "slack.monzo.code_owner_channel" -}}
{{- if .CommonLabels.code_owner }}
{{ template "__get_channel_for_code_owner" .CommonLabels.code_owner }}
{{- else -}}
monitoring
{{- end }}
{{- end }}
{{ define "slack.monzo.link_button_text" -}}
{{- if .CommonAnnotations.link_text -}}
{{- .CommonAnnotations.link_text -}}
{{- else -}}
Link
{{- end }} :link:
{{- end }}
@nhuray
Copy link

nhuray commented Feb 2, 2020

Can you share a screenshot of what it looks like in Slack ?

@manikanta-kondeti
Copy link

manikanta-kondeti commented Feb 4, 2020

screenshots of how it looks?

@Dmitry1987
Copy link

Dmitry1987 commented Feb 6, 2020

nice one, thanks for sharing. I think I'll try these.

@Sallyan
Copy link

Sallyan commented Mar 26, 2020

Hi, Thanks for sharing. I tried this.
but why it only shows the last one button?

@caiodelgadonew
Copy link

caiodelgadonew commented Apr 7, 2020

Hi, Thanks for sharing. I tried this.
but why it only shows the last one button?

Probabbly because your source doesn't have the metadata to show everything.

I'm getting those infos from prometheus, when the alert doesn't have the runbook the runbook button isn't shown, and so on

Copy link

ghost commented Apr 7, 2020

this doesnt work. button urls cant take parameters. thanks anyway.

@rust84
Copy link

rust84 commented Apr 18, 2020

These are great. Thanks for sharing.

@vosdev
Copy link

vosdev commented May 15, 2020

I used some snippets for our own! Thank you very much.

We also ran into an issue that I just solved, and I would like to share my solution with you so that you can fix it upstream.

Some of our label values had = marks in them. Because of this, Alertmanager was unable to parse the filter in the URL.

Fixed by url encoding the label value:

{{ define "__alert_silence_link" -}}
    {{ .ExternalURL }}/#/silences/new?filter=%7B
    {{- range .CommonLabels.SortedPairs -}}
        {{- if ne .Name "alertname" -}}
-            {{- .Name }}%3D"{{- .Value -}}"%2C%20
+            {{- .Name }}%3D"{{- .Value | urlquery -}}"%2C%20
        {{- end -}}
    {{- end -}}
    alertname%3D"{{ .CommonLabels.alertname }}"%7D
{{- end }}

@k1rk
Copy link

k1rk commented May 20, 2020

this is great, thank you

screenshot for those who asked
Screenshot 2020-05-20 at 8 26 52 PM

@yevgeniyo
Copy link

yevgeniyo commented May 25, 2020

Awesome thx, why second, third alert has indentation?

@dewey
Copy link

dewey commented Oct 21, 2020

Awesome thx, why second, third alert has indentation?

It's not an indentation, it's censoring the name of their Prometheus instance

@pyrodex
Copy link

pyrodex commented Oct 13, 2021

Anyone able to get the silence working? I have external URL defined but the buttons never come across. Not sure how to debug it. The Query button works and everything else but no Silence button.

@80kk
Copy link

80kk commented Oct 15, 2021

Anyone able to get the silence working? I have external URL defined but the buttons never come across. Not sure how to debug it. The Query button works and everything else but no Silence button.

To make buttons visible you need to add:
value field:

          actions:
            - type: button
              text: 'Text on the button :fire:'
              value: 'text'
              url: 'https://url.com'

@pyrodex
Copy link

pyrodex commented Oct 15, 2021

Anyone able to get the silence working? I have external URL defined but the buttons never come across. Not sure how to debug it. The Query button works and everything else but no Silence button.

To make buttons visible you need to add: value field:

          actions:
            - type: button
              text: 'Text on the button :fire:'
              value: 'text'
              url: 'https://url.com'

Nope... Are you using the automatically generated Silence function?

@80kk
Copy link

80kk commented Oct 15, 2021

Nope... Are you using the automatically generated Silence function?

Sorry I misread your initial message. I thought you can't make buttons visible at all.

@core3750x
Copy link

core3750x commented Oct 15, 2021

Hi, I can't get silence button visible on TargetMissing alert. How can I fix it?

@NerijusMacaitis
Copy link

NerijusMacaitis commented Feb 22, 2022

How you get values in the slack message? Do I need to use specific labeling or annotations for each alert?
Title, severity and summary seems to work fine for me at this point.

@max06
Copy link

max06 commented Mar 16, 2022

I've had similar issues with the silence button, sometimes showing, sometimes not. It turned out, the request URL was malformed due to Spaces in the label values like job="Node Exporter" (whoever that bright idea had).

Fix:

{{ define "__alert_silence_link" -}}
    {{ .ExternalURL }}/#/silences/new?filter=%7B
    {{- range .CommonLabels.SortedPairs -}}
        {{- if ne .Name "alertname" -}}
            {{- .Name }}%3D"{{- reReplaceAll " +" "%20" .Value -}}"%2C%20
        {{- end -}}
    {{- end -}}
    alertname%3D"{{- .CommonLabels.alertname -}}"%7D
{{- end }}

Not sure yet why slack still accepted and displayed the broken message, just without that silence button.

@BenjaminS-81
Copy link

BenjaminS-81 commented Apr 6, 2022

Did anyone happen find template examples of PagerDuty for AlertManager if there are any? (like pagerduty.tmpl ?)

@rs16w
Copy link

rs16w commented Apr 18, 2022

Can somebody help me, please
I want to do words with hyperlink, how i can do it?

@URDead4me
Copy link

URDead4me commented Apr 18, 2022

Hey @rs16w ,

Regarding your question, checkout this page - https://api.slack.com/legacy/interactive-message-field-guide#action_payload
You can create a type: "button" that will have a URL opened once clicked on as follows:

- name: custom-slack-config-name
  slack_configs:
  - api_url: https://hooks.slack.com/services/YOUR-SLACK-CHANNEL
    channel: '#channel-name'
    send_resolved: true
    title: like-in-main-example
    icon_emoji: like-in-main-example
    color: like-in-main-example
    text: like-in-main-example
    actions:
    - type: button
      text: 'Google Button :laughing:'
      url: https:google.com

Or you may try using action like:
- type: interactive_message
Or
- type: select
You should read about their structure also in the same link i have shared above

Hope it helped 😄

@naviat
Copy link

naviat commented Apr 28, 2022

@milesbxf Thanks for your big values, I tried to apply this my way but I cannot see any icon or information about Dashboard URL. Runbook, Query, and Silent work well.

- type: button
  text: "Dashboard :chart_with_upwards_trend:"
  url: "{{ (index .Alerts 0).Annotations.dashboard_url }}"

@bzakrzewski1
Copy link

bzakrzewski1 commented May 2, 2022

Thank for the template, I managed to adjust it to our needs.

Is there any way to add a hostname or IP address to the alert?

I've been trying the following method, then use .Labels.ip in the template but it doesn't work.

image

image

The above solution simply prints {{ $ip }} as a text.

It'd be helpful if I could preview the full message payload that's being sent from Alert Manager, but seems like it's not possible.

Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment