| ################################################### | |
| ## | |
| ## Alertmanager YAML configuration for routing. | |
| ## | |
| ## Will route alerts with a code_owner label to the slack-code-owners receiver | |
| ## configured above, but will continue processing them to send to both a | |
| ## central Slack channel (slack-monitoring) and PagerDuty receivers | |
| ## (pd-warning and pd-critical) | |
| ## | |
| routes: | |
| ################################################### | |
| ## Duplicate code_owner routes to teams | |
| ## These will send alerts to team channels but continue | |
| ## processing through the rest of the tree to handled by on-call | |
| - match_re: | |
| code_owner: '.+' | |
| routes: | |
| - match: {severity: info|warning|critical} | |
| continue: true | |
| receiver: slack-code-owners | |
| ################################################### | |
| ## Standard on-call routes | |
| - match_re: | |
| severity: info|warning|critical | |
| receiver: slack-monitoring | |
| continue: true | |
| ################################################### | |
| ## Pagerduty routes | |
| - match: {severity: warning} | |
| routes: | |
| - receiver: pd-warning | |
| - match: {severity: critical} | |
| routes: | |
| - receiver: pd-critical |
| receivers: | |
| ################################################### | |
| ## Slack Receivers | |
| - name: slack-code-owners | |
| slack_configs: | |
| - channel: '#{{- template "slack.monzo.code_owner_channel" . -}}' | |
| send_resolved: true | |
| title: '{{ template "slack.monzo.title" . }}' | |
| icon_emoji: '{{ template "slack.monzo.icon_emoji" . }}' | |
| color: '{{ template "slack.monzo.color" . }}' | |
| text: '{{ template "slack.monzo.text" . }}' | |
| actions: | |
| - type: button | |
| text: 'Runbook :green_book:' | |
| url: '{{ (index .Alerts 0).Annotations.runbook }}' | |
| - type: button | |
| text: 'Query :mag:' | |
| url: '{{ (index .Alerts 0).GeneratorURL }}' | |
| - type: button | |
| text: 'Dashboard :grafana:' | |
| url: '{{ (index .Alerts 0).Annotations.dashboard }}' | |
| - type: button | |
| text: 'Silence :no_bell:' | |
| url: '{{ template "__alert_silence_link" . }}' | |
| - type: button | |
| text: '{{ template "slack.monzo.link_button_text" . }}' | |
| url: '{{ .CommonAnnotations.link_url }}' |
| # This builds the silence URL. We exclude the alertname in the range | |
| # to avoid the issue of having trailing comma separator (%2C) at the end | |
| # of the generated URL | |
| {{ define "__alert_silence_link" -}} | |
| {{ .ExternalURL }}/#/silences/new?filter=%7B | |
| {{- range .CommonLabels.SortedPairs -}} | |
| {{- if ne .Name "alertname" -}} | |
| {{- .Name }}%3D"{{- .Value -}}"%2C%20 | |
| {{- end -}} | |
| {{- end -}} | |
| alertname%3D"{{ .CommonLabels.alertname }}"%7D | |
| {{- end }} | |
| {{ define "__alert_severity_prefix" -}} | |
| {{ if ne .Status "firing" -}} | |
| :lgtm: | |
| {{- else if eq .Labels.severity "critical" -}} | |
| :fire: | |
| {{- else if eq .Labels.severity "warning" -}} | |
| :warning: | |
| {{- else -}} | |
| :question: | |
| {{- end }} | |
| {{- end }} | |
| {{ define "__alert_severity_prefix_title" -}} | |
| {{ if ne .Status "firing" -}} | |
| :lgtm: | |
| {{- else if eq .CommonLabels.severity "critical" -}} | |
| :fire: | |
| {{- else if eq .CommonLabels.severity "warning" -}} | |
| :warning: | |
| {{- else if eq .CommonLabels.severity "info" -}} | |
| :information_source: | |
| {{- else -}} | |
| :question: | |
| {{- end }} | |
| {{- end }} | |
| {{/* First line of Slack alerts */}} | |
| {{ define "slack.monzo.title" -}} | |
| [{{ .Status | toUpper -}} | |
| {{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{- end -}} | |
| ] {{ template "__alert_severity_prefix_title" . }} {{ .CommonLabels.alertname }} | |
| {{- end }} | |
| {{/* Color of Slack attachment (appears as line next to alert )*/}} | |
| {{ define "slack.monzo.color" -}} | |
| {{ if eq .Status "firing" -}} | |
| {{ if eq .CommonLabels.severity "warning" -}} | |
| warning | |
| {{- else if eq .CommonLabels.severity "critical" -}} | |
| danger | |
| {{- else -}} | |
| #439FE0 | |
| {{- end -}} | |
| {{ else -}} | |
| good | |
| {{- end }} | |
| {{- end }} | |
| {{/* Emoji to display as user icon (custom emoji supported!) */}} | |
| {{ define "slack.monzo.icon_emoji" }}:prometheus:{{ end }} | |
| {{/* The test to display in the alert */}} | |
| {{ define "slack.monzo.text" -}} | |
| {{ range .Alerts }} | |
| {{- if .Annotations.message }} | |
| {{ .Annotations.message }} | |
| {{- end }} | |
| {{- if .Annotations.description }} | |
| {{ .Annotations.description }} | |
| {{- end }} | |
| {{- end }} | |
| {{- end }} | |
| {{- /* If none of the below matches, send to #monitoring-no-owner, and we | |
| can then assign the expected code_owner to the alert or map the code_owner | |
| to the correct channel */ -}} | |
| {{ define "__get_channel_for_code_owner" -}} | |
| {{- if eq . "platform-team" -}} | |
| platform-alerts | |
| {{- else if eq . "security-team" -}} | |
| security-alerts | |
| {{- else -}} | |
| monitoring-no-owner | |
| {{- end -}} | |
| {{- end }} | |
| {{- /* Select the channel based on the code_owner. We only expect to get | |
| into this template function if the code_owners label is present on an alert. | |
| This is to defend against us accidentally breaking the routing logic. */ -}} | |
| {{ define "slack.monzo.code_owner_channel" -}} | |
| {{- if .CommonLabels.code_owner }} | |
| {{ template "__get_channel_for_code_owner" .CommonLabels.code_owner }} | |
| {{- else -}} | |
| monitoring | |
| {{- end }} | |
| {{- end }} | |
| {{ define "slack.monzo.link_button_text" -}} | |
| {{- if .CommonAnnotations.link_text -}} | |
| {{- .CommonAnnotations.link_text -}} | |
| {{- else -}} | |
| Link | |
| {{- end }} :link: | |
| {{- end }} |
How you get values in the slack message? Do I need to use specific labeling or annotations for each alert?
Title, severity and summary seems to work fine for me at this point.
I've had similar issues with the silence button, sometimes showing, sometimes not. It turned out, the request URL was malformed due to Spaces in the label values like job="Node Exporter" (whoever that bright idea had).
Fix:
{{ define "__alert_silence_link" -}}
{{ .ExternalURL }}/#/silences/new?filter=%7B
{{- range .CommonLabels.SortedPairs -}}
{{- if ne .Name "alertname" -}}
{{- .Name }}%3D"{{- reReplaceAll " +" "%20" .Value -}}"%2C%20
{{- end -}}
{{- end -}}
alertname%3D"{{- .CommonLabels.alertname -}}"%7D
{{- end }}Not sure yet why slack still accepted and displayed the broken message, just without that silence button.
Did anyone happen find template examples of PagerDuty for AlertManager if there are any? (like pagerduty.tmpl ?)
Can somebody help me, please
I want to do words with hyperlink, how i can do it?
Hey @rs16w ,
Regarding your question, checkout this page - https://api.slack.com/legacy/interactive-message-field-guide#action_payload
You can create a type: "button" that will have a URL opened once clicked on as follows:
- name: custom-slack-config-name
slack_configs:
- api_url: https://hooks.slack.com/services/YOUR-SLACK-CHANNEL
channel: '#channel-name'
send_resolved: true
title: like-in-main-example
icon_emoji: like-in-main-example
color: like-in-main-example
text: like-in-main-example
actions:
- type: button
text: 'Google Button :laughing:'
url: https:google.com
Or you may try using action like:
- type: interactive_message
Or
- type: select
You should read about their structure also in the same link i have shared above
Hope it helped
@milesbxf Thanks for your big values, I tried to apply this my way but I cannot see any icon or information about Dashboard URL. Runbook, Query, and Silent work well.
- type: button
text: "Dashboard :chart_with_upwards_trend:"
url: "{{ (index .Alerts 0).Annotations.dashboard_url }}"
Thank for the template, I managed to adjust it to our needs.
Is there any way to add a hostname or IP address to the alert?
I've been trying the following method, then use .Labels.ip in the template but it doesn't work.
The above solution simply prints {{ $ip }} as a text.
It'd be helpful if I could preview the full message payload that's being sent from Alert Manager, but seems like it's not possible.
Any ideas?
Can we get extra labels into the slack notification?
I am new to this templating language.
Template will help
Thank for the template.
If anybody has issue with "Silence" button on mobile devices (when press Silence on phone nothing happens, button doesn't open Alertmanager silence link), just change quotes (") to (%22) in __alert_silence_link template.
Hope this help.
Awesome thx, why second, third alert has indentation?
It's not an indentation, it's censoring the name of their Prometheus instance
Hi @dewey ! ,
I think there is not about censoring , but actually some indentation appears at second , third , x, item of a range .
My example :
SSL certificate expires in less than 30 days for instance A .
SSL certificate expires in less than 30 days for instance B.
SSL certificate expires in less than 30 days for instance C .
SSL certificate expires in less than 30 days for instance D.
No idea , how to get rid of it .
I changed the definition of text to take only description annotations in template but unfortunately it doesn't work .
{{/* The text to display in the alert */}}
{{ define "slack.monzo.text" -}}
{{ range .Alerts }}
{{- if .Annotations.description }}
{{ .Annotations.description }}
{{- end }}
{{- end }}
{{- end }}
Have anyone ideas ?
@ioncoa Remove the indentation from the go template function. E.g.
{{/* The text to display in the alert */}}
{{ define "slack.monzo.text" -}}
{{ range .Alerts }}
{{- if .Annotations.description }}
{{ .Annotations.description }}
{{- end }}
{{- end }}
{{- end }}
Addition for Grafana Alerting integration with AlertManager:
alertmanger.externalUrl to your Grafana instance external URL
{{ define "__grafana_alert_silence_link" -}}
{{ .ExternalURL }}/alerting/silence/new?
{{- range .CommonLabels.SortedPairs -}}
matcher={{- .Name }}%3D{{- .Value | urlquery -}}&
{{- end -}}
{{- end }}
Addition for Grafana Alerting integration with AlertManager:
ℹ️ You have to setalertmanger.externalUrlto your Grafana instance external URL{{ define "__grafana_alert_silence_link" -}} {{ .ExternalURL }}/alerting/silence/new? {{- range .CommonLabels.SortedPairs -}} matcher={{- .Name }}%3D{{- .Value | urlquery -}}& {{- end -}} {{- end }}
Thanks @aorfanos
Have you also found a way to get the query button working towards Grafana? Alternatively, an external Prometheus?
Hi, I can't get silence button visible on TargetMissing alert. How can I fix it?