Skip to content

Instantly share code, notes, and snippets.

@gelldur

gelldur/Grafana Alert Template.md

Forked from Himura2la/Grafana Alert Template.md
Last active May 3, 2024 12:04
Show Gist options
  • Star 23 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
  • Save gelldur/94b57b2fa276fe9de180378bf6855877 to your computer and use it in GitHub Desktop.
Save gelldur/94b57b2fa276fe9de180378bf6855877 to your computer and use it in GitHub Desktop.
Download ZIP
How to use Grafana Alerts with the Telegram
Raw
Grafana Alert Template.md
  • Template name: telegram.message
  • Content: 
    {{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
{{ define "__alerts_list" -}}
{{ range . }}
{{if ne (index .Labels "alertname") "" -}}
{{ if eq .Status "firing" }}πŸ”΄{{ else }}🟒{{ end }}
    {{- if ne (index .Labels "severity") "" -}}
        <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
<b>{{ index .Labels "alertname" }}</b> πŸ•™ {{ .StartsAt.Format "15:04:05    πŸ—“οΈ 2006-01-02" }}{{ end -}}
{{ if len .Annotations }}
<i>Annotations:</i>
    {{ range .Annotations.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
{{ if len .Labels -}}
<i>Labels:</i>
    {{ range .Labels.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
<i>Value:</i> <pre>{{ .ValueString }}</pre>
    {{- if gt (len .GeneratorURL) 0 }}<a href="{{ .GeneratorURL }}">source</a>  |  {{ end }}
    {{- if gt (len .SilenceURL) 0 }}<a href="{{ .SilenceURL }}">πŸ”• silence</a>  |  {{ end }}
    {{- if gt (len .DashboardURL) 0 }}πŸ“ <a href="{{ .DashboardURL }}">dashboard</a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }}<a href="{{ .PanelURL }}">panel</a> {{- end -}}
    <pre>--------</pre>
{{- end -}} {{- /* range */ -}}
{{- end -}} {{- /* define __alerts_list */ -}}

{{ define "__telegram.title" -}}
{{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
{{ if ne (index .CommonLabels "alertname") "" -}}
    [{{ index .CommonLabels "alertname" }}]
{{- end -}}
{{- end -}}{{- /* define __telegram */ -}}

{{ define "telegram.message" }}
    {{ if gt (len .Alerts.Firing) 0 }}
    🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
    {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
    {{ if gt (len .Alerts.Resolved) 0 }}
    βœ… <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
    {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}

<a href="{{ .ExternalURL }}">πŸ“² Grafana</a>
{{- end -}}

- Contact point | Optional Telegram settings | Message: `{{ template "telegram.message" . }}`
- Add verbose annotations to your alerts. Example: `The <code>{{ $labels.host }}</code> host is low on free space! Used: {{ $values.display.Value }}%` (where `display` is one of the queries).
- I usually add the following queries: `data` (the datasource query), `metric` (the reduce expression), `condition`, and `display`, which is `round($metric)`. [Multi-dimensional rules documentation](https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule).
@gelldur
Copy link
Author

gelldur commented Mar 30, 2023

Updated for Grafana 9 where now we have .ValueString instead of nice map of values (just tell my why :/)
This is how it looks:
image

@gelldur
Copy link
Author

gelldur commented Mar 30, 2023

Updated look:
image

Links:
https://grafana.com/docs/grafana/v9.0/dashboards/manage-dashboard-links/#dashboard-links
https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/

@Himura2la
Copy link

Himura2la commented Apr 5, 2023 •
edited

Thanks for your work, looks great (though, it still has some sides to polish), I added the link to my original gist. I hope this template will develop and become even better while you use it!

JFYI: .ValueString was present in all versions, I just found it totally useless and super noisy. I decided to pass only required values in summary and description. This makes alerts cleaner and easier to quickly recognize.

@tobiasmuehl
Copy link

tobiasmuehl commented Aug 12, 2023 •
edited

Awesome template. I edited it by removing the time and date, since Telegram shows message timestamps itself (accurate enough for my usecase)

I have a lot of alerts, sometimes so many firing that the alert message is longer than 4096 characters, which is the max length supported in Telegram. How can I work around this?

Update: Fixed this by grouping by alert name instead of grafana folder instead (Default notification policy)

Screenshot 2023-08-12 at 14 13 34

@draincoder
Copy link

Hi! Thanks for the template, can you send an example of an alert rule in yaml to work with this template?

@elhananjair
Copy link

Hello @gelldur
Is it possible to include the server in value parameter of solved? The empty value will make ambiguity to identify which system have been solved.
Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

@arturucraft
Copy link

Updated for Grafana 9 where now we have .ValueString instead of nice map of values (just tell my why :/) This is how it looks: image

thanks a lot for the developed template. this is very useful.
just a question, the "href"s, they are not shown as links in my telegram channel, even tho i have set the parse mode to HTML in the contact point telegram optional settings. i am getting just texts

source | πŸ”• silence | πŸ“ dashboard | panel

can you guys tell me where and what settings i need to apply to get as hyperlinks? i have two channels (webhook + telegram) and in my webhook it is properly received as hyperlinks, but not in telegram

thanks in advance

@gelldur
Copy link
Author

gelldur commented Apr 26, 2024

Hello @gelldur Is it possible to include the server in value parameter of solved? The empty value will make ambiguity to identify which system have been solved. Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

Not sure, I really lost a lot of time on this...

Updated for Grafana 9 where now we have .ValueString instead of nice map of values (just tell my why :/) This is how it looks: image

thanks a lot for the developed template. this is very useful. just a question, the "href"s, they are not shown as links in my telegram channel, even tho i have set the parse mode to HTML in the contact point telegram optional settings. i am getting just texts

source | πŸ”• silence | πŸ“ dashboard | panel

can you guys tell me where and what settings i need to apply to get as hyperlinks? i have two channels (webhook + telegram) and in my webhook it is properly received as hyperlinks, but not in telegram

thanks in advance

I don't see any configuration in my grafana instance for this. Maybe this is issue on telegram client side? Trusted/untrusted stuff?

@arturucraft
Copy link

thanks for quick turnaround. i guess the issue is in here:

{{- if gt (len .GeneratorURL) 0 }} <a href={{ .GeneratorURL }}> source </a>  |  {{ end }}
{{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> πŸ”• silence </a>  |  {{ end }}
{{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> πŸ“ dashboard</a>  |  {{ end }}
{{- if gt (len .PanelURL) 0 }} <a href="http://{{ .PanelURL }}"> panel </a> {{- end -}}

because when i put just {{ .GeneratorURL }} in the template the URL is properly printed, but when it is in the if and in a href it is not showing up at all, leading to print just "source" in the alert. is it properly working on your side? I am on grafana 10. i guess i need to play around to see how i will overcome this.

thanks again!

@gelldur
Copy link
Author

gelldur commented Apr 26, 2024

Tested on Grafana 9.X only, so maybe yes on Grafana 10 it may not work.

@arturucraft
Copy link

arturucraft commented Apr 26, 2024 •
edited

thanks again, i have also tried to downgrade to 9.5.18, and it is still doing the same. i guess i am missing a step :(

{{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
{{ define "__alerts_list" -}}
{{ range . }}
{{if ne (index .Labels "alertname") "" -}}
{{ if eq .Status "firing" }}πŸ”΄{{ else }}🟒{{ end }}
    {{- if ne (index .Labels "severity") "" -}}
        <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
<b>{{ index .Labels "alertname" }}</b> πŸ•™ {{ .StartsAt.Format "15:04:05    πŸ—“οΈ 2006-01-02" }}{{ end -}}
{{ if len .Annotations }}
<i>Annotations:</i>
    {{ range .Annotations.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
{{ if len .Labels -}}
<i>Labels:</i>
    {{ range .Labels.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
<i>Value:</i> <pre>{{ .ValueString }}</pre>
    {{- if gt (len .GeneratorURL) 0 }} <a href="{{ .GeneratorURL }}"> source </a>  |  {{ end }}
    {{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> πŸ”• silence </a>  |  {{ end }}
    {{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> πŸ“ dashboard</a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }} <a href="{{ .PanelURL }}"> panel </a> {{- end -}}
    <pre>--------</pre>
{{- end -}} {{- /* range */ -}}
{{- end -}} {{- /* define __alerts_list */ -}}

{{ define "__telegram.title" -}}
{{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
{{ if ne (index .CommonLabels "alertname") "" -}}
    [{{ index .CommonLabels "alertname" }}]
{{- end -}}
{{- end -}}{{- /* define __telegram */ -}}

{{ define "telegram.message" }}
    {{ if gt (len .Alerts.Firing) 0 }}
    🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
    {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
    {{ if gt (len .Alerts.Resolved) 0 }}
    βœ… <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
    {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
<a href="{{ .ExternalURL }}">πŸ“² Grafana</a>
{{- end -}}

in the message i have put:

{{ template "telegram.message" . }}

and in parse mode i have put HTML.

not sure what else i need to change, but i am getting plain text in telegram without hyperlinks:

 source   |    πŸ”• silence   |    πŸ“ dashboard  |    panel 

copy
--------

πŸ“² Grafana

when i switch the parse mode to markdown i am clearly getting hyperlinks but not html formatted:

silence </a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1"> πŸ“ dashboard</a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1&viewPanel=1"> panel </a><pre>--------</pre>

<a href="http://localhost:3000/">πŸ“² Grafana</a>

thanks anyways, i will try to find the root cause for this.

by the way, which version of grafana are you using exactly ? i can try to reproduce on that one

@gelldur
Copy link
Author

gelldur commented Apr 30, 2024

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

@arturucraft
Copy link

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

ok, thanks! i am struggling with this till now. i have opened a bug, to see why my html content is not normally rendering

@gelldur
Copy link
Author

gelldur commented May 3, 2024

@arturucraft just had a thought. Maybe your message is too long and thats why telegram not parsing it as full html? Try sending only single link message to confirm.

@arturucraft
Copy link

i think that's not the problem, because even i add another html code, it parses/renders it as expected, when i put {{ .DashboardURL }} it displaysthe variable correctly, if i put <a href="http://example.com">aaaa</a> it renders it properly, the problem is only in combination of a href and {{ .DashboardURL }} where it does not render <a href="{{ .DashboardURL }}"> πŸ“ dashboard</a>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment