Skip to content

Instantly share code, notes, and snippets.

@Himura2la

Himura2la/Grafana Alert Template.md

Last active May 16, 2024 15:46
Show Gist options
  • Save Himura2la/342603218ba2ee871f7b0e35eff9eaae to your computer and use it in GitHub Desktop.
Save Himura2la/342603218ba2ee871f7b0e35eff9eaae to your computer and use it in GitHub Desktop.
Download ZIP
How to use Grafana Alerts with the Telegram
Raw
Grafana Alert Template.md

  • Template name: telegram.message

  • Content:

    {{ define "alert_list" }}{{ range . }}{{ .Labels.alertname }}
{{ range .Annotations.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
{{ end }}| {{ if gt (len .GeneratorURL) 0 }}<a href="{{ .GeneratorURL }}">source</a> | {{ end }}{{ if gt (len .SilenceURL) 0 }}<a href="{{ .SilenceURL }}">silence</a> | {{ end }}{{ if gt (len .DashboardURL) 0 }}<a href="{{ .DashboardURL }}">dashboard</a> | {{ end }}{{ if gt (len .PanelURL) 0 }}<a href="{{ .PanelURL }}">panel</a> |{{ end }}
—
{{ end }}{{ end }}
{{ define "telegram.message" }}
{{ if gt (len .Alerts.Firing) 0 }}<strong>FIRING!!!</strong>
—
{{ template "alert_list" .Alerts.Firing }}
{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}<strong>Resolved</strong>
—
{{ template "alert_list" .Alerts.Resolved }}
{{ end }}{{ end }}

  • Contact point | Optional Telegram settings | Message: {{ template "telegram.message" . }}

  • Add verbose annotations to your alerts. Example: The <code>{{ $labels.host }}</code> host is low on free space! Used: {{ $values.display.Value }}% (where display is one of the queries).

  • I usually add the following queries: data (the datasource query), metric (the reduce expression), condition, and display, which is round($metric). Multi-dimensional rules documentation.

Also, check out this fork by @gelldur, this templeate is much more cmprehensive.

@aruaam
Copy link

aruaam commented Jul 14, 2022

Thank you for the template. It has saved me hours of pain. Example from Grafana doc does not work with telegram for some reason.

@Himura2la
Copy link
Author

It was a real pain to figure out all peculiarities about grafana templates and compile this template, I'm glad that this work is helpful!

@Himura2la
Copy link
Author

@aruaam, I improved it for readability and cleanness, you can test the new version.

@0xtakamaka
Copy link

Thanks! How can I add metrics with labels and values (example below with raw telegram message from grafana) to be present in your template?

Value: [ var='C0' metric='SRV1' labels={host_name=SRV1} value=19.661864406779653 ]
Labels:

  • alertname = recv_lag lag
  • rule_uid = _CjmgAR4z
    Annotations:....

@Himura2la
Copy link
Author

@nejcb1, Add {{ $labels.host_name }} and {{ $values.C0.Value }} to Summary or Description

@0xtakamaka
Copy link

image

Like this?

@Himura2la
Copy link
Author

Himura2la commented Aug 1, 2022 •
edited
Loading

Looks good. You can test it by expanding any alert instance once the rule has checked

image

@0xtakamaka
Copy link

Thanks! Is there any tag, to insert also images, like in old grafana alarming?

@Himura2la
Copy link
Author

I don't know, never used the old one before.. If you find it, please let me know.

@0xtakamaka
Copy link

https://grafana.com/docs/grafana/latest/alerting/images-in-notifications/ - I am using telegram so looks like, there is no option currently for it.

Regarding values from "Expression". Is it possible to loop over values, with dynamic metric's and variable? Eg. B0, B1, as in below example?

Value: [ var='B0' metric='test1' labels={} value=0.8555555555555555 ], [ var='B1' metric='test2' labels={} value=0.8740740740740741 ]
Labels:

  • alertname = XX
  • rule_uid = aa

@Himura2la
Copy link
Author

Himura2la commented Aug 2, 2022 •
edited
Loading

I guess it is possible. My first version of this template included the following block to iterate over annotations and labels:

{{ range .Annotations.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
{{ end }}
{{ range .Labels.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
{{ end }}

You can try to do something like this for values. My guess is:

{{ range .Values }}<strong>{{ .Var }}</strong>: {{ .Value }}
{{ end }}

But it's not tested.

Maybe this would help https://grafana.com/docs/grafana/latest/alerting/contact-points/message-templating/template-data/

UPD: I can't find any proof that the data key .Values exist.... Looks like the raw .ValueString is the only option, but maybe it's just not documented

@0xtakamaka
Copy link

Thanks, .Values it did not work for me. Solved just with putting {{ value }} in summary field for every alarm i trigger.

@tobiasmuehl
Copy link

Has anyone figured out how to use emoji in Telegram alerts? Like 🚨 and ✅ for example

@Himura2la
Copy link
Author

Have you tried to paste these unicode symbols into a template?

@tobiasmuehl
Copy link

Just tried, it works. Easy peasy

@aruaam
Copy link

aruaam commented Sep 1, 2022

@aruaam, I improved it for readability and cleanness, you can test the new version.

Hi, thank you for concern. Didn't have time to check the new version, unfortunately. Previous version is good enough for our needs.

@otomay
Copy link

otomay commented Dec 5, 2022

Hello! Can you help me please?

Your template worked very well, but when an alert returns to the normal state, the firing of "resolved" gives an error due to the variable that I put in the description of my alert... for some reason it is empty when the alert is normal.

image

I put {{ $values.B0.Value }}, which is the only option that actually returned values when it is in "Firing" state... all other attempts, like A0, A1, B1... does not return values... and query B is conditional, right? So... I don't know what to do :c

@Himura2la
Copy link
Author

@otomay, If you are not confident with the meaning of these A/B/C stuff, there's an article you should read carefully https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule/
The meaning of A0, A1, B0, B1, etc. is unique to your alert configuration, you should clearly understand what each of them mean, because Grafana does not help with this at all. As I wrote in the original gist, I tend to give my queries meaningful names: data, metric, and condition, this spares me from guessing what does all these A/B/C mean. Your case is specific for your configuration and not related to the template itself, so I'm afraid this is not the right place to seek help, sorry.

@otomay
Copy link

otomay commented Dec 5, 2022 •
edited
Loading

@Himura2la Sorry and thanks for the recommendation... the 0 and 1 flags really don't matter for what I'm doing, now I realize that... I adjusted the queries according to what you put in the beginning, and the display returns the result correctly:

image

But still, the value is not displayed when the state is normal :/ Where should I ask for help?

image

I'm using the classic condition, btw... is this the problem?

@otomay
Copy link

otomay commented Dec 5, 2022

I'm using the classic condition, btw... is this the problem?

Yes, that's it. I'm lame, sorry to bother you and thanks for your patience KEKW
image

@Himura2la
Copy link
Author

@otomay yea, Grafana is quite picky about the flow described in here https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule/
And the flow itself is far not intuitive, you never figure out these details about three expressions without docs. This is a terrible design, IMHO, I hope they fix this one day...

@pigate-eu
Copy link

How to hide "summary" and "description" texts on Telegram message? Can we make a simple alert code with only the Name of Rule and a small description such as:

High Temperature
The temperature is under 30 oC

I can't figure out what to change I'm confused!

@olekgo
Copy link

olekgo commented Jan 13, 2023

image

Like this?

Noticed you are using Alert ID annotation - what does it do? Cannot find any documentation on it

@Konano
Copy link

Konano commented Mar 15, 2023

Helps a lot!
When I was learning how to write message templates, it was much better to read this template than to read the documentation directly.

@faustinoaq
Copy link

Thanks you I managed to find the soruce code and create my own custom alerts ✨

{{/* Plantilla Telegram */}}
{{ define "alertas" -}}
{{ range . }}
{{ if len .Values -}}
Values:
{{- range $refID, $value := .Values -}}
    - {{ $refID }} = {{ $value }}
{{ end -}}
{{ end -}}
{{ if len .Labels.SortedPairs }}
Labels:
{{ range .Labels.SortedPairs -}}
    - {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
{{ if len .Annotations.SortedPairs }}
Annotations:
{{ range .Annotations.SortedPairs -}}
    - {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
{{ end -}} {{/* range */}}
{{ end -}} {{/* define alertas */}}

{{ define "plantilla.message" }}
{{ if gt (len .Alerts.Firing) 0 }}
🚨 FIRING!!!
{{- template "alertas" .Alerts.Firing -}}
{{ end -}}
{{ if gt (len .Alerts.Resolved) 0 }}
✅ RESOLVED!
{{- template "alertas" .Alerts.Resolved -}}
{{ end -}}
<a href="192.168.0.11:3000">Dashboard</a>
{{ end }}

image

@gelldur
Copy link

gelldur commented Mar 30, 2023

Thanks for sharing!

Checkout my fork for Grafana9 update ;)
https://gist.github.com/gelldur/94b57b2fa276fe9de180378bf6855877

image

@jdnielss
Copy link

jdnielss commented Jul 4, 2023

@gelldur can you make a tutorial to how to use telegram notification template?

@elhananjair
Copy link

Hello @Himura2la
Is it possible to include the server in value parameter of solved? The empty value will make ambuigty to identify which system have been solved.
Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

@mathematicw
Copy link

Hey guys, hello @Himura2la, thank you so much. Please help a little bit. How to really use this template? How to add here the value i want to monitor? from my expression

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment