How to use Grafana Alerts with the Telegram

Grafana Alert Template.md

• Template name: telegram.message

• Content:

  {{ define "alert_list" }}{{ range . }}{{ .Labels.alertname }}
  {{ range .Annotations.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
  {{ end }}| {{ if gt (len .GeneratorURL) 0 }}<a href="{{ .GeneratorURL }}">source</a> | {{ end }}{{ if gt (len .SilenceURL) 0 }}<a href="{{ .SilenceURL }}">silence</a> | {{ end }}{{ if gt (len .DashboardURL) 0 }}<a href="{{ .DashboardURL }}">dashboard</a> | {{ end }}{{ if gt (len .PanelURL) 0 }}<a href="{{ .PanelURL }}">panel</a> |{{ end }}
  —
  {{ end }}{{ end }}
  {{ define "telegram.message" }}
  {{ if gt (len .Alerts.Firing) 0 }}<strong>FIRING!!!</strong>
  —
  {{ template "alert_list" .Alerts.Firing }}
  {{ if gt (len .Alerts.Resolved) 0 }}
  {{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}<strong>Resolved</strong>
  —
  {{ template "alert_list" .Alerts.Resolved }}
  {{ end }}{{ end }}
  

• Contact point | Optional Telegram settings | Message: {{ template "telegram.message" . }}

• Add verbose annotations to your alerts. Example: The <code>{{ $labels.host }}</code> host is low on free space! Used: {{ $values.display.Value }}% (where display is one of the queries).

• I usually add the following queries: data (the datasource query), metric (the reduce expression), condition, and display, which is round($metric). Multi-dimensional rules documentation.

Also, check out this fork by @gelldur, this templeate is much more cmprehensive.

Looks good. You can test it by expanding any alert instance once the rule has checked

Thanks! Is there any tag, to insert also images, like in old grafana alarming?

I don't know, never used the old one before.. If you find it, please let me know.

https://grafana.com/docs/grafana/latest/alerting/images-in-notifications/ - I am using telegram so looks like, there is no option currently for it.

Regarding values from "Expression". Is it possible to loop over values, with dynamic metric's and variable? Eg. B0, B1, as in below example?

Value: [ var='B0' metric='test1' labels={} value=0.8555555555555555 ], [ var='B1' metric='test2' labels={} value=0.8740740740740741 ]
Labels:

• alertname = XX
• rule_uid = aa

I guess it is possible. My first version of this template included the following block to iterate over annotations and labels:

{{ range .Annotations.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
{{ end }}
{{ range .Labels.SortedPairs }}<strong>{{ .Name }}</strong>: {{ .Value }}
{{ end }}

You can try to do something like this for values. My guess is:

{{ range .Values }}<strong>{{ .Var }}</strong>: {{ .Value }}
{{ end }}

But it's not tested.

Maybe this would help https://grafana.com/docs/grafana/latest/alerting/contact-points/message-templating/template-data/

UPD: I can't find any proof that the data key .Values exist.... Looks like the raw .ValueString is the only option, but maybe it's just not documented

Thanks, .Values it did not work for me. Solved just with putting {{ value }} in summary field for every alarm i trigger.

Has anyone figured out how to use emoji in Telegram alerts? Like 🚨 and ✅ for example

Have you tried to paste these unicode symbols into a template?

Just tried, it works. Easy peasy

@aruaam, I improved it for readability and cleanness, you can test the new version.

Hi, thank you for concern. Didn't have time to check the new version, unfortunately. Previous version is good enough for our needs.

Hello! Can you help me please?

Your template worked very well, but when an alert returns to the normal state, the firing of "resolved" gives an error due to the variable that I put in the description of my alert... for some reason it is empty when the alert is normal.

I put {{ $values.B0.Value }}, which is the only option that actually returned values when it is in "Firing" state... all other attempts, like A0, A1, B1... does not return values... and query B is conditional, right? So... I don't know what to do :c

@otomay, If you are not confident with the meaning of these A/B/C stuff, there's an article you should read carefully https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule/
The meaning of A0, A1, B0, B1, etc. is unique to your alert configuration, you should clearly understand what each of them mean, because Grafana does not help with this at all. As I wrote in the original gist, I tend to give my queries meaningful names: data, metric, and condition, this spares me from guessing what does all these A/B/C mean. Your case is specific for your configuration and not related to the template itself, so I'm afraid this is not the right place to seek help, sorry.

@Himura2la Sorry and thanks for the recommendation... the 0 and 1 flags really don't matter for what I'm doing, now I realize that... I adjusted the queries according to what you put in the beginning, and the display returns the result correctly:

But still, the value is not displayed when the state is normal :/ Where should I ask for help?

I'm using the classic condition, btw... is this the problem?

I'm using the classic condition, btw... is this the problem?

Yes, that's it. I'm lame, sorry to bother you and thanks for your patience KEKW

@otomay yea, Grafana is quite picky about the flow described in here https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule/
And the flow itself is far not intuitive, you never figure out these details about three expressions without docs. This is a terrible design, IMHO, I hope they fix this one day...

How to hide "summary" and "description" texts on Telegram message? Can we make a simple alert code with only the Name of Rule and a small description such as:

High Temperature
The temperature is under 30 oC

I can't figure out what to change I'm confused!

Like this?

Noticed you are using Alert ID annotation - what does it do? Cannot find any documentation on it

Helps a lot!
When I was learning how to write message templates, it was much better to read this template than to read the documentation directly.

Thanks you I managed to find the soruce code and create my own custom alerts ✨

{{/* Plantilla Telegram */}}
{{ define "alertas" -}}
{{ range . }}
{{ if len .Values -}}
Values:
{{- range $refID, $value := .Values -}}
    - {{ $refID }} = {{ $value }}
{{ end -}}
{{ end -}}
{{ if len .Labels.SortedPairs }}
Labels:
{{ range .Labels.SortedPairs -}}
    - {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
{{ if len .Annotations.SortedPairs }}
Annotations:
{{ range .Annotations.SortedPairs -}}
    - {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
{{ end -}} {{/* range */}}
{{ end -}} {{/* define alertas */}}

{{ define "plantilla.message" }}
{{ if gt (len .Alerts.Firing) 0 }}
🚨 FIRING!!!
{{- template "alertas" .Alerts.Firing -}}
{{ end -}}
{{ if gt (len .Alerts.Resolved) 0 }}
✅ RESOLVED!
{{- template "alertas" .Alerts.Resolved -}}
{{ end -}}
<a href="192.168.0.11:3000">Dashboard</a>
{{ end }}

Thanks for sharing!

Checkout my fork for Grafana9 update ;)
https://gist.github.com/gelldur/94b57b2fa276fe9de180378bf6855877

@gelldur can you make a tutorial to how to use telegram notification template?

Hello @Himura2la
Is it possible to include the server in value parameter of solved? The empty value will make ambuigty to identify which system have been solved.
Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

Hey guys, hello @Himura2la, thank you so much. Please help a little bit. How to really use this template? How to add here the value i want to monitor? from my expression