Skip to content

Instantly share code, notes, and snippets.

@dmitry-mightydevops
Created November 7, 2023 21:12
Show Gist options
  • Save dmitry-mightydevops/9f4e134c6c55f2dec9d27a4e287e8e83 to your computer and use it in GitHub Desktop.
Save dmitry-mightydevops/9f4e134c6c55f2dec9d27a4e287e8e83 to your computer and use it in GitHub Desktop.
example loki recording rule and alert configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-additional-rules
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
labels:
loki_rule: 'true'
data:
additional-loki-rules.yaml: |
groups:
- name: additional_loki_rules
interval: 1m
rules:
- record: job:loki:kex:application:sum_rate
expr: sum(rate({app="kubernetes-event-exporter"} | json | kind="Application" [5m])) by (name)
labels:
cluster: loki
- name: additional_loki_alerts
rules:
- alert: TooManyEventsInTheApplication
expr: |
sum(rate({app="kubernetes-event-exporter"} | json | kind="Application" [5m])) by (name) > 0.05
for: 0s
labels:
severity: page
annotations:
summary: High request latency
@dmitry-mightydevops
Copy link
Author

dmitry-mightydevops commented Nov 10, 2023

If you define to populate the rules using loki.rules of the HELM chart

loki:
  enable: true
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::11111:role/project-prod-eks-loki-role
  rules: []

then your syntax would be:

loki:
  enable: true
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::11111:role/project-prod-eks-loki-role

  rules:
    - name: additional_loki_rules
      interval: 1m
      rules:
        - record: job:loki:kex:application:sum_rate
          expr: sum(rate({app="kubernetes-event-exporter"} | json | kind="Application" [5m])) by (name)
          labels:
            cluster: loki
    - name: additional_loki_alerts
      rules:
        - alert: TooManyEventsInTheApplication
          expr: |
            sum(rate({app="kubernetes-event-exporter"} | json | kind="Application" [5m])) by (name) > 0.05
          for: 0s
          labels:
              severity: page
          annotations:
              summary: High request latency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment