Monitor Calico Felix using the Prometheus Operator (Kubernetes)

monitor-calico-felix-using-prom-op-in-k8s.md

Monitor Calico Felix using Prometheus Operator

Make sure you have these settings enabled in felix, in this case using enviornment variables:

FELIX_PROMETHEUSMETRICSENABLED = True
FELIX_PROMETHEUSMETRICSPORT = "9091"
FELIX_PROMETHEUSGOMETRICSENABLED = True
FELIX_PROMETHEUSPROCESSMETRICSENABLED = True

Create a Service with clusterIP set to None, the service will be used to enumerate endpoints to scrape:

apiVersion: v1
kind: Service
metadata:
  name: calico-felix-metrics
  namespace: kube-system
  labels:
    monitoring: calico-felix
spec:
  clusterIP: None
  ports:
  - port: 9091
    protocol: TCP 
    name: metrics  
  selector:
    k8s-app: calico-node

Create a ServiceMonitor to start scrape the calico-node Pods:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: calico-felix-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      monitoring: calico-felix
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: metrics
    relabelings:
    - sourceLabels:
      - __meta_kubernetes_endpoint_node_name
      targetLabel: instance

Note the relabeling above, this changes the instance label to something more descriptive when querying for Calico Felix metrics.

Install the official Grafana dashboard: https://grafana.com/grafana/dashboards/12175

Links

https://docs.projectcalico.org/reference/felix/prometheus https://docs.projectcalico.org/archive/v3.18/reference/felix/configuration