openshift-user-workload-monitoring.md

You can easily enable monitoring for your projects in OpenShift if your workload exposes a metrics endpoint that can be scraped by prometheus. For example, operator you develop using operator-sdk adds an endpoint with lots of useful metrics by deault. You can query them in OpenShift web console if you enable monitoring for user-defined projects.

To enable, edit cluster-monitoring-config ConfigMap in openshift-monitoring namespace and add the following under data

data:
  config.yaml: |
    enableUserWorkload: true

Above change will start following pods in openshift-user-workload-monitoring namespace

openshift-user-workload-monitoring get po

NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-59dc7bd487-7rlds   2/2     Running   0          34h
prometheus-user-workload-0             5/5     Running   0          39h
prometheus-user-workload-1             5/5     Running   0          39h
thanos-ruler-user-workload-0           3/3     Running   0          39h
thanos-ruler-user-workload-1           3/3     Running   0          39h

To monitor your workload, you need either a ServiceMonitor or PodMonitor resource.

For example, if you want to monitor your operator:

• Create a Service to expose monitoring endpoint

kind: Service
apiVersion: v1
metadata:
  name: operator-metrics
  namespace: <ns>
  labels:
    app: operator-metrics
spec:
  ports:
    - name: metrics
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    control-plane: controller-manager

• Create a ServiceMonitor for prometheus to scrape your metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: operator-metrics-monitor
  namespace: <ns>
spec:
  endpoints:
    - port: metrics
      path: /metrics
  selector:
    matchLabels:
      app: operator-metrics

• Verify metrics target is added to prometheus

oc -n openshift-user-workload-monitoring port-forward pod/prometheus-user-workload-0 9090

Open http://localhost:9090/targets in browser. This might take few seconds to appear.

Once you see your target, you can query for metrics in OpenShift web console at Monitoring -> Metrics

Example query for operator metrics:

rest_client_requests_total{namespace='<ns>', service='operator-metrics'}

As you cannot edit default dashboards in Grafana, you will need to deploy an instanace of Grafana to add your metrics to Grafana dashboard.

See also:

https://docs.openshift.com/container-platform/4.8/monitoring/enabling-monitoring-for-user-defined-projects.html

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.9/html/monitoring/troubleshooting-monitoring-issues