Using Prometheus and Grafana to Monitor Kubernetes Clusters and NGINX Metrics

prometheus-grafana-monitoring.md

Using Prometheus and Grafana to Monitor Kubernetes Clusters and NGINX Metrics

You can create your own Grafana dashboard from scratch, but this guide will show you how to import an already existing Grafana dashboard that contains most of the Kubernetes and NGINX metrics you would want to monitor.

Prerequisites

• Kubernetes cluster
• Helm 3

Note: If you need to migrate from Helm 2 to Helm 3, I recommend using this plugin.

Install NGINX and Enable Metrics

1. Create a Helm values file that you will use to enable NGINX metrics and Prometheus scraping

   The config portion of the yaml file is necessary if you want to be able to use http rather than exclusively https.

controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "10254"
  config:
    hsts: "false"
    redirect-ssl: "false"

2. Install NGINX using your values file

helm install nginx stable/nginx-ingress -f nginx-helm-values.yaml

3. Deploy the ingress(es) needed for your application.

Install Prometheus and Run It Locally

1. Install Prometheus using Helm

Note: Per this GitHub issue you need to use Prometheus version 9.0.0 otherwise you will get an error when you install the Helm chart.

helm install prometheus stable/prometheus --version 9.0.0

2. After installing Prometheus, the console will print out instructions for how to start your Prometheus server locally.

3. Follow the instructions in your console to start your Prometheus Server.

Install Grafana and Run It Locally

1. Install Grafana using Helm

helm install grafana stable/grafana

2. After installing Grafana, the console will print out instructions for how to start your Grafana server and how to get the password to login

3. Follow the instructions in your console to start your Grafana server.

Import a Grafana Dashboard

1. Add a data source

   a. Once you login you will be taken to the Grafana homepage. Click Create your first data source.

   b. This will take you to a page with a list of data sources. Hover over Prometheus under the Time series databases section and click Select.

   c. Under HTTP type in the DNS name for your Prometheus Server into the URL textbox. The DNS name for you Prometheus server will be:

   http://<name of prometheus server service>.<namespace>.svc.cluster.local

   If you've added the correct URL, you should see a green pop-up that says Data source is working.

   Note: If you leave out http:// or try to use http://localhost:9090, you will see a red HTTP Error Bad Gateway pop-up.

2. Add a Kubernetes Cluster Grafana dashboard

   a. Hover over the plus sign in the panel on the left hand side and click Import.

   b. In the Grafana.com Dashboard text box enter 7249 and then click Load at the bottom of the page. This will import this Grafana dashboard and take you to a new page titled Import.

   c. Under the Options section click the Select a Prometheus data source and select the data source, which should only have one option.

   Now you should see your dashboard!

Add a Requests Per Second Panel to Your Dashboard

1. Click the Add Panel button at the top left of the page. The button is a bar graph with a plus sign.

2. When the new panel appears click Choose Visualization.

3. Choose whichever visualization you would like and edit it to your liking.

4. On the left hand side panel click the Queries button, which is three disks stacked on top of each other.

5. Select Prometheus from the Query dropdown if it's not already selected.

6. In the query field enter:

sum(rate(nginx_ingress_controller_nginx_process_requests_total[2m])) by (instance)

This query calculates the total requests per second over the previous 2 minutes. The nginx_ingress_controller_nginx_process_requests_total is one of the NGINX metrics scraped by Prometheus.

ngnix ingress controller is another good dashboard to get started with as well