Skip to content

Instantly share code, notes, and snippets.

@myechuri

myechuri/blog.md Secret

Created November 7, 2019 12:09
Show Gist options
  • Save myechuri/1d5a8810b658d01bd41b979faa261f59 to your computer and use it in GitHub Desktop.
Save myechuri/1d5a8810b658d01bd41b979faa261f59 to your computer and use it in GitHub Desktop.

Monitoring Nodeless Kubernetes with Prometheus, Grafana

Step 1: Create 1-worker Nodeless Kubernetes cluster

Follow instructions in this repo to create a {1 master, 1 worker} Nodeless Kubernetes cluster.

Log on to Kubernetes master, verify cluster is up.

https://gist.github.com/7c1558bc3bbc259e49795ddf12b61ae3

Step 2: Deploy {Prometheus, kube-state-metrics, Grafana} stack

https://gist.github.com/7294922eba6687c717a826f3c095adc5

Create a 250GiB EBS volume in the same Availability Zone as your kubernetes cluster.

alt text

Insert volume-id of EBS volume in the manifest. https://gist.github.com/4ea760a4ef1f09a1309ed9ef76f3ef0f

Create {Prometheus, kube-state-metrics, Grafana} stack. https://gist.github.com/32d992f5263fa7e1f26c41b69ed152b5

Wait until all components of monitoring namespace are up and running. https://gist.github.com/a6e816305df7e3ac7184a077b80dd1ab

Our cluster now has following components.

alt text

Note: Stateful application Prometheus is deployed on the worker node, and stateless applications {Grafana, kube-state-metrics} are deployed via Nodeless fashion. Once persistent state support is available in Nodeless k8s, Prometheus will be deployed in Nodeless way as well.

Step 3: Create Grafana dashboard for Nodeless Kubernetes

Get external loadbalancer address for Grafana service and access Grafana through a web browser.

https://gist.github.com/1c10df6e646efa5540ca22f86a8904cb

Create Datasource for Prometheus called DS_Prometheus with http url set to http://prometheus-service:8080 since Grafana and Prometheus are running in the same k8s cluster.

alt text

Import Grafana dashboard 11124.

alt text

There are 3 Compute Cells in our cluster running {Grafana, kube-state-metrics} and a system pod.

Step 4: Deploy Nginx, monitor using Prometheus

Deploy Nginx deployment with 3 replicas.

https://gist.github.com/df2ff90b991c40e207250681874a939f

Our cluster now has the following components.

alt text

Grafana dashboard will reflect metrics from the 3 Nginx replicas running in just-in-time provisioned compute cells bumping up our cell count from 3 to 6.

alt text

Teardown

Follow teardown instructions from kubeadm repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment