rbrto/prometheus-grafana.md

## prometheus-grafana.md

      
    Raw
  

              prometheus-grafana.md
            
          
    Monitoring Kubernetes with Prometheus and Grafana via Helm


What Problem are we solving

Monitoring on Kubernetes.
tl;dr

tl;dr - Helm is a Kubernetes Package Manager.
tl;dr - Prometheus is the time series database.
tl;dr - Grafana provides a web dashboarding system.
Agenda


Install Helm
Install Prometheus
Install Grafana
Grafana Dashboards

Documentation


Helm : https://helm.sh/
Prometheus : https://prometheus.io/
Grafana : https://grafana.com/
Grafana Dashboards : https://grafana.com/dashboards

Requirements


Install a cluster using the Deploy a Kubernetes cluster and Microservices Application gist.
Complete the Rook on Digital Ocean to enable Block Storage.

Install Helm

Install Helm on ubuntu-s-2vcpu-4gb-sgp1-01 or kubeadm-001
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
helm init
helm search redis
root@kubeadm-001:~/deploy-to-kubernetes# helm search redis
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
stable/prometheus-redis-exporter        0.3.0           0.16.0          Prometheus exporter for Redis metrics
stable/redis                            3.7.3           4.0.10          Open source, advanced key-value store. It is of...
stable/redis-ha                         2.2.1           4.0.8-r0        Highly available Redis cluster with multiple se...
stable/sensu                            0.2.3           0.28            Sensu monitoring framework backed by the Redis ...

Install Prometheus


Enable StorageClass for Persistent Storage

vi prometheus-values.yml
alertmanager:
  enabled: true
  persistentVolume:
    storageClass: rook-ceph-block

server:
  persistentVolume:
    storageClass: rook-ceph-block
    size: 20Gi
Install Prometheus Helm Package

On ubuntu-s-2vcpu-4gb-sgp1-01 execute
helm install stable/prometheus --name prometheus --namespace monitoring -f prometheus-values.yml
Miscellaneous Commands

Inspect Prometheus Package
On ubuntu-s-2vcpu-4gb-sgp1-01: helm inspect stable/prometheus
Check Status of Install: watch -n1 kubectl get all -n monitoring
Sample Output:
Every 1.0s: kubectl get all -n monitoring                                                                     Fri Aug  3 00:20:37 2018

NAME                                                 READY     STATUS    RESTARTS   AGE
pod/grafana-657c4dbcc5-hzdsz                         1/1       Running   0          28m
pod/prometheus-alertmanager-78d596778d-r4nvd         2/2       Running   0          48m
pod/prometheus-kube-state-metrics-8668948654-dggml   1/1       Running   0          48m
pod/prometheus-pushgateway-58bbb659-xvqsv            1/1       Running   0          48m
pod/prometheus-server-796945cbb9-cmb2m               2/2       Running   0          48m

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/grafana                         ClusterIP   10.110.41.112    <none>        80/TCP     28m
service/prometheus-alertmanager         ClusterIP   10.111.207.202   <none>        80/TCP     48m
service/prometheus-kube-state-metrics   ClusterIP   None             <none>        80/TCP     48m
service/prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP   48m
service/prometheus-pushgateway          ClusterIP   10.104.145.45    <none>        9091/TCP   48m
service/prometheus-server               ClusterIP   10.100.72.77     <none>        80/TCP     48m

NAME                                      DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   0         0         0         0            0           <none>          48m

NAME                                            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana                         1         1         1            1           28m
deployment.apps/prometheus-alertmanager         1         1         1            1           48m
deployment.apps/prometheus-kube-state-metrics   1         1         1            1           48m
deployment.apps/prometheus-pushgateway          1         1         1            1           48m
deployment.apps/prometheus-server               1         1         1            1           48m

NAME                                                       DESIRED   CURRENT   READY     AGE
replicaset.apps/grafana-657c4dbcc5                         1         1         1         28m
replicaset.apps/prometheus-alertmanager-78d596778d         1         1         1         48m
replicaset.apps/prometheus-kube-state-metrics-8668948654   1         1         1         48m
replicaset.apps/prometheus-pushgateway-58bbb659            1         1         1         48m
replicaset.apps/prometheus-server-796945cbb9               1         1         1         48m

Delete installation (if required on error): helm del --purge prometheus
Notes

The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local

Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090

The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local

Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9093

The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.monitoring.svc.cluster.local

Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

Install Grafana

Install Grafana (create values.yml file first see below)
Disable requirement for Persistent Storage
On ubuntu-s-2vcpu-4gb-sgp1-01 execute
vi grafana-values.yml
persistence:
  enabled: false

datasources:
 datasources.yaml:
   apiVersion: 1
   datasources:
   - name: Prometheus
     type: prometheus
     url: http://prometheus-server
     access: proxy
     isDefault: true
Install Grafana Helm Package

helm install --name grafana stable/grafana -f grafana-values.yml --namespace monitoring
Miscellaneous Commands

Inspect Grafana Package: helm inspect stable/grafana
Check Status of Install: watch -n1 kubectl get all -n monitoring
Delete installation (if required on error): helm del --purge grafana
Notes

1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.monitoring.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:

     export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=grafana,component=" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace monitoring port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin

To access the Grafana Dashboard from the Digital Ocean Cluster from your local Linux instance:
mkdir .kube
cd .kube 

scp -i /home/jamesb/do/id_rsa-do root@<master IP>:/etc/kubernetes/admin.conf .

mv admin.conf config

kubectl get nodes

Step 3 in the Notes section is broken, you have two options on your local linux instance:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace monitoring port-forward $POD_NAME 3000

OR
kubectl port-forward --namespace=monitoring $(kubectl get pods --namespace=monitoring --selector=app=grafana --output=jsonpath='{.items[*].metadata.name}') 3000:3000

Access the Grafana UI: http://localhost:3000
Login Credentials

User: admin
Password: kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Monitoring Rook

Rook can be monitored with Prometheus and Grafana.
Grafana Dashboards

Download these charts and import them.

Ceph - Cluster
Ceph - OSD
Ceph - Pools

How do I import these into the dashboard?
Ceph - Cluster


Ceph - OSD


Ceph - Pools


Common Helm Commands

List Installed Packages: helm ls
Delete Packages: helm del --purge <name>
Search for Packages: helm search
End of Section