Prometheus up and running with Helm

prometheus-with-helm.md

Install Helm with a specific service account for tiller...

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
chmod 0700 get_helm.sh 
./get_helm.sh 
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'      
helm init --service-account tiller --upgrade

Install the prometheus operator with Helm...

helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
helm install coreos/prometheus-operator --name prometheus-operator
helm install coreos/kube-prometheus --name kube-prometheus --set global.rbacEnable=true
watch -n1 kubectl get pods --all-namespaces

Expose the prometheus service externally so you can open a browser to it...

kubectl expose svc kube-prometheus-prometheus --external-ip 192.168.1.183 --target-port=9090 --name externaldoug-kube-prometheus-prometheus

I then I had trouble with 403 Forbidden on the Kubelet endpoints when looking at the Prometheus targets, using a reference from the kubelet authorization docs, I was able to patch up a node like so.

Firstly, I made this change to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[root@kube-node-3 centos]# diff /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ~/orig.10-kubeadm.conf 
6c6
< Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --authentication-token-webhook"
---
> Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"

I then restarted the kubelet.

systemctl daemon-reload
systemctl restart kubelet

I could then make a query manually. To do so... I first pulled up the authorization token from the prometheus pod.

$ KUBE_TOKEN=$(kubectl exec -it prometheus-kube-prometheus-0 -c prometheus -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ curl -v -sSk -H "Authorization: Bearer $KUBE_TOKEN"  https://192.168.1.130:10250/metrics

I then refreshed the Prometheus dashboard, and saw that it got the token properly.

z-initial-attempt-rbac-change.md

Some of my failed exploration in trying to fix the issue...

Trying to edit up the role...

Originally I started poking at the rbac, because it's having trouble querying the Kubelet, at addresses like https://192.168.1.XXX:10250/metrics fails with 403's in the "targets" of Prometheus.

kubectl get rolebindings
kubectl describe rolebindings kube-prometheus-exporter-kube-state
kubectl describe role kube-prometheus-exporter-kube-state
curl https://192.168.1.183:10250/metrics
curl -k https://192.168.1.183:10250/metrics
kubectl get roles
kubectl get rolebinding 

Inspect the role...

[centos@kube-master ~]$ kubectl describe role kube-prometheus-exporter-kube-state
[centos@kube-master ~]$ kubectl get role kube-prometheus-exporter-kube-state -o yaml

Then we're going to try to replace the whole rules section with something like:

- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'

Which we'll do with an edit, like so:

[centos@kube-master ~]$ kubectl edit role kube-prometheus-exporter-kube-state -o yaml

Didn't do the trick. Next!