Prom-operator.md

prometheus-operator on pks

Prom-Operator Reference

https://github.com/helm/charts/tree/master/stable/prometheus-operator

Pre-reqs

*K8s Cluster (tested with PKS 1.4) *Helm client/server version =>2.13.1

Install the Operator

Get ETCD certs from BOSH

bosh scp -d service-instance_<blah> master/0:/var/vcap/jobs/etcd/config/etcd-ca.crt .
bosh scp -d service-instance_<blah> master/0:/var/vcap/jobs/etcd/config/etcdctl.* .

Create a monitoring namespace for the operator to live in

kubectl create ns monitoring

Set your context to that namespace

kubens monitoring

OR

kubectl config set-context $(kubectl config current-context) --namespace=monitoring

Create ETCD certs secret in K8s

kubectl create secret generic etcd-client --from-file=etcd-ca.crt --from-file=etcdctl.crt --from-file=etcdctl.key

Install the Operator with helm install

helm install --name prom-operator stable/prometheus-operator \
--set grafana.ingress.enabled=true \
--set grafana.ingress.hosts[0]=grafana.ing.vballin.com \
--set grafana.ingress.path="/*" \
--set prometheus.service.type=LoadBalancer \
--set prometheus.prometheusSpec.secrets[0]=etcd-client \
--set alertmanager.service.type=LoadBalancer \
--set grafana.adminPassword="VMware1\!" \
--set kubelet.serviceMonitor.https=true \
--set kubeEtcd.endpoints[0]=10.16.1.2 \
--set kubeEtcd.serviceMonitor.insecureSkipVerify=true \
--set kubeEtcd.serviceMonitor.scheme=https \
--set kubeEtcd.serviceMonitor.caFile="/etc/prometheus/secrets/etcd-client/etcd-ca.crt" \
--set kubeEtcd.serviceMonitor.certFile="/etc/prometheus/secrets/etcd-client/etcdctl.crt" \
--set kubeEtcd.serviceMonitor.keyFile="/etc/prometheus/secrets/etcd-client/etcdctl.key"

OR

override.yaml

grafana:
  ingress:
    enabled: true
    hosts:
      - grafana.ing.vballin.com
    path: "/*"
  adminPassword: "VMware1!"
prometheus:
  service:
    type: LoadBalancer
  prometheusSpec:
    secrets:
      - etcd-client
alertmanager:
  service:
    type: LoadBalancer
kubelet:
  serviceMonitor:
    https: true
kubeEtcd:
  endpoints:
    - 10.16.1.2
  serviceMonitor:
    insecureSkipVerify: true
    scheme: https
    caFile: "/etc/prometheus/secrets/etcd-client/etcd-ca.crt"
    certFile: "/etc/prometheus/secrets/etcd-client/etcdctl.crt"
    keyFile: "/etc/prometheus/secrets/etcd-client/etcdctl.key"

helm install -f override.yaml --name prom-operator stable/prometheus-operator

Delete the Operator and CRDs

helm delete prom-operator --purge

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com