prometheus-operator-helm.md

Prometheus-operator installation with Helm

• Prometheus, Grafana, kube-state-metrics, prometheus-node-exporter on Kubernetes

helm Chart

• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Prerequisites

• Kubernetes 1.19+
• Helm 3.2.0+
• A persistent storage resource and RW access to it
• Kubernetes StorageClass for dynamic provisioning

Document

• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#configuration

helm update

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

value-override

cat <<EOF > values-override.yaml
--- 
prometheus:
  prometheusSpec: 
    scrapeInterval: 5s
    externalLabels: 
      cluster: "crio-k8s-1.25"
    replicas: 1
grafana:
  adminPassword: prom-operator
EOF

Install

helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --create-namespace \
  --namespace monitoring \
  -f values-override.yaml
  

clusterIP to NodePort

kubectl patch svc prometheus-stack-grafana -n monitoring --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":32071}]'

kubectl patch svc prometheus-stack-kube-prom-prometheus -n monitoring --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":30097}]'

Connect

instance_public_ip="$(curl ifconfig.me --silent)"
echo "https://$instance_public_ip:32071"
echo "ID: admin"
echo "PW: " $(kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo)

(Option) Thanos Down 해결 방법

## kube-proxy
kubectl edit cm kube-proxy -n kube-system
metricsBindAddress: "" -> metricsBindAddress: 0.0.0.0:10249
kubectl rollout restart daemonset/kube-proxy -n kube-system



## etcd
/etc/kubernetes/manifests/etcd.yaml
127.0.0.1:2381 > 0.0.0.0:2381

## scheduler
/etc/kubernetes/manifests/kube-scheduler.yaml
bind-address=127.0.0.1 > bind-address=0.0.0.0


## controller-manager
/etc/kubernetes/manifests/kube-controller-manager.yaml
bind-address=127.0.0.1 > bind-address=0.0.0.0