prometheus-operator-helm.md

Prometheus-operator installation with Helm

• Prometheus, Grafana, kube-state-metrics, prometheus-node-exporter on Kubernetes

helm Chart

• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Prerequisites

• Kubernetes 1.19+
• Helm 3.2.0+
• A persistent storage resource and RW access to it
• Kubernetes StorageClass for dynamic provisioning

Document

• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#configuration

helm update

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

value-override

cat <<EOF > values-override.yaml
--- 
prometheus:
  prometheusSpec: 
    scrapeInterval: 5s
    externalLabels: 
      cluster: "crio-k8s-1.25"
    replicas: 1
grafana:
  adminPassword: prom-operator
EOF

Install

helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --create-namespace \
  --namespace monitoring \
  -f values-override.yaml
  

clusterIP to NodePort

kubectl patch svc prometheus-stack-grafana -n monitoring --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":32071}]'

kubectl patch svc prometheus-stack-kube-prom-prometheus -n monitoring --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":30097}]'

Connect

instance_public_ip="$(curl ifconfig.me --silent)"
echo "https://$instance_public_ip:32071"
echo "ID: admin"
echo "PW: " $(kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo)

ServiceMonitor

cat <<'EOF' | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph
  labels:
    team: rook
spec:
  namespaceSelector:
    matchNames:
      - rook-ceph
  selector:
    matchLabels:
      app: rook-ceph-mgr
      rook_cluster: rook-ceph
      ceph_daemon_id: a
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 5s
EOF

(Option) 멀티클러스터 시 thanos 수집

Prometheus 설치 후

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: prometheus-stack-kube-prom-thanos-discovery
  namespace: monitoring
spec:
  ports:
  - name: grpc
    port: 10901
    protocol: TCP
    nodePort: 32073
    targetPort: grpc
  - name: http
    port: 10902
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: prometheus
    prometheus: prometheus-stack-kube-prom-prometheus
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}
EOF

(Option) kubernetes v1.24 이상

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: crio-k8s
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: crio-k8s
type: kubernetes.io/service-account-token
EOF

kubectl get secrets -n kube-system
kubectl describe secrets/crio-k8s -n kube-system | grep token

(Option) Thanos Down 해결 방법

## kube-proxy
kubectl edit cm kube-proxy -n kube-system
metricsBindAddress: "" -> metricsBindAddress: 0.0.0.0:10249
kubectl rollout restart daemonset/kube-proxy -n kube-system



## etcd
/etc/kubernetes/manifests/etcd.yaml
127.0.0.1:2381 > 0.0.0.0:2381

## scheduler
/etc/kubernetes/manifests/kube-scheduler.yaml
bind-address=127.0.0.1 > bind-address=0.0.0.0


## controller-manager
/etc/kubernetes/manifests/kube-controller-manager.yaml
bind-address=127.0.0.1 > bind-address=0.0.0.0