Loki-Stack (Prometheus+Grafana+Loki+Promtail) Installation with Helm.md

Loki-Stack (Prometheus+Grafana+Loki+Promtail) Installation with Helm

Prerequisites

• Kubernetes 1.19+
• Helm 3.2.0+

helm Chart

• https://github.com/grafana/helm-charts/blob/main/charts/loki-stack
• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

helm update

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana

Install (Promtail + Loki)

kubectl create ns monitoring-system
helm upgrade --install loki --namespace=monitoring-system grafana/loki-stack

helm update

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

Install (Prometheus-Operator)

cat <<EOF > prometheus-value.yaml
--- 
prometheus:
  prometheusSpec: 
    scrapeInterval: 5s
    externalLabels: 
      cluster: "k8s-gpu-192-168-160-41"
    replicas: 1
    #additionalScrapeConfigs: |
    #  - job_name: "ceph-exporter"
    #    static_configs:
    #      - targets: ['rook-ceph-mgr.rook-ceph.svc.cluster.local:9283']
grafana:
  sidecar:
    datasources:
      enabled: true
      label: grafana_datasource
      searchNamespace: ALL
      defaultDatasourceEnabled: false
  additionalDataSources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-kube-prometheus-prometheus.monitoring-system:9090/
      basicAuth: false
      basicAuthUser:
      basicAuthPassword:
      isDefault: true
      editable: true
      jsonData:
        graphiteVersion: "1.1"
        tlsAuth: false
        tlsAuthWithCACert: false
    - name: Loki
      type: loki
      url: http://loki.monitoring-system:3100/
      basicAuth: false
      basicAuthUser:
      # secureJsonData:
      #   basicAuthPassword: test_password
      access: proxy
EOF

helm install prometheus prometheus-community/kube-prometheus-stack \
  --create-namespace \
  --namespace monitoring-system \
  --set grafana.adminPassword=password \
  -f prometheus-value.yaml

kubectl patch svc prometheus-grafana -n monitoring-system --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":32071}]'

How-to

instance_public_ip="$(curl ifconfig.me --silent)"
echo "https://$instance_public_ip:32071"
echo "ID: admin"
echo "PW: " $(kubectl get secret --namespace monitoring-system prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo)

Dashboard

• Loki Dashboard quick search | Grafana Labs

Screenshot

Etc

• ELK : Elasticsearch + Logstash + Kibana
• EFK : Elasticsearch + Fluentd + Kibana
• PLG : Promtail+ Loki + Grafana (+Prometheus)