grafana.md

Grafana Managed Prometheus and Kubecost

Grafana Cloud Prometheus is a fully managed Prometheus service, operated by Grafana Labs. The service allows you to query metrics from multiple Prometheus instances in a single location and provides long term storage for historical analysis and capacity planning.

Customers can use Kubecost to track cloud spend leveraging their Grafana Cloud Prometheus instance by using a custom prometheus integration.

Prerequisites

• You have access to a running Kubernetes cluster
• You have created a Grafana Cloud account
• You have permissions to create Grafana Cloud API keys

Step 1: Install the Grafana Agent in your cluster

We'll install the Agent in the kubecost namespace. Create that first.

Log into your Grafana Cloud account and follow the Grafana Agent for Kubernetes installation instructions.

Step 2: Configure Kubecost scraping configuration for the Grafana Agent

create a file called extra_scrape_configs.yaml with the following contents. Then kubectl apply to the namespace grafana-agent is in.

kind: ConfigMap
metadata:
  name: grafana-agent
apiVersion: v1
data:
  agent.yaml: |
    server:
      http_listen_port: 12345
    metrics:
      wal_directory: /tmp/grafana-agent-wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: cloud
      configs:
      - name: integrations
        remote_write:
        - url: <grafana_prometheus_remoteWrite_url>
          basic_auth:
            username: # from https://grafana.com/orgs/kcgrafanatest/hosted-metrics/291892
            password: # from; create an API key https://grafana.com/orgs/kcgrafanatest/hosted-metrics/291892
        scrape_configs:
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/kubernetes/cadvisor
          kubernetes_sd_configs:
              - role: node
          metric_relabel_configs:
              - source_labels: [__name__]
@

Restart Grafana Agent:

$ kubectl rollout restart deployment/grafana-agent -n kubecost

Step 3: Configure Kubecost recording rules for the Grafana Agent using cortextool

Download and install cortextool.

Create a file called kubecost_rules.yaml with these contents:

# kubecost_rules.yml
namespace: "kubecost"
groups:
  - name: CPU
    rules:
      - expr: sum(rate(container_cpu_usage_seconds_total{container_name!=""}[5m]))
        record: cluster:cpu_usage:rate5m
      - expr: rate(container_cpu_usage_seconds_total{container_name!=""}[5m])
        record: cluster:cpu_usage_nosum:rate5m
      - expr: avg(irate(container_cpu_usage_seconds_total{container_name!="POD", container_name!=""}[5m])) by (container_name,pod_name,namespace)
        record: kubecost_container_cpu_usage_irate
      - expr: sum(container_memory_working_set_bytes{container_name!="POD",container_name!=""}) by (container_name,pod_name,namespace)
        record: kubecost_container_memory_working_set_bytes
      - expr: sum(container_memory_working_set_bytes{container_name!="POD",container_name!=""})
        record: kubecost_cluster_memory_working_set_bytes
  - name: Savings
    rules:
      - expr: sum(avg(kube_pod_owner{owner_kind!="DaemonSet"}) by (pod) * sum(container_cpu_allocation) by (pod))
        record: kubecost_savings_cpu_allocation
        labels:
          daemonset: "false"
      - expr: sum(avg(kube_pod_owner{owner_kind="DaemonSet"}) by (pod) * sum(container_cpu_allocation) by (pod)) / sum(kube_node_info)
        record: kubecost_savings_cpu_allocation
        labels:
          daemonset: "true"
      - expr: sum(avg(kube_pod_owner{owner_kind!="DaemonSet"}) by (pod) * sum(container_memory_allocation_bytes) by (pod))
        record: kubecost_savings_memory_allocation_bytes
        labels:
          daemonset: "false"
      - expr: sum(avg(kube_pod_owner{owner_kind="DaemonSet"}) by (pod) * sum(container_memory_allocation_bytes) by (pod)) / sum(kube_node_info)
        record: kubecost_savings_memory_allocation_bytes
        labels:
          daemonset: "true"

Load the rules with cortextool. Use your prometheus url from https://grafana.com/orgs/<your_org>/hosted-metrics/

cortextool rules load \
--address=https://prometheus-prod-10-prod-us-central-0.grafana.net \
--user=<grafana_cloud_userId> \
--id=<grafana_cloud_org> \
--key=<grafana_cloud_api>key>

Verify that the rules were loaded by printing them out

cortextool rules print \
--address=https://prometheus-prod-10-prod-us-central-0.grafana.net \
--user=<grafana_cloud_userId> \
--id=<grafana_cloud_org> \
--key=<grafana_cloud_api>key>

Restart Grafana Agent:

$ kubectl rollout restart deployment/grafana-agent -n kubecost

Step 4: Install Kubecost on the cluster

Install Kubecost as usual using helm3, grabbing your Kubecost Token from kubecost.com/install:

$ helm repo add kubecost https://kubecost.github.io/cost-analyzer/
$ helm install kubecost kubecost/cost-analyzer --namespace kubecost --set kubecostToken="aGlAZWxlbmFsYXBlLmNvbQ==xm343yadf98"

Step 5: Configure Kubecost to query metrics from Grafana Cloud Prometheus

We'll need to use basic auth to be able to query from the Grafana Managed Prometheus.

Grab your Grafana Agent API keys and create two files, USERNAME and PASSWORD respectively. Use them to generate a Kubernetes secret dbsecret in the kubecost namespace.

$ kubectl create secret generic dbsecret -n kubecost --from-file=USERNAME --from-file=PASSWORD

$ helm upgrade kubecost kubecost/cost-analyzer --namespace kubecost --set global.prometheus.fqdn=<grafana_prometheus_query_url> --set global.prometheus.enabled=false --set global.prometheus.queryServiceBasicAuthSecretName=dbsecret