Skip to content

Instantly share code, notes, and snippets.

Created Feb 1, 2022
What would you like to do?

Grafana Managed Prometheus and Kubecost

Grafana Cloud Prometheus is a fully managed Prometheus service, operated by Grafana Labs. The service allows you to query metrics from multiple Prometheus instances in a single location and provides long term storage for historical analysis and capacity planning.

Customers can use Kubecost to track cloud spend leveraging their Grafana Cloud Prometheus instance by using a custom prometheus integration.


  • You have access to a running Kubernetes cluster
  • You have created a Grafana Cloud account
  • You have permissions to create Grafana Cloud API keys

Step 1: Install the Grafana Agent in your cluster

We'll install the Agent in the kubecost namespace. Create that first.

Log into your Grafana Cloud account and follow the Grafana Agent for Kubernetes installation instructions.

Step 2: Configure Kubecost scraping configuration for the Grafana Agent

create a file called extra_scrape_configs.yaml with the following contents. Then kubectl apply to the namespace grafana-agent is in.

kind: ConfigMap
  name: grafana-agent
apiVersion: v1
  agent.yaml: |
      http_listen_port: 12345
      wal_directory: /tmp/grafana-agent-wal
        scrape_interval: 60s
          cluster: cloud
      - name: integrations
        - url: <grafana_prometheus_remoteWrite_url>
            username: # from
            password: # from; create an API key
        - bearer_token_file: /var/run/secrets/
          job_name: integrations/kubernetes/cadvisor
              - role: node
              - source_labels: [__name__]

Restart Grafana Agent:

$ kubectl rollout restart deployment/grafana-agent -n kubecost

Step 3: Configure Kubecost recording rules for the Grafana Agent using cortextool

Download and install cortextool.

Create a file called kubecost_rules.yaml with these contents:

# kubecost_rules.yml
namespace: "kubecost"
  - name: CPU
      - expr: sum(rate(container_cpu_usage_seconds_total{container_name!=""}[5m]))
        record: cluster:cpu_usage:rate5m
      - expr: rate(container_cpu_usage_seconds_total{container_name!=""}[5m])
        record: cluster:cpu_usage_nosum:rate5m
      - expr: avg(irate(container_cpu_usage_seconds_total{container_name!="POD", container_name!=""}[5m])) by (container_name,pod_name,namespace)
        record: kubecost_container_cpu_usage_irate
      - expr: sum(container_memory_working_set_bytes{container_name!="POD",container_name!=""}) by (container_name,pod_name,namespace)
        record: kubecost_container_memory_working_set_bytes
      - expr: sum(container_memory_working_set_bytes{container_name!="POD",container_name!=""})
        record: kubecost_cluster_memory_working_set_bytes
  - name: Savings
      - expr: sum(avg(kube_pod_owner{owner_kind!="DaemonSet"}) by (pod) * sum(container_cpu_allocation) by (pod))
        record: kubecost_savings_cpu_allocation
          daemonset: "false"
      - expr: sum(avg(kube_pod_owner{owner_kind="DaemonSet"}) by (pod) * sum(container_cpu_allocation) by (pod)) / sum(kube_node_info)
        record: kubecost_savings_cpu_allocation
          daemonset: "true"
      - expr: sum(avg(kube_pod_owner{owner_kind!="DaemonSet"}) by (pod) * sum(container_memory_allocation_bytes) by (pod))
        record: kubecost_savings_memory_allocation_bytes
          daemonset: "false"
      - expr: sum(avg(kube_pod_owner{owner_kind="DaemonSet"}) by (pod) * sum(container_memory_allocation_bytes) by (pod)) / sum(kube_node_info)
        record: kubecost_savings_memory_allocation_bytes
          daemonset: "true"

Load the rules with cortextool. Use your prometheus url from<your_org>/hosted-metrics/

cortextool rules load \
--address= \
--user=<grafana_cloud_userId> \
--id=<grafana_cloud_org> \

Verify that the rules were loaded by printing them out

cortextool rules print \
--address= \
--user=<grafana_cloud_userId> \
--id=<grafana_cloud_org> \

Restart Grafana Agent:

$ kubectl rollout restart deployment/grafana-agent -n kubecost

Step 4: Install Kubecost on the cluster

Install Kubecost as usual using helm3, grabbing your Kubecost Token from

$ helm repo add kubecost
$ helm install kubecost kubecost/cost-analyzer --namespace kubecost --set kubecostToken="aGlAZWxlbmFsYXBlLmNvbQ==xm343yadf98"

Step 5: Configure Kubecost to query metrics from Grafana Cloud Prometheus

We'll need to use basic auth to be able to query from the Grafana Managed Prometheus.

Grab your Grafana Agent API keys and create two files, USERNAME and PASSWORD respectively. Use them to generate a Kubernetes secret dbsecret in the kubecost namespace.

$ kubectl create secret generic dbsecret -n kubecost --from-file=USERNAME --from-file=PASSWORD
$ helm upgrade kubecost kubecost/cost-analyzer --namespace kubecost --set global.prometheus.fqdn=<grafana_prometheus_query_url> --set global.prometheus.enabled=false --set global.prometheus.queryServiceBasicAuthSecretName=dbsecret
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment