Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
sample prometheus configuration explained
// For all the confusing Prometheus configuration and
// regular expressions,
// explained in examples.
// Remember, there are default values for each item if it's missing.
// regex is (.*),
// replacement is $1,
// separator is ;
// ,and action is replace
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
// keep targets with label __meta_kubernetes_service_annotation_prometheus_io_scrape equals 'true',
// which means the user added prometheus.io/scrape: true in the service's annotation.
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
// if __meta_kubernetes_service_annotation_prometheus_io_scheme is http or https, replace __scheme__ with its value that is the default replacement: $1.
// this means the user added prometheus.io/scheme in the service's annotation.
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
// this is also for service annotation of prometheus, if the user overrides the scrape path,
// its value will be put in __metrics_path__
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)(?::\d+);(\d+)
replacement: $1:$2
// if the user added prometheus.io/port in the service annotation,
// use this port to replace the port in __address__
// all above rules are for handling prometheus annotation: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
// all __meta_kubernetes_service_label_(.+) will be changed to the (.+)
// e.g. __meta_kubernetes_service_label_app='armada-api' to app='armada-api'
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
// if __meta_kubernetes_namespace matches .*, put its value in label kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service_name
// if __meta_kubernetes_service_name matches .*, put its value in label service_name
- source_labels: [__meta_kubernetes_pod_node_name]
action: replace
target_label: hostname
// if __meta_kubernetes_pod_node_name matches .*, put its value in label hostname
@girol

This comment has been minimized.

Copy link

@girol girol commented Sep 6, 2019

Thank you!!! 💓

@ludurr

This comment has been minimized.

Copy link

@ludurr ludurr commented Sep 30, 2019

Thx a lot!

@adrianmace

This comment has been minimized.

Copy link

@adrianmace adrianmace commented Oct 27, 2019

Posting my thanks - very helpful 👍

@nicocolt

This comment has been minimized.

Copy link

@nicocolt nicocolt commented Nov 4, 2019

Very helpful, thanks !

@reachlin

This comment has been minimized.

Copy link
Owner Author

@reachlin reachlin commented Nov 5, 2019

guys, thanks for all your comments, but pls check the latest document, since it has been changed a lot on prometheus configuration

@nicocolt

This comment has been minimized.

Copy link

@nicocolt nicocolt commented Nov 5, 2019

Thanks, can you provide us the updated link ?

@weshouman

This comment has been minimized.

Copy link

@weshouman weshouman commented Mar 4, 2020

Well done 🥇 ! That should really go into the docs

@maphysics

This comment has been minimized.

Copy link

@maphysics maphysics commented Jun 4, 2020

Thanks!

@bschaeffer

This comment has been minimized.

Copy link

@bschaeffer bschaeffer commented Jun 12, 2020

I am still confused... If I have a metric

some_metric{app="foo",baz="true"}
some_metric{app="bar",baz="true"}

How do I drop all baz labels and keep only app=foo metrics so that I end up with:

some_metric{app="foo"}

Also, what is separator? What am I separating? The only thing I can think of are labels, and I have never seen them separated by a ; which makes using it as the default very confusing.

@songleo

This comment has been minimized.

Copy link

@songleo songleo commented Jul 8, 2020

@bschaeffer you can use labeldrop action to do it. for details https://medium.com/quiq-blog/prometheus-relabeling-tricks-6ae62c56cbda

@kmurthyms

This comment has been minimized.

Copy link

@kmurthyms kmurthyms commented Jan 21, 2021

for replace action , should the target label be already present / if its not present does prometheus add one ?

@tal-ayalon

This comment has been minimized.

Copy link

@tal-ayalon tal-ayalon commented Jan 25, 2021

Amazing, made my day much easier, thanks!

@reachlin

This comment has been minimized.

Copy link
Owner Author

@reachlin reachlin commented Jan 26, 2021

thanks everyone, but really not in the mood to update this. i just got laid off by IBM :(

@nicocolt

This comment has been minimized.

Copy link

@nicocolt nicocolt commented Jan 26, 2021

I'm so sad to heard this :( good luck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment