Prometheus Relabel Config Example

prometheus_relabel_configs.md

Prometheus Relabling

Using a standard prometheus config to scrape two targets:

• ip-192-168-64-29.multipass:9100
• ip-192-168-64-30.multipass:9100

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'local'

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'multipass-nodes'
    static_configs:
    - targets: ['ip-192-168-64-29.multipass:9100']
      labels:
        test: 1
    - targets: ['ip-192-168-64-30.multipass:9100']
      labels:
        test: 1

The Result:

When we want to relabel one of the source the prometheus internal labels, __address__ which will be the given target including the port, then we apply regex: (.*) to catch everything from the source label, and since there is only one group we use the replacement as ${1}-randomtext and use that value to apply it as the value of the given target_label which in this case is for randomlabel, which will be in this case:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'local'

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'multipass-nodes'
    static_configs:
    - targets: ['ip-192-168-64-29.multipass:9100']
      labels:
        test: 3
    - targets: ['ip-192-168-64-30.multipass:9100']
      labels:
        test: 3
    relabel_configs:
    - source_labels: [__address__]
      regex: '(.+)'
      replacement: '${1}-randomtext'
      target_label: randomlabel

The Result:

In this case we want to relabel the __address__ and apply the value to the instance label, but we want to exclude the :9100 from the __address__ label:

# Config: https://github.com/prometheus/prometheus/blob/release-2.36/config/testdata/conf.good.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'local'

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'multipass-nodes'
    static_configs:
    - targets: ['ip-192-168-64-29.multipass:9100']
      labels:
        test: 4
    - targets: ['ip-192-168-64-30.multipass:9100']
      labels:
        test: 4
    relabel_configs:
    - source_labels: [__address__]
      separator: ':'
      regex: '(.*):(.*)'
      replacement: '${1}'
      target_label: instance

The Result:

Stack

The docker-compose used:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus
    container_name: 'prometheus'
    user: root
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=14d'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.external-url=http://prometheus.127.0.0.1.nip.io'
    ports:
      - 9090:9090
    networks:
      - public
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  public:
    name: public

volumes:
  prometheus-data: {}

References

Usful docs:

• https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/#internal-labels
• https://regexr.com/