Victoria Metrics Provisioning

README.md

Victoria Metrics Provisioning

Tested on Ubuntu 20.04LTS

If you have a problem to handle a much Prometheus Data Sources, you can alternatively add VictoriaMetrics to handle a much Prometheus Server just with one Data Sources on Grafana after this.

Before

Before you install this, make sure you have installed an monitoring stack NodeExporter/Prometheus/Grafana/etc.

Installation

• Download & Install Package

curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.69.0/victoria-metrics-amd64-v1.69.0.tar.gz --output victoria-metrics-amd64-v1.69.0.tar.gz
sudo tar xvf victoria-metrics-amd64-v1.69.0.tar.gz -C /usr/local/bin/
sudo groupadd -r victoriametrics
sudo useradd -g victoriametrics -d /var/lib/victoriametrics -s /sbin/nologin --system victoriametrics
sudo chown -v victoriametrics:victoriametrics /usr/local/bin/victoria-metrics-prod
sudo mkdir -v /data
sudo chown -v victoriametrics:victoriametrics /data

• Change Systemd Configuration

sudo nano /etc/systemd/system/victoriametrics.service

[Unit]
Description=VictoriaMetrics
After=network.target

[Service]
User=victoriametrics
Group=victoriametrics
Type=simple
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
PIDFile=/run/victoriametrics/victoriametrics.pid
ExecStart=/usr/local/bin/victoria-metrics-prod \        
        -storageDataPath /data \
        -retentionPeriod 6 \
        -selfScrapeInterval 30s \
        -maxConcurrentInserts=100 \
        -insert.maxQueueDuration=30s \
        -search.maxUniqueTimeseries=1800000 \
ExecStop=/bin/kill -s SIGTERM $MAINPID

[Install]
WantedBy=multi-user.target

[Service]
LimitNOFILE=32000
LimitNPROC=32000

• Enable & Reload Victoria Metrics

sudo systemctl daemon-reload
sudo systemctl start victoriametrics.service
sudo systemctl enable victoriametrics.service
sudo systemctl status victoriametrics.service

Important

So, where we can consume Victoria Metrics on existing Prometheus? You must define an Remote_Write on your prometheus.yml

• Configure prometheus.yml

sudo nano /etc/prometheus/prometheus.yml

remote_write:
  - url: http://<ip_victoriametrics>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      max_shards: 30

After this, if you want to efficiency storage on your existing prometheus let's do auto delete with Retention configuration.

• Adjust retention on your prometheus stack

sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--storage.tsdb.retention 0m

[Install]
WantedBy=multi-user.target

Please just focuss on --storage.tsdb.retention

• Reload Prometheus Service

sudo systemctl daemon-reload
sudo systemctl restart prometheus.service
sudo systemctl status prometheus.service

Final

• Now, you can change your endpoint of your Grafana Data Source from Prometheus to Victoria Metrics on Data Sources Page.
• Victoria Metrics have endpoint : http://localhost:8428