Step by step to Monitoring OpenStack Instances with Service Discovery Prometheus and Grafana

README.md

Monitoring Instances with Prometheus and Grafana

Download Prometheus Server

cd /tmp
wget -c https://github.com/prometheus/prometheus/releases/download/v2.13.1/prometheus-2.13.1.linux-amd64.tar.gz

Extract Prometheus Server

tar zxvf prometheus-2.13.1.linux-amd64.tar.gz

Move to some directory

mv prometheus-2.13.1.linux-amd64 /opt/prometheus-server
cd /opt/prometheus-server

Edit configuration file

• For example we want to monitor instance on endpoint 167.205.192.12

vim config.yml

# my global config
 
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
 
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
 
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
 
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: node-exporter-instances
    static_configs:
    - targets:
      - 167.205.192.12:9100

Configure Systemd Service for Prometheus Service

vim /etc/systemd/system/prometheus-server.service

[Unit]
Description=Prometheus Server
 
[Service]
User=root
ExecStart=/opt/prometheus-server/prometheus --config.file=/opt/prometheus-server/config.yml --web.external-url=http://ip-server:9090/
 
[Install]
WantedBy=default.target

Running Service

systemctl daemon-reload
systemctl enable prometheus-server.service
systemctl start prometheus-server.service

Verification Service

systemctl status prometheus-server.service
ss -tulpn |grep 9090

Install Grafana

cd /tmp
wget -c https://dl.grafana.com/oss/release/grafana_6.4.3_amd64.deb
dpkg -i grafana_6.4.3_amd64.deb

Running Service

systemctl daemon-reload
systemctl start grafana-server
systemctl enable grafana-server

Verification

systemctl status grafana-server
ss -tulpn | grep 3000

Configura Datasource and Dashboard on Grafana

• Login to Grafana
• Add data source
• Choose Prometheus

Name: Prometheus
Http settings
URL: http://localhost:9090

• Save and Test
• Add Dashboard

Create new instance with node exporter

Install node-exporter on ubuntu instances

cd /tmp/
wget -c https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar zxvf node_exporter-0.18.1.linux-amd64.tar.gz 
mv node_exporter-0.18.1.linux-amd64 /opt/node_exporter

Create systemd service and running node-exporter

cat << EOF >/etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
 
[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter
 
[Install]
WantedBy=default.target
EOF
 
systemctl daemon-reload
systemctl enable node_exporter.service
systemctl start node_exporter.service
systemctl status node_exporter.service

Best dashboard template

Dashboard Name	Grafana ID
Ceph - Cluster	2842
Ceph - Pools	5342
Ceph - OSD (Single)	5336
HAProxy	2428
Servers	11074
Galera/MariaDB	13106
Openstack Dashboard	9701
Docker Monitoring	8321

Best Query for Instance SysLoad Monitoring

avg(node_load5{instance="$node",job="$job"}) /  count(count(node_cpu_seconds_total{instance="$node",job="$job"}) by (cpu)) * 100

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)