Node Exporter + Prometheus + Grafana Provisioning

README.md

Node Exporter + Prometheus + Grafana Provisioning

This is my documentation about installing monitoring stack on Ubuntu Server 20.04 LTS.

Scheme : Node > Node Exporter > Prometheus > Grafana.

Installation

• Install Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xvf node_exporter-1.3.1.linux-amd64.tar.gz 
mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/ 
useradd -rs /bin/false node_exporter

• Configure Node Exporter

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
Environment=GOMAXPROCS=1
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

• with Collector

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Environment=GOMAXPROCS=1
Restart=always
RuntimeMaxSec=1d
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=0.0.0.0:9100 --collector.disable-defaults --collector.cpu --collector.cpufreq --collector.meminfo --collector.loadavg --collector.uname --collector.stat --collector.netclass --collector.netdev --collector.netstat --collector.ethtool --collector.filesystem --collector.diskstats --collector.systemd

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

• Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.1.0/prometheus-2.1.0.linux-amd64.tar.gz
tar -xf prometheus-2.1.0.linux-amd64.tar.gz
sudo mv prometheus-2.1.0.linux-amd64/prometheus prometheus-2.1.0.linux-amd64/promtool /usr/local/bin
sudo mkdir /etc/prometheus /var/lib/prometheus
sudo mv prometheus-2.1.0.linux-amd64/consoles prometheus-2.1.0.linux-amd64/console_libraries /etc/prometheus
rm -r prometheus-2.1.0.linux-amd64*

• Configure Prometheus

nano /etc/prometheus/prometheus.yml

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100','prometheus-target-1:9100','prometheus-target-2:9100']

sudo useradd -rs /bin/false prometheus
sudo chown -R prometheus: /etc/prometheus /var/lib/prometheus

nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --storage.tsdb.retention.time=1d \
    --storage.tsdb.max-block-duration=6h \
    --storage.tsdb.min-block-duration=6h \

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Access on http://<your_server_IP>:9090/

• Install Grafana

cd /tmp
wget -c https://dl.grafana.com/oss/release/grafana_7.4.1_amd64.deb
dpkg -i grafana_7.4.1_amd64.deb

Access on http://your.server.ip:3000 with Username admin and Password admin

• Restart

systemctl daemon-reload
systemctl enable grafana-server
systemctl restart grafana-server
systemctl status grafana-server

Example Nodeexporter JSON

• 1
• 1860

Alerting with Picture

• https://gist.github.com/gilangvperdana/859e4e80b9c1259c16fe98b55f34023d

BlockPortfromOutside.md

General

If you want to block port node exporter access from outside you can use this command

iptables -A INPUT -p tcp -s 10.7.0.0/24 --dport 9100 -j ACCEPT
iptables -A INPUT -p tcp -s localhost --dport 9100 -j ACCEPT
iptables -A INPUT -p tcp --dport 9100 -j DROP

• 10.7.0.0/24 & localhost are IP ALLOWED to access port 9100

WindowsExporter.md

General

This is tutorial for you guys if youw want to monitor your Windows Server with Windows Exporter

Installation Agent

• Download on Here
• Download .msi
• Exec this on Powershell (run as administrator)

msiexec /i windows_exporter-0.16.0-amd64.msi ENABLED_COLLECTORS="ad,adfs,cache,cpu,cpu_info,cs,container,dfsr,dhcp,dns,fsrmquota,iis,logical_disk,logon,memory,msmq,mssql,netframework_clrexceptions,netframework_clrinterop,netframework_clrjit,netframework_clrloading,netframework_clrlocksandthreads,netframework_clrmemory,netframework_clrremoting,netframework_clrsecurity,net,os,process,remote_fx,service,tcp,time,vmware" TEXTFILE_DIR="C:\custom_metrics" LISTEN_PORT="9115"

Add to prometheus.yml

  - job_name: 'windowsexporter'
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.100.243:9115']

Add Grafana Dashboard from Template

• 14694
• 14510
• 14451