#Grafana #Prometheus Linux & MySQL monitoring

grafana_prometheus.md

Grafana MySQL & Linux Monitoring - No Docker

Server Install

Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v1.3.1/prometheus-1.3.1.linux-amd64.tar.gz

tar xvfz prometheus-1.3.1.linux-amd64.tar.gz
sudo mv prometheus-1.3.1.linux-amd64 prometheus
cd prometheus

Configure prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'linux'
    static_configs:
      - targets: ['<client-ip>:9100', '<client-ip>:9100', '<client-ip>:9100', '<client-ip>:9100']
        labels:
          group: 'os-production'
      - targets: ['<client-ip>:9100']
        labels:
          group: 'os-stage'

  - job_name: 'mysql'
    static_configs:
      - targets: ['<client-ip>:9104', '<client-ip>:9104', '<client-ip>:9104', '<client-ip>:9104']
        labels:
          group: 'db-production'

Run at startup http://<ip>:9090

chmod +x /etc/rc.d/rc.local

sudo sh -c "cat << EOF > /root/start_prometheus.sh
/opt/prometheus/prometheus -config.file=/opt/prometheus/prometheus.yml
EOF"

chmod +x /root/start_prometheus.sh

Add /root/start_prometheus.sh to the bottom of /etc/rc.local

Run in background (until restart) http://<ip>:9090

sudo screen -dmS "prometheus" /root/start_prometheus.sh

or

sudo /root/start_prometheus.sh &
disown

Reload configuration file

curl -X POST http://localhost:9090/-/reload

Install Grafana

sudo yum install https://grafanarel.s3.amazonaws.com/builds/grafana-4.0.0-1478693311beta1.x86_64.rpm

Configure Grafana

/usr/share/grafana/conf/defaults.ini
/etc/grafana/grafana.ini

Start Grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Start Grafana at boot

sudo systemctl enable grafana-server.service

Upgrade Grafana

sudo yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-<version>.x86_64.rpm

Client Install

Install linux metrics

Replace <client ip> with client IP

sudo wget https://github.com/prometheus/node_exporter/releases/download/v0.14.0/node_exporter-0.14.0.linux-amd64.tar.gz
sudo mkdir /opt/prometheus_exporters
sudo tar zxf node_exporter-0.14.0.linux-amd64.tar.gz -C /opt/prometheus_exporters
sudo mv /opt/prometheus_exporters/node_exporter-0.14.0.linux-amd64/ /opt/prometheus_exporters/node_exporter/

Run at startup http://<client ip>:10100

sudo chmod +x /etc/rc.d/rc.local

sudo sh -c "cat << EOF > /root/start_node_exporter.sh
/opt/prometheus_exporters/node_exporter/node_exporter -web.listen-address=\"<client ip>:10100\"
EOF"

sudo chmod +x /root/start_node_exporter.sh

Add /root/start_node_exporter.sh to the bottom of /etc/rc.local

Run in background (until restart) http://<client ip>:10100

sudo screen -dmS "node_exporter" /root/start_node_exporter.sh

or

sudo /root/start_node_exporter.sh &
disown

Install MySQL metrics

Replace <client ip> with client IP

sudo wget https://github.com/prometheus/mysqld_exporter/releases/download/0.7.1/mysqld_exporter-0.7.1.linux-amd64.tar.gz
sudo mkdir /opt/prometheus_exporters
sudo tar zxf mysqld_exporter-0.7.1.linux-amd64.tar.gz -C /opt/prometheus_exporters

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'passw0rd';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost'
  WITH MAX_USER_CONNECTIONS 3;

sudo sh -c "cat << EOF > /opt/prometheus_exporters/.my.cnf
[client]
user=exporter
password=passw0rd
EOF"

Run at startup http://<client ip>:10104

sudo chmod +x /etc/rc.d/rc.local

sudo sh -c "cat << EOF > /root/start_mysqld_exporter.sh
/opt/prometheus_exporters/mysqld_exporter -config.my-cnf=\"/opt/prometheus_exporters/.my.cnf\" -web.listen-address=\"<client ip>:10104\"
EOF"

sudo chmod +x /root/start_mysqld_exporter.sh

Add /root/start_mysqld_exporter.sh to the bottom of /etc/rc.local

Run in background (until restart) http://<client ip>:10104

sudo screen -dmS "mysqld_exporter" /root/start_mysqld_exporter.sh

or

sudo /root/start_mysqld_exporter.sh &
disown

View & attach backgound screen

sudo screen -ls

sudo screen -r "node_exporter"

Tips

• When importing Grafana MySQL dashboard json, you may be required to replace mysql_up with mysql_exporter_scrapes_total
• Use -web.listed-address to change exporter port:

/opt/prometheus_exporters/mysqld_exporter -config.my-cnf="/opt/prometheus_exporters/.my.cnf" -web.listen-address="10.20.30.40:10101"

• Open exporter ports

sudo firewall-cmd --zone=public --permanent --add-port=10100/tcp
sudo firewall-cmd --zone=public --add-port=10100/tcp

• Find used ports

sudo sh -c "netstat -anp | grep 10100"

• kill process

sudo kill -15 <pid>