Skip to content

Instantly share code, notes, and snippets.

@petarnikolovski

petarnikolovski/prometheus.md

Last active October 12, 2021 01:19
Show Gist options
  • Save petarnikolovski/18ae938aaef4c4ff58189df8a4fc7de9 to your computer and use it in GitHub Desktop.
Save petarnikolovski/18ae938aaef4c4ff58189df8a4fc7de9 to your computer and use it in GitHub Desktop.
Download ZIP
Prometheus 2.x installation on Ubuntu 16.04 server.
Raw
prometheus.md

Installing Prometheus on Ubuntu 16.04

This gist is a compilation of two tutorials. You can find the original tutorials here and here. What should you know before using this? Everything can be executed from the home folder. For easier cleanup at the end you can make directory where you'll download everything, and then just use rm -rf .. Although, you should be careful. If some strange bugs arise unexpectedly somewhere sometimes, just keep in mind that some user names have underscores in them (this is probably nothing to worry about).

Create Users

sudo adduser --no-create-home --disabled-login --shell /bin/false --gecos "Prometheus Monitoring User" prometheus
sudo adduser --no-create-home --disabled-login --shell /bin/false --gecos "Node Exporter User" node_exporter
sudo adduser --no-create-home --disabled-login --shell /bin/false --gecos "Alertmanager User" alertmanager
sudo adduser --no-create-home --disabled-login --shell /bin/false --gecos "Blackbox Exporter User" blackbox_exporter

Create Directories

sudo mkdir /etc/prometheus
sudo mkdir /etc/alertmanager
sudo mkdir /etc/alertmanager/data
sudo mkdir /etc/alertmanager/template
sudo mkdir /etc/blackbox
sudo mkdir /var/lib/prometheus

Create Dummy Config Files

sudo touch /etc/prometheus/prometheus.yml
sudo touch /etc/alertmanager/alertmanager.yml
sudo touch /etc/blackbox/blackbox.yml

Assign Ownership

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R alertmanager:alertmanager /etc/alertmanager
sudo chown -R blackbox_exporter:blackbox_exporter /etc/blackbox
sudo chown prometheus:prometheus /var/lib/prometheus

Download Binaries

wget https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
wget https://github.com/prometheus/alertmanager/releases/download/v0.12.0/alertmanager-0.12.0.linux-amd64.tar.gz
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.11.0/blackbox_exporter-0.11.0.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.6.3_amd64.deb

Untar

tar xvzf prometheus-2.0.0.linux-amd64.tar.gz
tar xvzf alertmanager-0.12.0.linux-amd64.tar.gz
tar xvzf blackbox_exporter-0.11.0.linux-amd64.tar.gz
tar xvzf node_exporter-0.15.2.linux-amd64.tar.gz

Copy Binaries to /usr/local/bin (and for main app copy libs...)

sudo cp prometheus-2.0.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.0.0.linux-amd64/promtool /usr/local/bin/
sudo cp -r prometheus-2.0.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.0.0.linux-amd64/console_libraries /etc/prometheus
sudo cp alertmanager-0.12.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.12.0.linux-amd64/amtool /usr/local/bin/
sudo cp blackbox_exporter-0.11.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo cp node_exporter-0.15.2.linux-amd64/node_exporter /usr/local/bin/

Assign Ownership Again

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter

Sample Files

Prometheus

sudo nano /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

Blackbox

sudo nano /etc/blackbox/blackbox.yml
modules:
  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET

Alertmanager

sudo nano /etc/alertmanager/alertmanager.yml
global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

# templates:
# - '/etc/alertmanager/template/*.tmpl'

route:
  repeat_interval: 3h 
  receiver: team-X-mails

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'team-X+alerts@example.org'

systemd Configuration

Prometheus Unit

sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Node Exporter Unit

sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

Blackbox Exporter Unit

sudo nano /etc/systemd/system/blackbox_exporter.service
[Unit]
Description=Prometheus blackbox exporter
After=network.target auditd.service

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox/blackbox.yml
Restart=on-failure

[Install]
WantedBy=default.target

sudo systemctl daemon-reload
sudo systemctl enable blackbox_exporter
sudo systemctl start blackbox_exporter
sudo systemctl status blackbox_exporter

Alertmanager Unit

sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alert Manager service
Wants=network-online.target
After=network.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml --storage.path /etc/alertmanager/data
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

Configuring Firewall

Think about this step. What is your current server firewall setup? What ports do you really need?

sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 9090
sudo ufw allow 9100
sudo ufw allow 9093
sudo ufw allow 9115
sudo ufw allow 3000

sudo ufw enable

Installing Grafana

sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_4.6.3_amd64.deb

or

sudo apt-get update
sudo apt-get install -y gdebi
sudo gdebi grafana_4.6.3_amd64.deb

and then!

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

Cleanup

sudo rm prometheus-2.0.0.linux-amd64.tar.gz
sudo rm alertmanager-0.12.0.linux-amd64.tar.gz
sudo rm blackbox_exporter-0.11.0.linux-amd64.tar.gz
sudo rm node_exporter-0.15.2.linux-amd64.tar.gz
sudo rm grafana_4.6.3_amd64.deb

sudo rm -rf prometheus-2.0.0.linux-amd64
sudo rm -rf alertmanager-0.12.0.linux-amd64
sudo rm -rf blackbox_exporter-0.11.0.linux-amd64
sudo rm -rf node_exporter-0.15.2.linux-amd64

or

(do the following only if you're sure what you're doing! don't do stupid stuff) sudo rm -rf .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment