Setup prometheus-node-exporter and push stats to Pushgateway with cron job

export-node-stats.md

(Assuming a Debian 8-like system)

• Install prometheus-node-exporter

  $ sudo apt update && sudo apt install prometheus-node-exporter

• Configure prometheus-node-exporter to expose metrics only to localhost, not on to all networks. Modify file /etc/default/prometheus-node-exporter:

  # Set the command-line arguments to pass to the server.
  ARGS="-collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ \
        -collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
        -collector.textfile.directory=/var/lib/prometheus/node-exporter" \
        -web.listen-address localhost:9100  # added by ADMIN (also `\` characted in line above)

• Enable and restart prometheus-node-exporter service:

  $ sudo systemctl enable prometheus-node-exporter
  $ sudo systemctl restart prometheus-node-exporter
  # check restart was successful
  $ sudo systemctl status prometheus-node-exporter

• Check port 9100 is bound only to loopback address 127.0.0.1:

  $ sudo netstat -atnp
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  ...
  tcp        0      0 127.0.0.1:9100          0.0.0.0:*               LISTEN      31397/prometheus-no
  ...

• Create new file for push-to-pull script, e.g. file path ~root/push_node_exporter_metrics.sh

  #!/bin/bash
  PUSHGATEWAY_SERVER=https://PUSHGATEWAY.EXAMPLE.COM
  NODE_NAME=`hostname`
  curl -s localhost:9100/metrics | curl -u USERNAME:PASSWORD --data-binary @- $PUSHGATEWAY_SERVER/metrics/job/node-exporter/instance/$NODE_NAME

  Restrict file access since it contains a password and make it executable for its owner

  $ sudo chmod og-rwx ~root/push_node_exporter_metrics.sh
  $ sudo chmod u+x ~root/push_node_exporter_metrics.sh

• Setup cron job to regularly push data to Prometheus Pushgateway

  $ sudo crontab -e

  Add

  # added by ADMIN to push node stats to Prometheus Pushgateway every minute
  */1 * * * * /root/push_node_exporter_metrics.sh &> /dev/null