Skip to content

Instantly share code, notes, and snippets.

@ScriptingSquirrel
Last active April 4, 2023 02:18
Show Gist options
  • Save ScriptingSquirrel/60cbbdcec4d3f1519ce35dc23242ac1f to your computer and use it in GitHub Desktop.
Save ScriptingSquirrel/60cbbdcec4d3f1519ce35dc23242ac1f to your computer and use it in GitHub Desktop.
Setup prometheus-node-exporter and push stats to Pushgateway with cron job

(Assuming a Debian 8-like system)

  • Install prometheus-node-exporter

    $ sudo apt update && sudo apt install prometheus-node-exporter
  • Configure prometheus-node-exporter to expose metrics only to localhost, not on to all networks. Modify file /etc/default/prometheus-node-exporter:

    # Set the command-line arguments to pass to the server.
    ARGS="-collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ \
          -collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
          -collector.textfile.directory=/var/lib/prometheus/node-exporter" \
          -web.listen-address localhost:9100  # added by ADMIN (also `\` characted in line above)
  • Enable and restart prometheus-node-exporter service:

    $ sudo systemctl enable prometheus-node-exporter
    $ sudo systemctl restart prometheus-node-exporter
    # check restart was successful
    $ sudo systemctl status prometheus-node-exporter
  • Check port 9100 is bound only to loopback address 127.0.0.1:

    $ sudo netstat -atnp
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    ...
    tcp        0      0 127.0.0.1:9100          0.0.0.0:*               LISTEN      31397/prometheus-no
    ...
  • Create new file for push-to-pull script, e.g. file path ~root/push_node_exporter_metrics.sh

    #!/bin/bash
    PUSHGATEWAY_SERVER=https://PUSHGATEWAY.EXAMPLE.COM
    NODE_NAME=`hostname`
    curl -s localhost:9100/metrics | curl -u USERNAME:PASSWORD --data-binary @- $PUSHGATEWAY_SERVER/metrics/job/node-exporter/instance/$NODE_NAME

    Restrict file access since it contains a password and make it executable for its owner

    $ sudo chmod og-rwx ~root/push_node_exporter_metrics.sh
    $ sudo chmod u+x ~root/push_node_exporter_metrics.sh
  • Setup cron job to regularly push data to Prometheus Pushgateway

    $ sudo crontab -e

    Add

    # added by ADMIN to push node stats to Prometheus Pushgateway every minute
    */1 * * * * /root/push_node_exporter_metrics.sh &> /dev/null
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment