Skip to content

Instantly share code, notes, and snippets.

@vladak
Last active May 12, 2023 09:37
Show Gist options
  • Save vladak/5d2447ae774bc4886fe1ab549fff8d8b to your computer and use it in GitHub Desktop.
Save vladak/5d2447ae774bc4886fe1ab549fff8d8b to your computer and use it in GitHub Desktop.
setup for PF monitoring in Grafana

Presenting PF counters in Grafana

using https://yetiops.net/posts/openbsd-snmp-exporter/

The goal is to present basic metrics such as bytes passed through the external interface of the router. The main idea is to run SNMP on the OpenBSD router that can get the metrics from PF. Then SNMP exporter from Prometheus will convert the data to Prometheus format that can be scraped by Grafana.

The constraint is that this should be done in reasonably secure way: the IoT devices are in separate VLAN, however the router should have miminal exposure. Using snmpd with privilege separation and in flight data protection is sufficient. Further, the data collection and graphing will be done on single machine.

The result looks like this: Grafana PF dashboard

Router setup

This assumes that the PF configuration on the OpenBSD router has something like this:

set loginterface $ExtIF

and that PF is enabled and active. Also, there are PF rules in place that allow SNMP traffic to the interface of the IoT network/VLAN.

  • create snmpd setup:
echo snmpd_flags= >> /etc/rc.conf.local
cat << EOF >/etc/snmpd.conf
## in OpenBSD 7.0 it should be possible to say 'listen on any' and rely on PF rules.
listen on 127.0.0.1 read
# IoT
listen on 172.40.0.1 read

# Adjust the local system information
#system contact "Charlie Root (root@myhost.example.com)"
#system description "Powered by OpenBSD"
#system location "Rack A1-24, Room 13"
system services 74

seclevel enc
user "grafana" authkey "bae8aX3i" enc aes enckey "AhChaBa8"
EOF
  • either reboot or run:
snmpd

Grafana server setup

This assumes Raspberry Pi setup ala https://github.com/vladak/pirig/blob/master/weatherpi.md

First confirm that SNMP works:

snmpwalk -l authPriv -a SHA -A bae8aX3i -x AES -X AhChaBa8 -u grafana -v 3 gw

This should produce the output like:

iso.3.6.1.2.1.1.1.0 = STRING: "OpenBSD ... 6.9 GENERIC.MP#551 octeon"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.30155.23.1
iso.3.6.1.2.1.1.3.0 = Timeticks: (13084198) 1 day, 12:20:41.98
iso.3.6.1.2.1.1.4.0 = STRING: "root@..."
iso.3.6.1.2.1.1.5.0 = STRING: "..."
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 74
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
...

Create snmp_exporter configuration

First create ~/snmp/generator.yml:

modules:
  openbsd_pf:
    walk:
      # counters from HOST-RESOURCES-MIB (http://www.net-snmp.org/docs/mibs/host.html)
      - hrSystemUptime
      - hrSystemDate
      - hrProcessorTable
      - hrStorageTable
      # PF counters
      - pfRunning
      - pfRuntime
      - pfHostid
      - pfStateCount
      - pfStateSearches
      - pfStateInserts
      - pfStateRemovals
      - pfLogIfName
      - pfLogIfIpBytesIn
      - pfLogIfIpBytesOut
      - pfLogIfIpPktsInPass
      - pfLogIfIpPktsInDrop
      - pfLogIfIpPktsOutPass
      - pfLogIfIpPktsOutDrop
      - pfIfIn4PassPkts
      - pfIfIn4PassBytes
      - pfIfIn4BlockPkts
      - pfIfIn4BlockBytes
      - pfIfOut4PassPkts
      - pfIfOut4PassBytes
      - pfIfOut4BlockPkts
      - pfIfOut4BlockBytes
      - pfIfDescr
      - pfIfIndex
    version: 3
    lookups:
    - source_indexes: [pfIfIndex]
      lookup: pfIfDescr
      drop_source_indexes: false
    auth:
      username: "grafana"
      security_level: authPriv
      password: "bae8aX3i"
      auth_protocol: SHA
      priv_protocol: AES
      priv_password: "AhChaBa8"

then generate ~/snmp/snmp.yml config file for snmp_exporter:

sudo apt-get install -y golang
sudo apt-get install -y libsnmp-dev # delivers net-snmp/net-snmp-config.h needed by the Go generator
mkdir ~/golang
export GOPATH=~/golang
go get github.com/prometheus/snmp_exporter/generator
cd ~/snmp
export MIBDIRS=/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp:$PWD/mibs
sudo apt-get install -y snmp-mibs-downloader  # needed for base MIBs (in /usr/share/snmp/mibs/) referenced by OpenBSD MIBs
scp -r gw:/usr/share/snmp/mibs mibs  # gw is OpenBSD
$GOPATH/bin/generator generate

Setup snmp_exporter service

Use https://gist.github.com/vladak/eae9a33c4f1b4dd7d0c92aeea4a519a2 to setup SNMP exporter service. Append the generated file to the snmp.yml file used by the SNMP exporter service and restart the service.

  • verify that the data can be retrieved:
curl 'http://localhost:9116/snmp?target=gw&module=openbsd_pf'

The curl output should contain data such as:

# HELP pfIfOut4BlockBytes The number of outgoing IPv4 bytes blocked. - 1.3.6.1.4.1.30155.1.8.128.1.13
# TYPE pfIfOut4BlockBytes counter
pfIfOut4BlockBytes{pfIfDescr="all",pfIfIndex="1"} 0
pfIfOut4BlockBytes{pfIfDescr="carp",pfIfIndex="2"} 0
pfIfOut4BlockBytes{pfIfDescr="cnmac0",pfIfIndex="3"} 659404
pfIfOut4BlockBytes{pfIfDescr="cnmac1",pfIfIndex="4"} 0
pfIfOut4BlockBytes{pfIfDescr="cnmac2",pfIfIndex="5"} 0
...

Update Prometheus configuration

update /etc/prometheus/prometheus.yml with (because the service running on port 9116 requires the target and module parameters, these are added through relabeling):

scrape_configs:
  - job_name: openbsd_pf_snmp
    static_configs:
      - targets:
        - gw
    scrape_interval: 15s
    metrics_path: /snmp
    params:
      module: [openbsd_pf]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9116
  • restart Prometheus:
sudo systemctl restart prometheus
  • use journalctl _SYSTEMD_UNIT=prometheus.service to check that the configuration was reloaded correctly (unlike Solaris' SMF there does not seem to be a concept of degraded service if the configuration was not parsed successfully).

Grafana configuration

setup new dashboard and add a few panels referencing metrics like pfLogIfIpBytesIn or pfLogIfIpBytesOut.

{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "9.0.3"
},
{
"type": "panel",
"id": "graph",
"name": "Graph (old)",
"version": ""
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "Bps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 2,
"links": [],
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "9.0.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfLogIfIpBytesIn[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "in",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfLogIfIpBytesOut[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "out",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "delta(pfLogIfIpBytesIn[1h])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"refId": "C"
}
],
"title": "external interface traffic (1m)",
"type": "timeseries"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"hiddenSeries": false,
"id": 4,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "9.0.3",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "pfStateCount",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "states",
"refId": "A"
}
],
"thresholds": [],
"timeRegions": [],
"title": "PF states count",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"mode": "time",
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"show": true
},
{
"format": "short",
"logBase": 1,
"show": true
}
],
"yaxis": {
"align": false
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 8
},
"hiddenSeries": false,
"id": 8,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "9.0.3",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfLogIfIpPktsInDrop[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "in",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfLogIfIpPktsOutDrop[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "out",
"refId": "B"
}
],
"thresholds": [],
"timeRegions": [],
"title": "rate of dropped packets on the external interface (1m)",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"mode": "time",
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"show": true
},
{
"format": "short",
"logBase": 1,
"show": true
}
],
"yaxis": {
"align": false
}
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 8
},
"id": 6,
"links": [],
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.3.6",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfStateInserts[5m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "inserts",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfStateRemovals[5m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "removals",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "rate(pfStateSearches[5m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "searches",
"refId": "C"
}
],
"title": "PF state actions (5m)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "in percent",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 15
},
"id": 12,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "hrProcessorLoad{job=\"openbsd_pf_snmp\"}",
"legendFormat": "{{hrDeviceIndex}}",
"range": true,
"refId": "A"
}
],
"title": "CPU load",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "Bps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 15
},
"id": 10,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "rate(pfIfIn4PassBytes[5m])",
"interval": "",
"legendFormat": "in:{{pfIfDescr}}",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "rate(pfIfOut4PassBytes[5m])",
"hide": false,
"interval": "",
"legendFormat": "out:{{pfIfDescr}}",
"range": true,
"refId": "B"
}
],
"title": "Interface pass bytes",
"type": "timeseries"
}
],
"refresh": "1m",
"schemaVersion": 36,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Router",
"uid": "3tt57oZgz",
"version": 19,
"weekStart": ""
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment