Skip to content

Instantly share code, notes, and snippets.

@vladak
Last active May 16, 2023 09:28
Show Gist options
  • Save vladak/872073237312d0a06567f6b97036090b to your computer and use it in GitHub Desktop.
Save vladak/872073237312d0a06567f6b97036090b to your computer and use it in GitHub Desktop.
APC NetBotz temperature/humidity monitoring with Grafana

The APC NetBotz computer lab monitoring units allow SNMP access. This gist summarizes how to collect and present the metrics from these units via Grafana.

The units look like this: APC Netbotz 200

APC Sensor Pod

The result looks like this: Grafana dashboard

This was tested for APC NetBotz Rack Monitor 200 with bunch of APC NetBotz Sensor Pod 150 attached.

Firstly, make sure that basic SNMP access (SNMP v1) works:

snmpwalk -OT -v1 -c public example.com .1

This should output bunch of data.

Next, acquire PowerNet MIB file and use it:

# install pre-requisite tools and MIBs first
sudo apt-get install snmp
sudo apt-get install snmp-mibs-downloader
# walk the walk
snmpwalk -m ./powernet395.mib -OT -v1 -c public example.com memSensorsStatusTable

This should provide output similar to the following (depends on the APC units installed):

PowerNet-MIB::memSensorsStatusModuleNumber.0.1 = INTEGER: 0
PowerNet-MIB::memSensorsStatusModuleNumber.0.2 = INTEGER: 0
PowerNet-MIB::memSensorsStatusModuleNumber.1.1 = INTEGER: 1
PowerNet-MIB::memSensorsStatusModuleNumber.1.2 = INTEGER: 1
PowerNet-MIB::memSensorsStatusModuleNumber.2.1 = INTEGER: 2
PowerNet-MIB::memSensorsStatusModuleNumber.2.2 = INTEGER: 2
PowerNet-MIB::memSensorsStatusModuleNumber.3.3 = INTEGER: 3
PowerNet-MIB::memSensorsStatusModuleNumber.3.5 = INTEGER: 3
PowerNet-MIB::memSensorsStatusModuleNumber.3.6 = INTEGER: 3
PowerNet-MIB::memSensorsStatusSensorNumber.0.1 = INTEGER: 1
PowerNet-MIB::memSensorsStatusSensorNumber.0.2 = INTEGER: 2
PowerNet-MIB::memSensorsStatusSensorNumber.1.1 = INTEGER: 1
PowerNet-MIB::memSensorsStatusSensorNumber.1.2 = INTEGER: 2
PowerNet-MIB::memSensorsStatusSensorNumber.2.1 = INTEGER: 1
PowerNet-MIB::memSensorsStatusSensorNumber.2.2 = INTEGER: 2
PowerNet-MIB::memSensorsStatusSensorNumber.3.3 = INTEGER: 3
PowerNet-MIB::memSensorsStatusSensorNumber.3.5 = INTEGER: 5
PowerNet-MIB::memSensorsStatusSensorNumber.3.6 = INTEGER: 6
PowerNet-MIB::memSensorsStatusSensorName.0.1 = STRING: "Sensor 4"
PowerNet-MIB::memSensorsStatusSensorName.0.2 = STRING: "Sensor 7"
PowerNet-MIB::memSensorsStatusSensorName.1.1 = STRING: "Sensor 3"
PowerNet-MIB::memSensorsStatusSensorName.1.2 = STRING: "Sensor 6"
PowerNet-MIB::memSensorsStatusSensorName.2.1 = STRING: "Sensor 1"
PowerNet-MIB::memSensorsStatusSensorName.2.2 = STRING: "Sensor 2"
PowerNet-MIB::memSensorsStatusSensorName.3.3 = STRING: "Sensor 5"
PowerNet-MIB::memSensorsStatusSensorName.3.5 = STRING: "Sensor 8"
PowerNet-MIB::memSensorsStatusSensorName.3.6 = STRING: "Sensor 9"
PowerNet-MIB::memSensorsStatusSensorLocation.0.1 = STRING: "Cold Aisle C1"
PowerNet-MIB::memSensorsStatusSensorLocation.0.2 = STRING: "Hot Aisle C2"
PowerNet-MIB::memSensorsStatusSensorLocation.1.1 = STRING: "Hot Aisle B8"
PowerNet-MIB::memSensorsStatusSensorLocation.1.2 = STRING: "Cold Aisle C12"
PowerNet-MIB::memSensorsStatusSensorLocation.2.1 = STRING: "Cold Aisle A5"
...

Next, install SNMP exporter generator and use the following configuration:

modules:
  labtemp:
    walk:
      - memSensorsStatusTable
      - memSensorsConfigTable
    version: 1
    timeout: 60s
    lookups:
      - source_indexes: [memSensorsStatusModuleNumber, memSensorsStatusSensorNumber]
        lookup: memSensorsLocation

to generate the SNMP exporter configuration like so (assumes working Golang and the PowerNET MIB placed in $PWD/mibs):

export MIBDIRS=/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp:$PWD/mibs
~/go/bin/generator generate

Next, run the SNMP exporter using the generated snmp.yml configuration file (use https://github.com/prometheus/snmp_exporter/tree/main/examples/systemd as inspiration to run this as a service):

snmp_exporter --config.file=snmp.yml

Verify that the configuration works:

curl 'http://localhost:9116/snmp?target=example.com&module=labtemp'

This should return the metrics in Prometheus format, e.g.:

...
# HELP memSensorsTemperature The sensor's current temperature reading - 1.3.6.1.4.1.318.1.1.10.4.2.3.1.5
# TYPE memSensorsTemperature gauge
memSensorsTemperature{memSensorsLocation="Cold Aisle A5",memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="1"} 22
memSensorsTemperature{memSensorsLocation="Cold Aisle C1",memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="1"} 26
memSensorsTemperature{memSensorsLocation="Cold Aisle C12",memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="2"} 20
memSensorsTemperature{memSensorsLocation="Cold Aisle C9",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="3"} 20
memSensorsTemperature{memSensorsLocation="Hot Aisle B2",memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="2"} 30
memSensorsTemperature{memSensorsLocation="Hot Aisle B8",memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="1"} 30
memSensorsTemperature{memSensorsLocation="Hot Aisle C12",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="6"} 29
memSensorsTemperature{memSensorsLocation="Hot Aisle C2",memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="2"} 32
memSensorsTemperature{memSensorsLocation="Hot Aisle C7",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="5"} 31

The lookups section in the SNMP exporter configuration assigns the location label (memSensorsLocation) to the temperature/humidity metrics, based on the index (the 2 values - memSensorsStatusModuleNumber and memSensorsStatusSensorNumber).

Next, add the following to Prometheus configuration (usually /etc/prometheus/prometheus.yml):

  - job_name: labtemp_snmp
    static_configs:
      - targets:
        - example.com
    scrape_interval: 60s
    scrape_timeout: 60s
    metrics_path: /snmp
    params:
      module: [labtemp]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9116

Note that the APC units might take multiple/many seconds to finish the SNMP request. In my case, with the above configuration, the APC NetBotz 200 unit responded in 16 seconds in average, however sometimes the SNMP operation can take up to one minute or more. That's why the timeouts are explicitly set to higher values.

Next, restart Prometheus.

Next, install Grafana add the Prometheus data source to Grafana. Then the attached dashboard can be used to display the metrics.

There is pre-defined alert in the Grafana dashboard based on the average of temperatures from all sensors over 5 minutes.

{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": [],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "8.5.25"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"alert": {
"alertRuleTags": {},
"conditions": [
{
"evaluator": {
"params": [
30
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"params": [
"B",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "alerting",
"for": "5m",
"frequency": "1m",
"handler": 1,
"message": "lab temperature too high",
"name": "lab temperature alert",
"noDataState": "no_data",
"notifications": []
},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": 300000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "celsius"
},
"overrides": []
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 0
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "memSensorsTemperature",
"legendFormat": "{{memSensorsLocation}}",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum(sum_over_time(memSensorsTemperature[5m])) / sum(count_over_time(memSensorsTemperature[5m]))",
"hide": false,
"legendFormat": "average 5m",
"range": true,
"refId": "B"
}
],
"thresholds": [
{
"colorMode": "critical",
"op": "gt",
"value": 30,
"visible": true
}
],
"title": "lab temperature",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": 300000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "humidity"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 10
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "memSensorsHumidity",
"legendFormat": "{{memSensorsLocation}}",
"range": true,
"refId": "A"
}
],
"title": "lab humidty",
"type": "timeseries"
}
],
"refresh": "5m",
"schemaVersion": 36,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "lab",
"uid": "tSQylb8Vz",
"version": 8,
"weekStart": ""
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment