Custom OpenTelemetry collector for system-level monitoring

README.md

Demo: Building a custom OpenTelemetry collector for system-level monitoring

In this demo we will build a custom OpenTelemetry collector and collect logs and metrics from the operating system.

Prerequisites:

• You need Go in v1.17 or above locally installed.
• socat
• netstat
• jq (optional, for nicer JSON output rendering)

STEP 1: Build the collector

First make sure you have the OTel collector builder tool installed:

GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest

Now build the collector using the supplied otelcol-builder.yaml builder config file:

builder --config=otelcol-builder.yaml --output-path=./myotelcol

STEP 2: Launch signal generators

Next, download the generator files and make them executable (chmod +x ...):

For logs we use the generate-logs.sh script, so download generate-logs.sh. The script tails a local log file and sends a line to the collector's syslog receiver, using socat.

For metrics, we use the generate-metrics.sh script, so download generate-metrics.sh. The script reads network stats via netstat and sends a random connection's receive data to the collector's statsd receiver, using socat.

Next, run the two scripts to generate signals.

STEP 3: Launch the collector

Now it's time to collect & ingest signals so start the OTel collector as follows with the supplied otel-config.yaml collector config file:

./myotelcol --config otel-config.yaml       

STEP 4: View resulting signals

The collected metrics and logs are written to a local file using the File exporter, so create a sub-dir called data and then you can follow the resulting signals using:

tail -f -n 1 data/out.json | jq

generate-logs.sh

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail

sleep_time_sec=${1:-1}
source_log=${2:-"/var/log/acmed.log"}

while : ;
do
  tail -f -n 1 $source_log | socat -v -t 0 STDIN TCP:localhost:54526
  sleep $sleep_time_sec
done

generate-metrics.sh

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail

sleep_time_sec=${1:-1}

while : ;
do
  data_recvq=$(netstat -np TCP | grep -v 127.0.0.1 | grep ESTABLISHED | head -1 | awk '{ print $3 }' | cut -d: -f1)
  local_ip=$(netstat -np TCP | grep -v 127.0.0.1 | grep ESTABLISHED | head -1 | awk '{ print $4 }' | cut -d: -f1)
  foreign_ip=$(netstat -np TCP | grep -v 127.0.0.1 | grep ESTABLISHED | head -1 | awk '{ print $5 }' | cut -d: -f1)
  payload="local.netstat#local_ip=$local_ip,foreign_ip=$foreign_ip:$data_recvq|g"
  echo "sending $payload"
  echo $payload| socat -t 0 STDIN UDP:localhost:8125
  sleep $sleep_time_sec
done

otel-config.yaml

receivers:
  syslog:
    tcp:
      listen_address: "0.0.0.0:54526"
    protocol: rfc5424
  statsd:
    endpoint: "localhost:8125"
    aggregation_interval: 1s
    enable_metric_type: true
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "gauge"
      - statsd_type: "timing"
        observer_type: "gauge"

exporters:
  logging:
    loglevel: debug
    sampling_initial: 2
    sampling_thereafter: 2
  file:
    path: ./data/out.json

service:
  telemetry:
    logs:
      level: debug
    metrics:
      level: detailed
  pipelines:
    logs:
      receivers: [ syslog ]
      exporters: [ logging, file ]
    metrics:
      receivers: [ statsd ]
      exporters: [ logging, file ]

otelcol-builder.yaml

receivers:
- gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.56.0"
- gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/statsdreceiver v0.56.0"

exporters:
- import: "go.opentelemetry.io/collector/exporter/loggingexporter"
  gomod: "go.opentelemetry.io/collector v0.56.0"
- gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.56.0"