Setting Up a Monitoring Stack with Grafana, Prometheus, Node Exporter, and cAdvisor in Docker Compose, Exposed via NGINX

1. GRAFANA.md

Setting Up a Monitoring Stack with Grafana, Prometheus, Node Exporter, cAdvisor and Loki in Docker Compose, Exposed via NGINX

This guide provides instructions on setting up a comprehensive monitoring stack using Grafana, Prometheus, Node Exporter, cAdvisor and Loki. These components are orchestrated with Docker Compose and exposed via an NGINX reverse proxy, making them accessible through a single domain.

Components

• Grafana: The analytics and monitoring solution with support for multiple data sources, including Prometheus.
• Prometheus: The monitoring and alerting toolkit, collecting metrics from configured targets at specified intervals.
• Node Exporter: A Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.
• cAdvisor: Analyzes resource usage and performance characteristics of running containers.
• Loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system.
• NGINX: Used as a reverse proxy to expose Grafana on the internet securely.

Prerequisites

• Docker and Docker Compose installed on your host machine.
• Domain name configured to point to the host machine (for NGINX configuration).

Configuration Files Overview

• docker-compose.yml: Defines the services, networks, and volumes for the Docker containers.
• prometheus.yml: Configuration file for Prometheus to define scrape targets and intervals.
• grafana.ini: Configuration file for Grafana's settings, including SMTP for email notifications.
• loki-config.yml: Configuration file for Loki to define the storage backend and other settings.
• domain.conf: NGINX configuration for proxying requests to Grafana and handling WebSocket connections.

Installation Steps

1. Configure Docker Daemon: Enable Docker metrics by adding the following to /etc/docker/daemon.json and restarting Docker:

   {
     "metrics-addr": "127.0.0.1:9323"
   }
   # Then reload Docker configurations:
   sudo systemctl reload docker

2. Install Loki plugin for Docker: Install the Loki plugin for Docker to enable log collection:

   docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

3. Clone the Repository: Clone the repository to your host machine:

   git clone https://gist.github.com/398e94fb0aebb994b25223d99e7ae769.git monitoring-stack
   cd monitoring-stack

4. Start the Services: Start the monitoring stack using Docker Compose:

   docker-compose up -d

5. Access Grafana: Access Grafana at http://<your-domain> and log in with the default credentials (username: admin, password: admin).

Note

Important Notes:

• Replace placeholder values in grafana.ini with your actual SMTP credentials to enable email notifications.

• Adjust the domain.conf to match your domain and specific requirements.

• Review Docker and NGINX logs in case of any issues during setup.

docker-compose.yml

version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    network_mode: "host"
    volumes:
      - "./loki-config.yml:/mnt/config/loki-config.yml"
      - "loki-data:/loki"
    command: -config.file=/mnt/config/loki-config.yml
    restart: unless-stopped
    expose:
      - "3100"
      - "9096"

  node-exporter:
    image: prom/node-exporter:latest
    network_mode: "host"
    pid: "host"
    volumes:
      - "/:/host:ro,rslave"
    command:
      - '--path.rootfs=/host'
    restart: unless-stopped
    expose:
      - "9100"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: "host"
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
    restart: unless-stopped
    expose:
      - "8080"

  prometheus:
    image: prom/prometheus:latest
    network_mode: "host"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "prometheus-data:/prometheus"
    restart: unless-stopped
    expose:
      - "9090"

  grafana:
    image: grafana/grafana:latest
    network_mode: "host"
    volumes:
      - "./grafana.ini:/etc/grafana/grafana.ini"
      - "grafana-storage:/var/lib/grafana"
    restart: unless-stopped
    expose:
      - "3000"

  nginx:
    image: nginx:latest
    network_mode: "host"
    volumes:
      - "./domain.conf:/etc/nginx/conf.d/domain.conf:ro"
    logging:
      driver: "loki"
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
    restart: unless-stopped
    expose:
      - "80"

volumes:
  prometheus-data:
  grafana-storage:

domain.conf

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  listen [::]:80;

  server_name grafana.diegocornejo.com;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}

grafana.ini

[smtp]
enabled = true
host = email-smtp.us-east-1.amazonaws.com:587
user = <username>
password = <password>
skip_verify = false
from_address = grafana@diegocornejo.com
from_name = Grafana

loki-config.yml

# This file has been downloaded from https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'docker'
    static_configs:
      - targets: ['localhost:9323']
  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:8080']