kubernetes-grafana

README.md

A basic example for deploying OLD Grafana with:

• Session storage in Redis (users logged into web UI)
• Dashboard storage in MySQL or MariaDB
• AWS CloudWatch access via grafana-cloudwatch-access secret (should contain .aws/ credentials & config files)
• Basic user authentication. Change this if you want LDAP, Google OAuth, etc...

Note: This was to work around bug grafana/grafana#3824. This stack was for Grafana 3.1.1, and Kubernetes v1.6.4! If I were deploying this into a newer production cluster, I'd use Deployment instead of ReplicationController. NFS still seems to be the most well supported data store supporting multiple pods mounting at the same time.

On the NFS mount, the directory structure used was the following:

/app/grafana-3.1.1/
├── etc
│   └── grafana.ini
├── redis-session-store
│   ├── master-data
│   │   ├── appendonly.aof
│   │   └── appendonly.aof.bak
│   └── var-lib-redis
└── var-lib
    ├── plugins
    ├── plugins-nfs
    │   ├── fetzerch-sunandmoon-datasource
    │   ├── grafana-clock-panel
    │   ├── mtanda-heatmap-epoch-panel
    │   └── raintank-worldping-app
    └── README

11 directories, 4 files

Note, that the only file you need to initialize when starting a fresh stack is the /app/grafana-3.1.1/etc/grafana.ini file. The pods just need the rest of the directory structure to mount to the right places.

grafana-3.1.1-rc.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    app: grafana
  template:
    metadata:
      labels:
        app: grafana
      name: grafana
    spec:
      containers:
      - env:
        - name: GRAFANA_DEFAULT_DASHBOARD
          value: /dashboard/db/home
        - name: GF_PATHS_PLUGINS
          value: /var/lib/grafana/plugins-nfs
        - name: GF_PLUGIN_DIR
          value: /var/lib/grafana/plugins-nfs
        image: grafana/grafana:3.1.1
        imagePullPolicy: IfNotPresent
        name: grafana
        ports:
        - containerPort: 3000
          name: http-grafana
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /etc/grafana
          name: etc-grafana
        - mountPath: /var/lib/grafana
          name: var-lib-grafana
        - mountPath: /usr/share/grafana/.aws
          name: grafana-cloudwatch-access-volume
          readOnly: true
      - command:
        - /usr/local/bin/redis-server
        - /etc/redis/redis.conf
        - --appendonly
        - "yes"
        image: redis
        imagePullPolicy: Always
        name: grafana-redis-master
        ports:
        - containerPort: 6379
          name: redis-server
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /data
          name: redis-master-data
        - mountPath: /var/lib/redis
          name: redis-node-data
        - mountPath: /etc/redis
          name: redis-conf
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-grafana
        nfs:
          path: /app/grafana-3.1.1/etc
          server: PUT_YOUR_NFS_HOST_HERE
      - name: var-lib-grafana
        nfs:
          path: /app/grafana-3.1.1/var-lib
          server: PUT_YOUR_NFS_HOST_HERE
      - name: redis-master-data
        nfs:
          path: /app/grafana-3.1.1/redis-session-store/master-data
          server: PUT_YOUR_NFS_HOST_HERE
      - name: redis-node-data
        nfs:
          path: /app/grafana-3.1.1/redis-session-store/var-lib-redis
          server: PUT_YOUR_NFS_HOST_HERE
      - configMap:
          items:
          - key: redis.conf
            path: redis.conf
          name: redis-conf
        name: redis-conf
      - name: grafana-cloudwatch-access-volume
        secret:
          secretName: grafana-cloudwatch-access

grafana.ini

##################### Grafana Configuration Example #####################
#
# Everything has defaults so you only need to uncomment things you want to
# change

# possible values : production, development
; app_mode = production

# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
; instance_name = ${HOSTNAME}

#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
#
;data = /var/lib/grafana
#
# Directory where grafana can store logs
#
;logs = /var/log/grafana
#
# Directory where grafana will automatically scan and look for plugins
#
;plugins = /var/lib/grafana/plugins

#
#################################### Server ####################################
[server]
# Protocol (http or https)
; protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = grafana.example.com

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url
;root_url = %(protocol)s://%(domain)s:%(http_port)s/

# Log web requests
;router_logging = false

# the path relative working path
;static_root_path = public

# enable gzip
;enable_gzip = false

# https certs & key file
;cert_file =
;cert_key =

#################################### Database ####################################
[database]
# Either "mysql", "postgres" or "sqlite3", it's your choice
type = mysql
host = grafana-storage.example.us-east-1.rds.amazonaws.com:3306
name = grafana
user = grafana_exampleuser
password = PUT_YOUR_MYSQL_PASSWORD_HERE

# For "postgres" only, either "disable", "require" or "verify-full"
;ssl_mode = disable

# For "sqlite3" only, path relative to data_path setting
;path = grafana.db

#################################### Session ####################################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", default is "file"
;provider = file
provider = redis

# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# mysql: go-sql-driver/mysql dsn config string, e.g. `user:password@tcp(127.0.0.1:3306)/database_name`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
;provider_config = sessions
provider_config = addr=127.0.0.1:6379,pool_size=100,db=grafana

# Session cookie name
;cookie_name = grafana_sess

# If you use session in https only, default is false
;cookie_secure = false

# Session life time, default is 86400
;session_life_time = 86400

#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
;reporting_enabled = true

# Set to false to disable all checks to https://grafana.net
# for new vesions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to http://grafana.net to get latest versions
check_for_updates = true

# Google Analytics universal tracking code, only enabled if you specify an id here
;google_analytics_ua_id =

#################################### Security ####################################
[security]
# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = PUT_YOUR_DEFAULT_GRAFANA_UI_ADMIN_PASSWORD_HERE

# used for signing
secret_key = PUT_YOUR_RANDOM_SECRET_KEY_STRING_HERE

# Auto-login remember days
;login_remember_days = 7
;cookie_username = grafana_user
;cookie_remember_name = grafana_remember

# disable gravatar profile images
;disable_gravatar = false

# data source proxy whitelist (ip_or_domain:port separated by spaces)
;data_source_proxy_whitelist =

[snapshots]
# snapshot sharing options
;external_enabled = true
;external_snapshot_url = https://snapshots-origin.raintank.io
;external_snapshot_name = Publish to snapshot.raintank.io

#################################### Users ####################################
[users]
# disable user signup / registration
allow_sign_up = false

# Allow non admin users to create organizations
;allow_org_create = true

# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true

# Default role new users will be automatically assigned (if disabled above is set to true)
;auto_assign_org_role = Viewer

# Background text for the user field on the login page
;login_hint = email or username

# Default UI theme ("dark" or "light")
;default_theme = dark

#################################### Anonymous Auth ##########################
[auth.anonymous]
# enable anonymous access
enabled = false

# specify organization name that should be used for unauthenticated users
org_name = PUT_YOUR_ORG_NAME_HERE

# specify role for unauthenticated users
;org_role = Viewer

#################################### Github Auth ##########################
[auth.github]
;enabled = false
;allow_sign_up = false
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://github.com/login/oauth/authorize
;token_url = https://github.com/login/oauth/access_token
;api_url = https://api.github.com/user
;team_ids =
;allowed_organizations =

#################################### Google Auth ##########################
[auth.google]
;enabled = false
;allow_sign_up = false
;client_id = some_client_id
;client_secret = some_client_secret
;scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
;auth_url = https://accounts.google.com/o/oauth2/auth
;token_url = https://accounts.google.com/o/oauth2/token
;api_url = https://www.googleapis.com/oauth2/v1/userinfo
;allowed_domains =

#################################### Auth Proxy ##########################
[auth.proxy]
;enabled = false
;header_name = X-WEBAUTH-USER
;header_property = username
;auto_sign_up = true

#################################### Basic Auth ##########################
[auth.basic]
;enabled = true

#################################### Auth LDAP ##########################
[auth.ldap]
;enabled = false
;config_file = /etc/grafana/ldap.toml

#################################### SMTP / Emailing ##########################
[smtp]
;enabled = false
;host = localhost:25
;user =
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost

[emails]
;welcome_email_on_sign_up = false

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console, file

# Either "trace", "debug", "info", "warn", "error", "critical", default is "info"
;level = info

# For "console" mode only
[log.console]
;level =

# log line format, valid options are text, console and json
;format = console

# For "file" mode only
[log.file]
;level =

# log line format, valid options are text, console and json
;format = text

# This enables automated log rotate(switch of following options), default is true
;log_rotate = true

# Max line number of single file, default is 1000000
;max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28

# Segment log daily, default is true
;daily_rotate = true

# Expired days of log file(delete after max days), default is 7
;max_days = 7

[log.syslog]
;level =

# log line format, valid options are text, console and json
;format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =

# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =

# Syslog tag. By default, the process' argv[0] is used.
;tag =


#################################### AMQP Event Publisher ##########################
[event_publisher]
;enabled = false
;rabbitmq_url = amqp://localhost/
;exchange = grafana_events

;#################################### Dashboard JSON files ##########################
[dashboards.json]
;enabled = false
;path = /var/lib/grafana/dashboards

#################################### Internal Grafana Metrics ##########################
# Metrics available at HTTP API Url /api/metrics
[metrics]
# Disable / Enable internal metrics
enabled           = true

# Publish interval
;interval_seconds  = 10

# Send internal metrics to Graphite
[metrics.graphite]
address = metrics.example.com:2003
prefix = server.prod.grafana.%(instance_name)s.

#################################### Internal Grafana Metrics ##########################
# Url used to to import dashboards directly from Grafana.net
[grafana_net]
url = https://grafana.net

recover-grafana-redis.yaml

apiVersion: v1
kind: Pod
metadata:
  name: redis-recovery
  namespace: default
  labels:
    component: redis-recovery
spec:
  containers:
  - command:
    - redis-check-aof
    - --fix
    - "/data/appendonly.aof"
    image: redis
    imagePullPolicy: Always
    name: redis-recovery
    volumeMounts:
    - mountPath: /data
      name: redis-master-data
    - mountPath: /var/lib/redis
      name: redis-node-data
    - mountPath: /etc/redis
      name: redis-conf
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  volumes:
  - name: redis-master-data
    nfs:
      path: /app/grafana-3.1.1/redis-session-store/master-data
      server: PUT_YOUR_NFS_HOST_HERE
  - name: redis-node-data
    nfs:
      path: /app/grafana-3.1.1/redis-session-store/var-lib-redis
      server: PUT_YOUR_NFS_HOST_HERE
  - configMap:
      items:
      - key: redis.conf
        path: redis.conf
      name: redis-conf
    name: redis-conf

redis-conf-cm.yaml

kind: ConfigMap
metadata:
  name: redis-conf
apiVersion: v1
data:
  redis.conf: |
    appendonly yes
    #cluster-enabled yes
    #cluster-config-file /var/lib/redis/nodes.conf
    #cluster-node-timeout 5000
    maxmemory-policy allkeys-lru
    maxmemory-samples 10
    maxmemory 80mb
    dir /data
    port 6379