Skip to content

Instantly share code, notes, and snippets.

@jannegpriv
Last active October 14, 2023 19:37
Show Gist options
  • Save jannegpriv/eec6941a06094157c4d6ac533c54c1d2 to your computer and use it in GitHub Desktop.
Save jannegpriv/eec6941a06094157c4d6ac533c54c1d2 to your computer and use it in GitHub Desktop.
InfluxDb and Grafana in K3s

InfluxDb and Grafana in K3s

The installation steps requires that you have configured NFS client as described in: https://gist.github.com/jannegpriv/f78b59d93c7a815b1fbcb940277bb762

The steps below are based on information in this article.

InfluxDb

Check for the latest official docker image for InfluxDb on DockerHub. Create an initial deployment by issuing:

k create deployment influxdb --image=docker.io/influxdb:1.7.8

The image should now be pulled from DockerHub, can take a while. You can check that the pod is running OK with:

k get pods

and check the pulling status via:

k describe pod <id of pod>

When the deployment/pod is running we will get the deployment yaml-file by issuing:

k get deployment influxdb -o yaml > influxdb-deployment.yaml

Generate secret for InfluxDb credentials:

k create secret generic influxdb-creds \
  --from-literal=INFLUXDB_DATABASE=twittergraph \
  --from-literal=INFLUXDB_USERNAME=root \
  --from-literal=INFLUXDB_PASSWORD=root \
  --from-literal=INFLUXDB_HOST=influxdb

Get and describe influxdb-creds:

# Get secret
get secret influxdb-creds
NAME             TYPE     DATA   AGE
influxdb-creds   Opaque   4      47s

# Describe secret
k describe secret influxdb-creds
Name:         influxdb-creds
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
INFLUXDB_DATABASE:  12 bytes
INFLUXDB_HOST:      8 bytes
INFLUXDB_PASSWORD:  4 bytes
INFLUXDB_USERNAME:  4 bytes

Trim the influxdb-deployment.yaml and add the following:

spec:
      containers:
      - image: docker.io/influxdb:1.7.8
        imagePullPolicy: IfNotPresent
        name: influxdb
        envFrom:
        - secretRef:
           name: influxdb-creds

The influxdb-deployment.yaml should then contain:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: influxdb
  name: influxdb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: influxdb
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: influxdb
    spec:
      containers:
      - image: docker.io/influxdb:1.7.8
        imagePullPolicy: IfNotPresent
        name: influxdb
        envFrom:
        - secretRef:
           name: influxdb-creds
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

Then re-create the influxdb deployment:

k delete deployment influxdb
k apply -f influxdb-deployment.yaml 

# Describe the influxdb deployment, look for Environment Variables From:
k describe deployment influxdb
Name:                   influxdb
Namespace:              default
CreationTimestamp:      Mon, 30 Sep 2019 10:17:58 +0100
Labels:                 app=influxdb
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"influxdb"},"name":"influxdb","namespa...
Selector:               app=influxdb
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=influxdb
  Containers:
   influxdb:
    Image:      docker.io/influxdb:1.7.8
    Port:       <none>
    Host Port:  <none>
    Environment Variables from:
      influxdb-creds  Secret  Optional: false
    Environment:      <none>
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   influxdb-7d7c99555c (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  8m2s  deployment-controller  Scaled up replica set influxdb-7d7c99555c to 1

We need persistent storage for InfluxDb, create a influxdb_claim.yaml file with the following content:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: influxdb-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 8Gi

If you need more than 8GB storage, then change storage accordingly. Apply the PVC claim:

k apply -f influxdb-claim.yaml

Log into K3s dashboard and check for it! It should say Status: Bound if everything has gone well.

Now we must adapt the influxdb-deployment.yaml so that InfluxDb uses our PV:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: influxdb
  name: influxdb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: influxdb
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: influxdb
    spec:
      containers:
      - image: docker.io/influxdb:1.7.8
        imagePullPolicy: IfNotPresent
        name: influxdb
        volumeMounts:
        - mountPath: /var/lib/influxdb
          name: var-lib-influxdb
        envFrom:
        - secretRef:
           name: influxdb-creds
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30  
      volumes:
      - name: var-lib-influxdb
        persistentVolumeClaim:
          claimName: influxdb

NOTE: The claimName must match the name in the influxdb-claim.yaml file.

Re-apply the influxdb deployment to get PVs on NFS:

k apply -f influxdb-deployment.yaml

Log into K3s dashboard and check that InfluxDb deployment/pod now uses PVs.

To let Grafana "speak" to InfluxDb we need to create a Service that exposes InfluxDb pod to the cluster by issuing:

k expose deployment influxdb --port=8086 --target-port=8086 --protocol=TCP --type=ClusterIP

Get info about the svc:

# Get status of svc
k get svc influxdb
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
influxdb   ClusterIP   10.43.239.12   <none>        8086/TCP   4m39s

# Describe svc
k describe svc influxdb
Name:              influxdb
Namespace:         default
Labels:            app=influxdb
Annotations:       <none>
Selector:          app=influxdb
Type:              ClusterIP
IP:                10.43.239.12
Port:              <unset>  8086/TCP
TargetPort:        8086/TCP
Endpoints:         10.42.3.32:8086
Session Affinity:  None
Events:            <none>

Finally we will export the service to a file named influxdb-service.yaml:

k get svc influxdb -o yaml > influxdb-service.yaml

After trimming it looks like:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: influxdb
  name: influxdb
spec:
  ports:
  - port: 8086
    protocol: TCP
    targetPort: 8086
  selector:
    app: influxdb

Grafana

Check for the latest official docker image for Grafana on DockerHub. Create an initial deployment by issuing:

k create deployment grafana --image=docker.io/grafana/grafana:6.3.6

The image should now be pulled from DockerHub, can take a while. You can check that the pod is running OK with:

k get pods

and check the pulling status via:

k describe pod <id of pod>

When the deployment/pod is running we will get the deployment yaml-file by issuing:

k get deployment grafana -o yaml > grafana-deployment.yaml

Generate secret for Grafana credentials:

k create secret generic grafana-creds \
  --from-literal=GF_SECURITY_ADMIN_USER=admin \
  --from-literal=GF_SECURITY_ADMIN_PASSWORD=grafana

Get and describe grafana-creds:

# Get secret
k get secret grafana-creds
NAME            TYPE     DATA   AGE
grafana-creds   Opaque   2      3s

# Describe secret
k describe secret grafana-creds
Name:         grafana-creds
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
GF_SECURITY_ADMIN_PASSWORD:  7 bytes
GF_SECURITY_ADMIN_USER:      5 bytes

Trim the grafana-deployment.yaml and add the following:

spec:
      containers:
      - image: docker.io/grafana/grafana:6.3.6
        imagePullPolicy: IfNotPresent
        name: grafana
        envFrom:
        - secretRef:
            name: grafana-creds

The grafana-deployment.yaml should then contain:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: docker.io/grafana/grafana:6.3.6
        imagePullPolicy: IfNotPresent
        name: grafana
        envFrom:
        - secretRef:
            name: grafana-creds
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

Then re-create the grafana deployment:

k delete deployment grafana
k apply -f grafana-deployment.yaml 

# Describe the grafana deployment, look for Environment Variables From:
k describe deployment grafana
Name:                   grafana
Namespace:              default
CreationTimestamp:      Mon, 30 Sep 2019 13:29:11 +0100
Labels:                 app=grafana
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace...
Selector:               app=grafana
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=grafana
  Containers:
   grafana:
    Image:      docker.io/grafana/grafana:6.3.6
    Port:       <none>
    Host Port:  <none>
    Environment Variables from:
      grafana-creds  Secret  Optional: false
    Environment:     <none>
    Mounts:          <none>
  Volumes:           <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
OldReplicaSets:  <none>
NewReplicaSet:   grafana-5d6745f559 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  20s   deployment-controller  Scaled up replica set grafana-5d6745f559 to 1

Grafana needs some configuration files, at least it needs to know how to commmunicate with InfluxDb. That information can be stored locally in a yaml-file that via Kubernetes ConfigMap can be added to a deployment/pod. NOTE:: The Service name influxdb is reachable via DNS if you have DNS running, can be checked via:

k get services kube-dns --namespace=kube-system

Create a influxdb-datasource.yaml file with the following content:

# config file version
apiVersion: 1

# list of datasources to insert/update depending
# what's available in the database
datasources:
  # <string, required> name of the datasource. Required
- name: influxdb
  # <string, required> datasource type. Required
  type: influxdb
  # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
  access: proxy
  # <int> org id. will default to orgId 1 if not specified
  orgId: 1
  # <string> url
  url: http://influxdb:8086
  # <string> database password, if used
  password: root
  # <string> database user, if used
  user: root
  # <string> database name, if used
  database: influxdb
  # version
  version: 1
  # <bool> allow users to edit datasources from the UI.
  editable: true

NOTE: The url contains hostname influxdb that will point to the InfluxDb Service via DNS.

From the same directory as where the influxdb-datasource.yaml is stored, issue:

kubectl create configmap grafana-config --from-file=influxdb-datasource.yml=influxdb-datasource.yml

to create a ConfigMap. Describe the ConfigMap grafana-config:

k describe configmap grafana-config
ame:         grafana-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
influxdb-datasource.yml:
----
# config file version
apiVersion: 1

# list of datasources to insert/update depending
# what's available in the database
datasources:
  # <string, required> name of the datasource. Required
- name: influxdb
  # <string, required> datasource type. Required
  type: influxdb
  # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
  access: proxy
  # <int> org id. will default to orgId 1 if not specified
  orgId: 1
  # <string> url
  url: http://influxdb:8086
  # <string> database password, if used
  password: root
  # <string> database user, if used
  user: root
  # <string> database name, if used
  database: influxdb
  # version
  version: 1
  # <bool> allow users to edit datasources from the UI.
  editable: true


Events:  <none>

Now we need to update the grafana.deployment.yaml file with information of the configmap. Add the following to .spec.template.spec.volumes:

spec:
  template:
    spec:
      volumes:
        - name: grafana-config
          configMap:
            name: grafana-config

and the following to .spec.template.spec.containers:

spec:
  template:
    spec:
      containers:
      - name: grafana
        volumeMounts:
        - mountPath: /etc/grafana/provisioning/datasources/influxdb-datasource.yml
          name: grafana-config
          readOnly: true
          subPath: influxdb-datasource.yml

The content of the grafana-deployment.yamlfile should look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: docker.io/grafana/grafana:6.3.6
        imagePullPolicy: IfNotPresent
        name: grafana
        envFrom:
        - secretRef:
            name: grafana-creds
        volumeMounts:
        - mountPath: /etc/grafana/provisioning/datasources/influxdb-datasource.yml
          name: grafana-config
          readOnly: true
          subPath: influxdb-datasource.yml
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: grafana-config
          configMap:
            name: grafana-config

Re-apply the grafana deployment:

k apply -f grafana-deployment.yaml

When the pod is running, check that the /etc/grafana/provisioning/datasources/influxdb-datasource.yml on the pod contains the correct info NOTE Exchange your grafana pod id in the command below (k get pods):

k exec -it grafana-b84d897fb-mj5fc cat /etc/grafana/provisioning/datasources/influxdb-datasource.yml
# config file version
apiVersion: 1

# list of datasources to insert/update depending
# what's available in the database
datasources:
  # <string, required> name of the datasource. Required
- name: influxdb
  # <string, required> datasource type. Required
  type: influxdb
  # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
  access: proxy
  # <int> org id. will default to orgId 1 if not specified
  orgId: 1
  # <string> url
  url: http://influxdb:8086
  # <string> database password, if used
  password: root
  # <string> database user, if used
  user: root
  # <string> database name, if used
  database: influxdb
  # version
  version: 1
  # <bool> allow users to edit datasources from the UI.
  editable: true

To be able to connect to Grafana in a web browser, we need to expose a Service and an Ingress:

First create the file grafana-service.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    app: grafana
  ports:
    - port: 3000
      protocol: TCP
      targetPort: 3000
      name: grafana-http

and then the file grafana-traefik.yaml with the following content:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: openhab
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: k3s-grafana.example.org
    http:
      paths:
      - path: /
        backend:
          serviceName: grafana
          servicePort: grafana-http

Apply both files:

k apply -f grafana-service.yaml
k apply -f grafana-traefik.yaml

Then add k3s-grafana.example.org to /etc/hosts on your local computer, the surf to: http://k3s-grafana.example.org

Log in as admin/grafana as configured earlier.

NOTE: The database is not yet created on InfluxDb, so the DS will fail to connect. Log in to the InfuxDb pod and create the database:

# Log in to pod with bash (NOTE: change to your specific pod id
k exec -it influxdb-7989845b67-29h8g /bin/bash

# Login to influxdb cmdline
root@influxdb-7989845b67-29h8g:/# influx
Connected to http://localhost:8086 version 1.7.8
InfluxDB shell version: 1.7.8
> show databases;
name: databases
name
----
_internal
> 
> create database influxdb
> show databases
name: databases
name
----
_internal
influxdb
> exit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment