Creating a k8s cluster on Raspberries.

k3s_raspberry_pi_cluster.md

Ubuntu

Flash SD card with Ubuntu image

xzcat ubuntu-19.10.1-preinstalled-server-arm64+raspi3.img.xz | sudo dd of=/dev/disk6 bs=32m`

Boot Raspberry. Headless possibel with Ubuntu image as SSH login enabled by default in image.

Login / Set-up SSH

$ ssh ubuntu@192.168.1.80 - test login via ssh (ubuntu/ubuntu)

$ ssh-copy-id ubuntu@192.168.1.80 - needed for easier access and k3sup

Set hostname

sudo hostnamectl set-hostname k8srollmaster
hostnamectl set-hostname k8srollmaster

Run update/upgrade: sudo apt-get update && sudo apt-get -y upgrade

If you want ifconfig available for checking network interfaces also sudo apt install net-tools required.

Create and enable swap file (optional but recommended though)

Note: Enabling swap on Kubernetes nodes/master is not generally recommended. However, seems to be ok for k3s. See discussion at kubernetes/kubernetes#53533 Verify if swap file or partition exists sudo swapon --show or free -ht. To create:

sudo dd if=/dev/zero of=/swapfile bs=1024 count=524288  _OR_
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Permanently enable edit fstab sudo vi /etc/fstab and add

 /swapfile swap swap defaults 0 0

Verify again with sudo swapon --show or free -ht.

Enabling cgroups in kernel

If not enabled (pre-req for Docker) you'll see errors showing in /var/log/syslog or, respectively when running systemctl status k3s

Dec 26 20:16:34 kuberollmaster systemd[1]: Started Lightweight Kubernetes.
Dec 26 20:16:34 kuberollmaster k3s[2342]: time="2019-12-26T20:16:34.536883337Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"

Even though in 19.10.1 firmware/README it reads

usercfg.txt - the file in which user modified configuration should be placed, included by config.txt

the correct file is the same as in 19.10 i.e. /boot/firmware/nobtcmd.txt to place cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Use cat /proc/cmdline to verify if parameters were passed on to kernel as expected (after reboot). (see also k3s-io/k3s#1078)

New since Ubuntu 21.10

Needs sudo apt install linux-modules-extra-raspi see https://bugs.launchpad.net/bugs/1947628

Run k3sup to setup k8s

k3sup install --ip 192.168.1.80 --user ubuntu Note: IP configs (fixed IP via DHCP MAC address)

• Master: x.x.x.80
• Nodes: x.x.x.81, x.x.x.x.82

To test:

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node

If kubectl is not present install e.g. with brew install kubernetes-cli (MacOS).

Add a worker node (agent) to the k8s setup

See also section "Join some agents to your Kubernetes server" at https://github.com/alexellis/k3sup

k3sup join --ip 192.168.1.81 --server-ip 192.168.1.80 --user ubuntu

Links / Documentation

• Good hardware list / recommendation: https://www.hanselman.com/blog/HowToBuildAKubernetesClusterWithARMRaspberryPiThenRunNETCoreOnOpenFaas.aspx

Alternative base image to try out...(one day)

Hypriot https://blog.hypriot.com/

...HypriotOS, which is a minimal Debian-based operating systems that is optimized to run Docker. It made it dead easy use Docker on any Raspberry Pi ...

k3s_ubuntu_containerd_raspberry_cheatsheet.md

containerd

Delete all non-arm images (i.e. all amd64 only images) on a node.

ctr i ls  | awk  '{print $1, $6}' | grep -v arm | awk 'NR>1 {print $1}' | xargs sudo ctr image remove

List images (name, size, arch)

sudo ctr i ls | awk '{print $1, $4, $5, $6}'

traefik (default ingress controller on k3s) http://www.traefik.io/

Enable traefik dashboard

Edit traefik's configmap via kubectl -n kube-system edit configmap traefik and add [api] \ dashboard = true:

data:
  traefik.toml: |
    # traefik.toml
    logLevel = "info"
    defaultEntryPoints = ["http","https"]
    [api]
      dashbaord = true
    [entryPoints]
      [entryPoints.http]

To connect from localhost kubectl -n kube-system port-forward deployment/traefik 8080 and access at http://localhost:8080/dashboard/

k3s

Storage: nfs PersistentVolume(Claim)

• Setting up an Ubuntu host as nfs server (x.x.x.88)

sudo apt install nfs-common
sudo apt install nfs-kernel-server -y
sudo mkdir /var/nfs
sudo vi /etc/exports #add: /var/nfs    *(rw,sync,no_root_squash,no_subtree_check)
sudo exportfs -a
sudo systemctl restart nfs-kernel-server

nfs-common potentially needed on clients (nodes/master) as well sudo apt install nfs-common -y.

• On Kubernetes install NFS-Client Provisioner. See https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client

helm install stable/nfs-client-provisioner --set nfs.server=x.x.x.88 --set nfs.path=/var/nfs --generate-name --set image.repository=quay.io/external_storage/nfs-client-provisioner-arm
kubectl get storageclass nfs-client -o wide

• Provision/test a PVC pvs-nfs.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs-client"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1G

kubectl apply -f pvc-nfs.yaml
kubectl get persistentvolumes
kubectl get persistentvolumeclaims
kubectl delete -f pvc-nfs.yaml

kubernetes-dashboard

Deploy dashboard on k3s:

Note: Check https://github.com/kubernetes/dashboard/releases for latest release available.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc3/aio/deploy/recommended.yaml

Create token to access / login to dashboard kubectl apply -f dashboard-admin.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin-user
  namespace: kubernetes-dashboard

To get token:

kubectl -n kubernetes-dashboard describe secrets `kubectl -n kubernetes-dashboard get secret | grep dashboard-admin-user | awk '{print $1}'`

Run kubectl proxy and access dashboard via http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Raspberry Pi

Ubuntu

Edits/additions are in file /boot/firmware/config.txt

• Running headless gpu memory can be restricted to minimum (more available for arm cpu...)

gpu_mem=16

• Overclocking

arm_freq=1800
over_voltage=2

Temperature on Raspberry Pi (Ubuntu)

Note: vcgencmd as known from Raspian not available per default on Ubuntu.

    alias temperature='echo "scale=2; $(cat /sys/class/thermal/thermal_zone0/temp)/1000" | bc'

#!/bin/bash
while [ true ] ;
        FREQ=`cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq`;
        TEMP=`cat /sys/class/thermal/thermal_zone0/temp`;
        TEMP_IN_C=$(($TEMP/1000))
        echo `date "+%H:%M:%S"`
        echo "CPUFrequency=$FREQ";
        echo "Temparature=$TEMP_IN_C""'C" "  ("$TEMP")"
        echo "";
        do sleep 10;
done