Installation steps for running K3s on a Raspberry PI cluster

installation_steps_k3s_on_rpi.md

Installation steps for running K3s on a Raspberry PI cluster

Install latest Raspbian by following these instructions:

https://desertbot.io/blog/headless-raspberry-pi-3-bplus-ssh-wifi-setup

Installation steps for running K3s on a Raspberry PI cluster. My RPIs:

# Model
pi@k3s-master-1:~ $ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Plus Rev 1.3
pi@k3s-worker-2:~ $ cat /sys/firmware/devicetree/base/model
Raspberry Pi 4 Model B Rev 1.1
pi@k3s-worker-3:~ $ cat /sys/firmware/devicetree/base/model
Raspberry Pi 4 Model B Rev 1.1
# OS version
pi@k3s-master-1:~ $ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@k3s-master-1:~ $ uname -a
Linux k3s-master-1 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

Install latest Rapsbian on all RPIs using standard procedures. Configure password for user pi, change hostname and expand file system using sudo raspi-config. I use the following naming standard for hostnames:

• master: k3s-master-1
• worker: k3s-worker-1, k3s-worker-2, k3s-worker-3

Add hostnames to /etc/hosts on all RPIs.

Update to latest OS version on all RPIs:

sudo apt update && sudo apt dist-upgrade
sudo apt clean
sudo reboot

Currently (20190927) on there is a bug with incompatibility issues between Kube proxy and IP tables v1.8. kubernetes/kubernetes#71305 Therefor to get networking to work the following work-around is needed on all RPIs in the cluster (it will configure legacy mode for IP tables):

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy 
sudo sysctl net.bridge.bridge-nf-call-iptables=1 
# Check version
iptables -V 
iptables v1.8.2 (legacy) 

Turm off swap:

sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove

We also need to configure Rasbian kernel to support container features, edit /boot/cmdline.txt on all RPIs and add the following to the end of the line:

 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Then reboot all RPIs again.

K3s uses K3d (containerd) as default container runtime (use can use Docker with the --docker swtich), but since the my applications uses Docker we will install the Docker daemon and docker-compose on the master node:

# Install Docker:
curl -sSL https://get.docker.com | sh
# Install docker-compose via pip, it takes a while
sudo apt-get install libffi-dev libssl-dev
sudo apt install python3-dev
sudo apt-get install -y python3 python3-pip
With pip3 and python3 installed, it’s possible to install Docker-Compose via the command;

‍sudo pip3 install docker-compose   

Currently (20190927) there is a bug with docker push: docker/for-linux#711

To fix that and also to prepare for having a local docker registry we'll add the following lines to /etc/docker/daemon.json (NOTE: If the daemon.json does not exist, then create it. Change to another hostname if you do not follow my naming standard):

pi@k3s-worker-2:~ $ cat /etc/docker/daemon.json  

{ 
    "max-concurrent-uploads": 1, 
    "insecure-registries" : ["k3s-master-1:5000"] 
} 

Restart docker-daemon:

sudo systemctl restart docker.service

Time to install K3s, on master node (NOTE:: Instructions tested on K3S version v1.18.9+k3s1)

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Check that the k3s service is running:

sudo systemctl status k3s

Find out K3s master node join code:

sudo cat /var/lib/rancher/k3s/server/node-token

The join code is used to grant access for all worker nodes. On all worker nodes (NOTE: If hostname not present in /etc/hosts, use master node's IP below):

export K3S_URL="https://k3s-master-1:6443"
export K3S_TOKEN="K10e9fd828806f453fa2293e4fa7e5db51696b29565819b2a5a0320c3b620572b01::node:519d266f594a54673f1f52bef47443e1"
curl -sfL https://get.k3s.io | sh -
# Repeat for all working nodes

Optional but convenient, add following aliases to .bashrc on master node, then source via . bashrc:

alias k='kubectl'
alias ks='kubectl -n kube-system'

Set roles 'worker' on the worker nodes:

kubectl label nodes k3s-worker-2 node-role.kubernetes.io/worker=worker
kubectl label nodes k3s-worker-3 node-role.kubernetes.io/worker=worker

Check that master and all worker nodes are running (NOTE: Examples below requires alias being set):

k get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION         INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
k3s-master-1   Ready    master   9h    v1.15.4-k3s.1   192.168.1.150   <none>        Raspbian GNU/Linux 10 (buster)   4.19.66-v7+      containerd://1.2.8-k3s.1
k3s-worker-2   Ready    worker   9h    v1.15.4-k3s.1   192.168.1.161   <none>        Raspbian GNU/Linux 10 (buster)   4.19.66-v7l+     containerd://1.2.8-k3s.1
k3s-worker-3   Ready    worker   9h    v1.15.4-k3s.1   192.168.1.149   <none>        Raspbian GNU/Linux 10 (buster)   4.19.66-v7l+     containerd://1.2.8-k3s.1

If you want to be able to run kubectl from your local computer and not from the cluster master node, then install kubectl on your local computer: https://kubernetes.io/docs/tasks/tools/install-kubectl/

Then create the file ~/.kube/config on your local computer and paste the contents of the file cat /etc/rancher/k3s/k3s.yaml on your master node. NOTE: You need to replace 127.0.0.1 with the IP of your master node.

Then use @ruan's excellent tutorial (skip the docker installation part) to try your new K3s cluster out! :-) https://sysadmins.co.za/develop-build-and-deploy-a-golang-app-to-k3s/