Set up k3s for kubernetes on Ubuntu on Raspberry Pi 4

k3s-on-rpi4-ubuntu.md

Steps for setting up k3s on Ubuntu 20.04.2 on Raspberry Pi 4 Cluster

We use three Raspberry Pi 4's (RPI4's) for the setup. One of them is used as the control plane, and the other two are used as workers.

1. On all three RPI4's, snstall Ubuntu server 20.04.

   • Follow these instructions
   • Assign static IPs to these RPI4s (via router's DHCP settings), call them rpi4-k8s-ctrl, rpi4-k8s-wkr00, and rpi4-k8s-wkr01, respectively, and add them to /etc/hosts for all three RPI4s as well as the PC used for the setup

2. On all three RPI4's, enable cgroup (reference)

   • Add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to the end of /boot/firmware/cmdline.txt, and reboot

3. On all three RPI4's, install docker (reference)

   • Install docker

     $ curl -fsSL https://get.docker.com -o get-docker.sh
     $ sudo sh get-docker.sh   
     

   • Rootless setup

     $ sudo sh -eux <<EOF
     # Install newuidmap & newgidmap binaries
     apt-get install -y uidmap
     EOF
     $ dockerd-rootless-setuptool.sh install
     

   • Add the following lines to the end of ~/.bashrc

     # docker
     export PATH=/usr/bin:$PATH
     export DOCKER_HOST=unix:///run/user/1000/docker.sock
     

4. Install k3s on control plane node rpi4-k8s-ctrl (reference)

   • One-click installation

     $ curl -sfL https://get.k3s.io | sh -s - --docker
     

   • Check installation

     $ sudo systemctl status k3s
     $ sudo kubectl get nodes -o wide
     

   • Configure firewall (reference)

     $ sudo ufw allow 6443/tcp
     $ sudo ufw allow 443/tcp
     

   • Extract node token from the control plane node for worker setup later

     $ sudo cat sudo cat /var/lib/rancher/k3s/server/node-token
     

5. Install k3s on both worker nodes, rpi4-k8s-wkr00 and rpi4-k8s-wkr01(reference)

   • One-click installation, with information for control plane node

      # Adapt K3S_URL and K3S_TOKEN to your control plane node's hostname/IP and node token, respectively
      $ curl -sfL https://get.k3s.io | K3S_URL=https://rpi4-k8s-ctrl:6443 K3S_TOKEN=<NODE-TOKEN> sh -s - --docker
     

   • Check installation

     On worker node

     $ sudo systemctl status k3s-agent
     

     On control plane node

     $ sudo kubectl get nodes
     # Should see something like below
     NAME                 STATUS   ROLES                  AGE   VERSION
     rpi4-k8s-wkr00   Ready    <none>                 14m   v1.20.7+k3s1
     rpi4-k8s-ctrl    Ready    control-plane,master   8d    v1.20.7+k3s1
     rpi4-k8s-wkr01   Ready    <none>                 17s   v1.20.7+k3s1
     

6. On PC (localhost), install kubectl, by following the official documentation. Copy content of rpi4-k8s-ctrl:/etc/rancher/k3s/k3s.yaml to localhost:~/.kube/config, and change server: https://127.0.0.1:6443 to server: https://rpi4-k8s-ctrl:6443. Check installation with kubectl get nodes on local PC.

7. On local PC, deploy a microservice, openfaas-figlet (reference).

   • Create a service

     cat <<EOF > openfaas-figlet-svc.yaml
     apiVersion: v1
     kind: Service
     metadata:
       name: openfaas-figlet
       labels:
         app: openfaas-figlet
     spec:
       type: NodePort
       ports:
         - port: 8080
           protocol: TCP
           targetPort: 8080
           nodePort: 31111
       selector:
         app: openfaas-figlet
     EOF
     

   • Create a deployment

     cat <<EOF > openfaas-figlet-dep.yaml
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: openfaas-figlet
       labels:
        app: openfaas-figlet
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: openfaas-figlet
       template:
         metadata:
           labels:
             app: openfaas-figlet
         spec:
           containers:
           - name: openfaas-figlet
             image: functions/figlet:latest-armhf
             imagePullPolicy: Always
             ports:
             - containerPort: 8080
               protocol: TCP
     EOF
     

   • Apply the configuration

     $ kubectl apply -f openfaas-figlet-dep.yaml,openfaas-figlet-svc.yaml
     deployment.apps/openfaas-figlet created
     service/openfaas-figlet created
     # Check status
     $ kubectl rollout status deploy/openfaas-figlet
     kubectl rollout status deploy/openfaas-figlet
     

   • Invoke the function

     $ echo -n "Hello SN" | curl --data-binary @- http://rpi4-k8s-ctrl:31111
      _   _      _ _         ____  _   _ 
     | | | | ___| | | ___   / ___|| \ | |
     | |_| |/ _ \ | |/ _ \  \___ \|  \| |
     |  _  |  __/ | | (_) |  ___) | |\  |
     |_| |_|\___|_|_|\___/  |____/|_| \_|
     
     

   • Scale up (or down)

     $ kubectl scale deploy/openfaas-figlet --replicas=3
     

   • Check pods info

     $ kubectl get pods -o wide
     NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE                  NOMINATED NODE   READINESS GATES
     openfaas-figlet-77cb68d644-kchcr   1/1     Running   0          60m   10.42.1.3    rpi4-k8s-wkr00   <none>           <none>
     openfaas-figlet-77cb68d644-kbssv   1/1     Running   0          60m   10.42.0.20   rpi4-k8s-ctrl    <none>           <none>
     openfaas-figlet-77cb68d644-2c55b   1/1     Running   0          60m   10.42.1.4    rpi4-k8s-wkr00   <none>           <none>
     

   • Get logs for all pods

     $ kubectl logs -f -l app=openfaas-figlet --all-containers