Instantly share code, notes, and snippets.

Embed
What would you like to do?
K8s (v1.10.2) on Raspbian (April 2018)

Moved to https://github.com/aaronkjones/rpi-k8s-node-prep

Kubernetes on (vanilla) Raspbian Lite

Yes - you can create a Kubernetes cluster with Raspberry Pis with the default operating system called Raspbian. This means you can carry on using all the tools and packages you're used to with the officially-supported OS.

Pre-reqs:

  • You must use an RPi 2 or 3 for use with Kubernetes
  • I'm assuming you're using wired ethernet (Wi-Fi also works, but it's not recommended)

Master node setup

  • Flash Raspbian to a fresh SD card.

You can use Etcher.io to burn the SD card.

Before booting set up an empty file called ssh in /boot/ on the SD card.

Use Raspbian Stretch Lite

Update: I previously recommended downloading Raspbian Jessie instead of Stretch. At time of writing (3 Jan 2018) Stretch is now fully compatible.

https://www.raspberrypi.org/downloads/raspbian/

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-master-1 or similar and then reboot.

  • Set a static IP address

It's not fun when your cluste breaks because the IP of your master changed. Let's fix that problem ahead of time:

cat >> /etc/dhcpcd.conf

Paste this block:

profile static_eth0
static ip_address=192.168.0.100/24
static routers=192.168.0.1
static domain_name_servers=8.8.8.8

Hit Control + D.

Change 100 for 101, 102, 103 etc.

You may also need to make a reservation on your router's DHCP table so these addresses don't get given out to other devices on your network.

  • Install Docker

This installs 17.12 or newer.

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
  • Disable swap

For Kubernetes 1.7 and newer you will get an error if swap space is enabled.

Turn off swap:

$ sudo dphys-swapfile swapoff && \
  sudo dphys-swapfile uninstall && \
  sudo update-rc.d dphys-swapfile remove

This should now show no entries:

$ sudo swapon --summary
  • Edit /boot/cmdline.txt

Add this text at the end of the line, but don't create any new lines:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Now reboot - do not skip this step.

  • Add repo lists & install kubeadm
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

To install a later version, remove the version flag at the end (e.g. sudo apt-get install -qy kubeadm) I realise this says 'xenial' in the apt listing, don't worry. It still works.

  • You now have two new commands installed:

  • kubeadm - used to create new clusters or join an existing one

  • kubectl - the CLI administration tool for Kubernetes

  • Modify 10-kubeadm.conf on the master node only

$ sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
  • Initialize your master node:
$ sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

Optionally also pass --apiserver-advertise-address=192.168.0.27 with the IP of the Pi.

Note: This step will take a long time, even up to 15 minutes.

After the init is complete run the snippet given to you on the command-line:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

This step takes the key generated for cluster administration and makes it available in a default location for use with kubectl.

  • Now save your join-token

Your join token is valid for 24 hours, so save it into a text file. Here's an example of mine:

$ kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea
  • Check everything worked:
$ kubectl get pods --namespace=kube-system
NAME                           READY     STATUS    RESTARTS   AGE                
etcd-of-2                      1/1       Running   0          12m                
kube-apiserver-of-2            1/1       Running   2          12m                
kube-controller-manager-of-2   1/1       Running   1          11m                
kube-dns-66ffd5c588-d8292      3/3       Running   0          11m                
kube-proxy-xcj5h               1/1       Running   0          11m                
kube-scheduler-of-2            1/1       Running   0          11m                
weave-net-zz9rz                2/2       Running   0          5m 

You should see the "READY" count showing as 1/1 for all services as above. DNS uses three pods, so you'll see 3/3 for that.

  • Setup networking

Install Flannel network driver

$ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

Join other nodes

On the other RPis, repeat everything apart from kubeadm init.

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-worker-1 or similar and then reboot.

  • Join the cluster

Copy and paste the command provided after Kubeadm init completes. It should look something like this:

$ sudo kubeadm join <master ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:1c06faa186e7f85...

You can now run this on the master:

$ kubectl get nodes
NAME      STATUS     AGE       VERSION
k8s-1     Ready      5m        v1.7.4
k8s-2     Ready      10m       v1.7.4

Deploy a container

This container will expose a HTTP port and convert Markdown to HTML. Just post a body to it via curl - follow the instructions below.

function.yml

apiVersion: v1
kind: Service
metadata:
  name: markdownrender
  labels:
    app: markdownrender
spec:
  type: NodePort
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 31118
  selector:
    app: markdownrender
---
apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: markdownrender
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: markdownrender
    spec:
      containers:
      - name: markdownrender
        image: functions/markdownrender:latest-armhf
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP

Deploy and test:

$ kubectl create -f function.yml

Once the Docker image has been pulled from the hub and the Pod is running you can access it via curl:

$ curl -4 http://127.0.0.1:31118 -d "# test"
<p><h1>test</h1></p>

If you want to call the service from a remote machine such as your laptop then use the IP address of your Kubernetes master node and try the same again.

Start up the dashboard

The dashboard can be useful for visualising the state and health of your system but it does require the equivalent of "root" in the cluster. If you want to proceed you should first run in a ClusterRole from the docs.

echo -n 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system' | kubectl apply -f -

This is the development/alternative dashboard which has TLS disabled and is easier to use.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard-arm.yaml

You can then find the IP and port via kubectl get svc -n kube-system. To access this from your laptop you will need to use kubectl proxy and navigate to http://localhost:8001/ on the master, or tunnel to this address with ssh.

Remove the test deployment

Now on the Kubernetes master remove the test deployment:

$ kubectl delete -f function.yml

Moving on

Now head back over to the tutorial and deploy OpenFaaS to put the cluster through its paces.

#!/bin/sh
# This installs the base instructions up to the point of joining / creating a cluster
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update -q && \
sudo apt-get install -qy kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
echo Adding " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory " to /boot/cmdline.txt
sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"
echo "$orig" | sudo tee /boot/cmdline.txt
echo "Remove "KUBELET_NETWORK_ARGS" from 10-kubeadm.conf on the master node only (use sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf)"
echo "After, please reboot"

Use this to setup quickly

# curl -sL \
 https://gist.githubusercontent.com/aaronkjones/d996f1a441bc80875fd4929866ca65ad/raw/354f2aded07426d3d64f41c2cbcd61a10370c7f7/prep.sh \
 | sudo sh
@FFFEGO

This comment has been minimized.

FFFEGO commented May 12, 2018

Hey, thanks for the guide with flannel.

Loks like sudo is missing here. At least it was required in my case.
$ sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

@FFFEGO

This comment has been minimized.

FFFEGO commented May 12, 2018

Init process is getting stuck with the latest docker (18.05.0\~ce\~3-0\~raspbian) on the following step:

[init] This might take a minute or longer if the control plane images have to be pulled.

Downgrade to 17.12.1\~ce-0\~raspbian helped me.

@aaronkjones

This comment has been minimized.

Owner

aaronkjones commented May 14, 2018

Thanks @FFFEGO, I updated the gist. I believe Docker 18.04.0 worked. I will try it again and see what happens.

@toolboc

This comment has been minimized.

toolboc commented May 19, 2018

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

@toolboc

This comment has been minimized.

toolboc commented May 20, 2018

After following this guide, there is no internet connectivity from within my containers. Any suggestions?

@BathAndy

This comment has been minimized.

BathAndy commented May 20, 2018

Frustrating but fun . I finally have a 6 node raspberry k8s cluster running kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00 and 18.04: and flannel 0.9.1

I have built and rebuilt this cluster because after a powerdown (clean) test it never worked again for me It was fixed with three changes - not sure which one fixed it to be honest and I can't face rebuilding it again to find out

  1. on each node and master - setup for your network address scheme
    sudo vi /etc/network/interfaces.d/eth0

allow-hotplug eth0
auto eth0
iface eth0 inet static
address 192.168.1.62
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1

sudo systemctl disable dhcpcd

and probably the step that made it work is to remember to run
3)
sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # DO this on MASTER NODE ONLY

Previously I had run the sed on each node not just the master , and it would join the cluster fine and then just be flakey, it would always show that all the nodes and kube-system pods were running but after running
kubectl apply -f function.yml

It would never connect on the master - running curl -4 http://localhost:31118 -d '# test' would work on the node it was running on but running the exact same command on the master node would fail

Anyways it finally works - now I can play with actually using it !

@GastonLagaffe2013

This comment has been minimized.

GastonLagaffe2013 commented May 20, 2018

Salut, just spent two days trying to get it to run and had the same issue as @toolboc. Downgrading as described solved my problem as well. I have three brand new Raspberry PI 3B+ using Samsung Evo 32GB Sd card.
Thanks a lot ofr the help - can sleep now again ;)
Ciao, Mathias

@aaronkjones

This comment has been minimized.

Owner

aaronkjones commented May 20, 2018

I had it working successfully with:

  • Raspbian Apr 2018
  • kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
  • Docker 18.04.0-ce
  • Flannel 0.9.1

Right now I have it running:

  • Hypriot 1.9.0
  • kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
  • Docker 18.04.0-ce
  • Flannel 0.9.1
@aaronkjones

This comment has been minimized.

Owner

aaronkjones commented May 21, 2018

I made a more robust script that allows you to select specific versions and pin them.
You can find it here, https://github.com/aaronkjones/rpi-k8s-node-prep

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment