K8s on Raspbian

k8s-pi.md

This guide has moved to a GitHub repository to enable collaboration and community input via pull-requests.

https://github.com/alexellis/k8s-on-raspbian

Alex

prep.sh

#!/bin/sh

# This installs the base instructions up to the point of joining / creating a cluster

curl -sSL get.docker.com | sh && \
  sudo usermod pi -aG docker

sudo dphys-swapfile swapoff && \
  sudo dphys-swapfile uninstall && \
  sudo update-rc.d dphys-swapfile remove

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubeadm
  
echo Adding " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory" to /boot/cmdline.txt

sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"
echo $orig | sudo tee /boot/cmdline.txt

echo Please reboot

quick.md

Use this to setup quickly

# curl -sL \
 https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/b04f1e9250c61a8ff554bfe3475b6dd050062484/prep.sh \
 | sudo sh

This is great. It'd be very cool to have this operate unattended. (or as unattended as possible)

That dashboard is a 404. Should it be https://rawgit.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard-arm.yaml

The swapfile turns back on when you reboot unless you

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

For this line curl localhost:31118 -d "# test" I had to use the full host name. Localhost is still 127.0.0.1 and it doesn't seem to be listening

Kubernetes please stop changing every other day 👎

I followed the instructions and got everything installed on a 2x Raspberry PI 3 cluster (1 master and 1 node). But, I have not been able to get the Dashboard up and running.

olavt@k8s-master-1:~ $ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 5h
kubernetes-dashboard ClusterIP 10.104.85.132 443/TCP 4h
olavt@k8s-master-1:~ $ kubectl proxy
Starting to serve on 127.0.0.1:8001

What is the Url I should use from my other computer to connect to the Dashboard?

OK for the dashboard you need to run kubectl on your own PC/laptop. Maybe an SSH tunnel would work?

ssh -L 8001:127.0.01:8001 pi@k8s-master-1.local

then try 127.0.0.1:8001 on your local machine

That didn't work for me.

First of all thanks for the detailed setup process.

After updating raspbian i ran into the problem that sudo kubeadm join raised the error CGROUPS_MEMORY: missing. The boot option is no longer cgroup_enable=memory but cgroup_memory=1

See https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086#p57035 and raspberrypi/linux@ba742b5

after installation the status of all pods in namespace kube-system is pending except kube-proxy (NodeLost). Any ideas?
Using docker 17.10 and K8S 1.8.2

My dashboard wouldn't work properly until I did:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccou nt=kube-system:kubernetes-dashboard

I could get to the dashboard using kubectl proxy and opened the url http://localhost:8001/ui in a browser, but it couldn't get any data from the api.

@alexellis it should be cgroup_memory=1 not cgroup_enable=memory

cgroup_enable=memory seems to be fine under kernel 4.9.35-v7.

I've updated the instructions for the newer RPi kernel.

I had to run the "set up networking" step (install weave) in order to get "Running" back from the 3 DNS pods. Before that, they reported "Pending"... move the "set up networking" step before "check everything worked" in your instructions?

I was also only able to get both Master and 1 "slave" node to the Ready status when I first installed the "weave" networking on the master, and only after that joined the worker. K8s version 1.9.

Has anyone experienced an issue kubeadm? I'm getting Illegal instruction when I try to run it.

Running on Raspian Stretch 4.9.59+.

@caedev - no, you are definitely using a Raspberry Pi 2 or 3?

Sorry, just realised I was ssh'ing into the wrong pi; this works absolutely fine on my Pi 2. Thanks for writing this @alexellis - much appreciated.

same experience as @charliesolomon, DNS doesn't come up until you install the weave network driver.

Basically change to below:

• Install network driver kubectl apply -f https://git.io/weave-kube-1.6
• Check status: kubectl get pods --namespace=kube-system

Note: Be patient on the 2nd step, the weave driver comes up first. Once it is Running DNS goes from Pending to ContainerCreating to Running.

In the dashboard section, you might want to mention the need for rbac: https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

An excellent guide, thank you!

The instructions are unclear for accessing the cluster remotely but are explained here:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master

Effectively make a copy on the local machine of the master's /etc/kubernetes/admin.conf perhaps named k8s_pi.conf

Then kubectl --kubeconfig ./k8s_pi.conf get nodes

Or, per your example to create a proxy: kubectl --kubeconfig ./k8s_pi.conf proxy &

To avoid specifying --kubeconfig repeatedly, you can merge the contents of k8s_pi.conf into the default config ~/.kube/config

Follow-up (kubeadm) question: What's the process to shutdown and restart the cluster?

kubeadm reset seems more of a teardown.

What if you'd just like to shut the cluster down correctly to then shutdown the underlying Pis and restart subsequently?

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

• Edit kubernetes-dashboard service.

$ kubectl -n kube-system edit service kubernetes-dashboard

• You should the see yaml representation of the service. Change type: ClusterIP to type: NodePort and save file.
• Check port on which Dashboard was exposed.

$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m

• Create a proxy to view within your browser

$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

• Browse localhost:8001

Thanks again Alex!

Hi, Alex, thank for share this tutorial. I builded a raspberry pi cluster and is running kubernetes and OpenFaas as expected it. the only thing is that the auto-scaling don't in OpenfaaS does work! on my computer works but it does work in the cluster!

Do I have to change something in the .yml files? I check them but they look the same.

I had to add both cgroup_memory=memory AND cgroup_memory=1 to the cmdline.txt to get it to work.

Great and very understandable post !!
I've set the kubernetes dashboard through the Nodeport and can access it on my host but the certificates still give a lot of issues.
Is it possible to use Let's encrypt for the kubernetes dashboard ?
As i am new to the concept of certificates through websites can anyone point me how i can do this through an authomatic service like let's encrypt.

Thanks for the fantastic guide, I had great fun learning about all these topics in practice over a weekend. As a switch I'm having great success with the 5-port D-Link DGS-1005D, newer versions of which use mini-USB for power.

I had issues getting Weave to work on Raspbian Stretch and the Pi3 B+. Shortly after running
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
the master and connected nodes would reboot unexpectedly, and would leave the cluster in an error state.
I ended up using flannel:

• Use --pod-network-cidr=10.244.0.0/16 when initializing the cluster
  $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=<internal master ip> --pod-network-cidr=10.244.0.0/16
• Install flannel with
  $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

I also managed to set up the master as a router, with Wifi on the WAN side, using the steps in this particular post https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=132674&start=50#p1252309

Thanks @Jickelsen I had to do the same.
In addition to that I also my nodes stuck in a not ready state due to the following error:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Fixed this by removing KUBELET_NETWORK_ARGS from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf then rebooting according to this issue: kubernetes/kubernetes#38653

I was then able to run
curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
without issue.

I can't seem to get init past [init] This might take a minute or longer if the control plane images have to be pulled.. There's so many issues on kubernetes/kubadm about this. I used a fresh install of rasbian lite (march 3rd update). Anyone else get this or know a workaround?

Thank You Alex. Very detailed steps. I am using a b plus Pi as a master. Any idea why the Pi goes dead slow on initiating the Kube master.

Thanks @Jickelsen & @DerfOh! I spent all my spare time in the last three weeks trying to get kubernetes to work again. The gist worked great at Xmas but now once you get weavenet up on the node & synced to the master, both crash with an oops:
kernel:[ 4286.584219] Internal error: Oops: 80000007 [#1] SMP ARM
kernel:[ 4287.037510] Process weaver (pid: 13327, stack limit = 0x9bb12210)
kernel:[ 4287.059886] Stack: (0x9bb139f0 to 0x9bb14000)
kernel:[ 4287.081698] 39e0: 00000000 00000000 5001a8c0 9bb13a88
kernel:[ 4287.125181] 3a00: 0000801a 0000db84 9bab4150 9bab4118 9bb13d2c 7f63bad0 00000001 9bb13a5c
Finally I can finish writing my ansible play-book to automate the whole thing.

I've had strange issues with getting weavenet running

NAMESPACE     NAME                                      READY     STATUS              RESTARTS   AGE
kube-system   weave-net-8t7zd                           2/2       Running             494        1d
kube-system   weave-net-gpcnj                           1/2       CrashLoopBackOff    417        1d
kube-system   weave-net-m7tnn                           1/2       ImageInspectError   0          1d
kube-system   weave-net-qmjwk                           1/2       ImageInspectError   0          1d
kube-system   weave-net-rvwpj                           2/2       Running             534        1d

Still debuging it but it has been a fun learning experience getting K8s running on a Raspberry Pi cluster.

@micedwards, I ended up writting an ansible playbook as kept rebuilding my cluster to see why weave kept crashing. Wrote it after running kubeadm reset on the master accidently or on a node. Now have a playbook that sets up my cluster and adds nodes to it as well. Any improvements would be great, https://github.com/carlosroman/ansible-k8s-raspberry-playbook.

Good Evening.

I have been having trouble getting kubernetes+docker running as a 2 RPI cluster. My master node continues to reboot. I followed all the steps above to configure two fresh nodes, except I used my router to establish a static IP for my master and worker node. Interestingly my worker node seems stable so far right now. In previous attempts, when I had set up 4 additional nodes they too became unstable.
The master node was stable before I joined my first worker node

Docker version: 18.03.0-ce, build 0520e24
Kubernetes version : 1.10

Master node:

pi@k8boss1:~ $ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE
etcd-k8boss1 1/1 Running 33 1d
kube-apiserver-k8boss1 1/1 Running 34 1d
kube-controller-manager-k8boss1 1/1 Running 34 1d
kube-dns-686d6fb9c-hwxxw 0/3 Error 0 1d
kube-proxy-8v8z7 0/1 Error 33 1d
kube-proxy-dgqxp 1/1 Running 0 1h
kube-scheduler-k8boss1 1/1 Running 34 1d
weave-net-ggxwp 2/2 Running 0 1h
weave-net-l7xsl 0/2 Error 71 1d

pi@k8boss1:~ $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8boss1 Ready master 1d v1.10.0
k8worker1 Ready 1h v1.10.0

pi@k8boss1:~ $ uptime
01:48:50 up 0 min, 1 user, load average: 1.37, 0.41, 0.14
pi@k8boss1:~ $

Worker:
pi@k8worker1:~ $ uptime
01:49:35 up 1:58, 1 user, load average: 0.11, 0.21, 0.19
pi@k8worker1:~ $

Any thoughts?

On Raspbian Stretch Lite, the installation halts during the master setup phase (sudo kubeadm init --token-ttl=0) with the following output:

[init] This might take a minute or longer if the control plane images have to be pulled.

I found it necessary to install Kubernetes 1.9.6:

sudo apt-get install -y kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

Took 552.013509 seconds to complete, but it's up and running now!

Thanks for a great tutorial!

I am running into the same problems as @carlosroman and @micedwards after applying weave on a 4 RPi 3 cluster:

Raspbian GNU/Linux 9 (stretch)
Docker version 18.04.0-ce, build 3d479c0
Kubernetes v1.10.1

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS              RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running             22         10h
kube-apiserver-k8s-master-1            1/1       Running             39         10h
kube-controller-manager-k8s-master-1   1/1       Running             13         10h
kube-dns-686d6fb9c-qn2mp               0/3       Pending             0          10h
kube-proxy-6dlz4                       1/1       Running             11         9h
kube-proxy-7s977                       1/1       Running             2          9h
kube-proxy-q7jlh                       1/1       Running             11         10h
kube-proxy-qdmp7                       1/1       Running             2          9h
kube-scheduler-k8s-master-1            1/1       Running             13         10h
weave-net-5scxb                        2/2       Running             1          2m
weave-net-5vxzw                        1/2       CrashLoopBackOff    4          2m
weave-net-jmlzc                        1/2       ImageInspectError   0          2m
weave-net-xc2f8                        1/2       ImageInspectError   1          2m
pi@k8s-master-1:~ $
Message from syslogd@k8s-master-1 at Apr 22 08:04:14 ...
 kernel:[  155.252476] Internal error: Oops: 80000007 [#1] SMP ARM

I am having more luck with flannel

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.1.1.200 --pod-network-cidr=10.244.0.0/16

curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          5m
kube-apiserver-k8s-master-1            1/1       Running   0          5m
kube-controller-manager-k8s-master-1   1/1       Running   0          5m
kube-dns-686d6fb9c-xxrbg               3/3       Running   0          5m
kube-flannel-ds-gxt4n                  1/1       Running   0          23s
kube-flannel-ds-hngfv                  1/1       Running   0          2m
kube-flannel-ds-mgxdn                  1/1       Running   0          1m
kube-flannel-ds-qb8ch                  1/1       Running   0          3m
kube-proxy-4kxr8                       1/1       Running   0          1m
kube-proxy-54q5g                       1/1       Running   0          5m
kube-proxy-7zb4p                       1/1       Running   0          23s
kube-proxy-rwvp4                       1/1       Running   0          2m
kube-scheduler-k8s-master-1            1/1       Running   0          5m

If you use flannel instead of weave networking the kernel oops does not occur. You can install flannel using
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
Make sure you install flannel before joining any nodes to the cluster.

I have the same issue - hanging at 'sudo kubeadm init' on master at line:
'[init] This might take a minute or longer if the control plane images have to be pulled.'
Tried so many different versions and options but still no luck.

I am using a raspberry pi 2 B+

Have used various raspbians Wheezy/Stretch various kubernetes up to latest (inc. 1.9.6 as suggested by PeterKing above) and various docker versions.

Anyone with this running on raspberry Pi 2 with recent raspbian, able to share version of all components (raspbian + kubernetes + docker)?

Please im sick of reflashing my SD :)

Many many thanks for this bootstrap introduction !

I was facing issues with the latest version ( v1.10.2 - 28-04-2018 ) and after loosing some (more) hair - kube-apiserver was dying in loop ultimately leading to fail of kudeadm init - , I tried to downgrade both kubeadm and kubelet to 1.9.7-00 and - for now as it's a fresh start - things are up on my RPI3 cluster ... Cross finger :)
My cluster is a result of this simple :

sudo kubeadm init

Kudo for @Jickelsen

@alexellis: Thanks for a great guide! 👍
@Creamen: was facing exact same issue with same version, running on rpi3b+. The apiserver was looping, dying and geting stuck in a timeout on kubeadm init. After a downgrade it went through.

Im seeing Weavenet being mentioned but I can’t see that anyone has logged an issue with kubeadm or weave - I’d suggest doing that if you are seeing unexpected behaviour with newer versions of the components.

The init step can take several minutes.

I have attempted to get Weave Net to work with k8s v1.10.2 and exausted all options it seems. I could only get it working with flannel as the CNI. I also had to add and change some commands.

I forked your gist and made the modifications including changing the script (prep.sh)

Try it out
https://github.com/aaronkjones/rpi-k8s-node-prep

pi@k8s-master-1:~ $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    2h        v1.10.2
k8s-slave-1    Ready     <none>    2h        v1.10.2
k8s-slave-2    Ready     <none>    33m       v1.10.2
k8s-slave-3    Ready     <none>    33m       v1.10.2
pi@k8s-master-1:~ $ kubectl get pods -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          2h
kube-apiserver-k8s-master-1            1/1       Running   0          2h
kube-controller-manager-k8s-master-1   1/1       Running   0          2h
kube-dns-686d6fb9c-glz8x               3/3       Running   0          2h
kube-flannel-ds-5grwv                  1/1       Running   0          2h
kube-flannel-ds-756mt                  1/1       Running   0          33m
kube-flannel-ds-7hvdg                  1/1       Running   0          33m
kube-flannel-ds-k6hsn                  1/1       Running   0          2h
kube-proxy-hzpkw                       1/1       Running   0          33m
kube-proxy-wsj9v                       1/1       Running   0          2h
kube-proxy-xjvvp                       1/1       Running   0          2h
kube-proxy-z5ngl                       1/1       Running   0          33m
kube-scheduler-k8s-master-1            1/1       Running   0          2h
kubernetes-dashboard-64d66bcc8-vcc5v   1/1       Running   0          22m

@alexellis : thanks for the guide, it really helped me. I've been trying to get it working with Weave for a couple of days, but in the end I gave up and went with @aaronkjones 's idea. I used flannel as the CNI and got it working on the first try.

Same here: @aaronkjones 's guide is what worked for me as well. I took the liberty of creating a variant of this gist for those who want to use Hypriot. It also covers networking setup a bit more in-depth (local ethernet for the cluster, wifi connection via the master to reach the outside world): https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f . Still a work in progress.

Just as a heads up - @aaronkjones solution was working for me perfectly last week, but I added new worker nodes to my existing cluster today and the new nodes don't initialise flannel or kube-proxy:

flannel:

Error: failed to start container "install-cni": Error response from daemon: linux mounts: Could not find source mount of /var/lib/kubelet/pods/532b1489-569a-11e8-aed4-b827eb359bc2/volumes/kubernetes.io~configmap/flannel-cfg
Back-off restarting failed container

kube-proxy:

Error: failed to start container "kube-proxy": Error response from daemon: linux mounts: Could not find source mount of /lib/modules
Back-off restarting failed container

Turns out, as of the last couple of days the get-docker install script now installs docker-ce 18.05 as the latest version, and this seems to cause this error.

Step in @alexellis instructions above (which installs latest version of docker):

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

I just downgraded the docker version on my nodes to docker-ce 18.04, rebooted, and everything seems OK now and my nodes initialise correctly.

$ sudo apt-get install docker-ce=18.04.0~ce~3-0~raspbian

You can check out the packages added to the repo lists by using:

apt-cache madison docker-ce

Hope this helps someone! Massive thanks to @alexellis and everyone else in this thread who have got me a working K8s cluster on rPi's - learnt loads!

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

https://github.com/aaronkjones/rpi-k8s-node-prep

I modified the script to allow for a specific version of Docker and Kubeadm to be installed and also pinned to prevent upgrade.

I have 4 RPis, so i made two two-node clusters and tried different combinations of Kubeadm/Docker.

Docker 18.04 and Kubeadm 1.10.2-00 work for me. It has been running on Hypriot for a few days.

Where or how do we report the issue to?

I couldn't get 1.10.3-00 working either.

For 1.10.2-00, in addition to downloading, installing, and holding the right packages, when you init the master, you need to set the version there too (otherwise, it'll default download the latest stable control images, which are 1.10.3)

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.10.2

@njohnsn, probably an issue on the Kubernetes repository at https://github.com/kubernetes/kubernetes/issues

I ran into the same issue, was getting errors like the one in this comment with the latest version of kubelet: geerlingguy/raspberry-pi-dramble#100 (comment)

I uninstalled docker-ce then reinstalled with sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian.

The init command I used (after installing with sudo apt-get install -y kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00) was:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=PI_IP_HERE --kubernetes-version v1.10.2

I had to downgrade both Kubernetes (from 1.10.3 to 1.10.2) and Docker CE (from 18.05.0 to 18.04.0) to Kubernetes to boot and run on Debian Stretch (Raspbian Lite)... but I finally got to:

# kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                  STATUS     ROLES     AGE       VERSION
kube1.pidramble.com   NotReady   master    5m        v1.10.2

For some reason I can't get my fourth node to go into the ready state.

I've blown the SD card a way and reinstalled everything from scratch twice, but unlike the other 3 nodes, it won't come up.

Here is the output from syslog:

May 24 02:50:56 k8s-node-4 dockerd[479]: time="2018-05-24T02:50:56.309407492Z" level=error msg="Handler for GET /v1.31/images/weaveworks/weave-kube:2.3.0/json returned error: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311048     298 remote_image.go:83] ImageStatus "weaveworks/weave-kube:2.3.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311172     298 kuberuntime_image.go:87] ImageStatus for image {"weaveworks/weave-kube:2.3.0"} failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311326     298 kuberuntime_manager.go:733] container start failed: ImageInspectError: Failed to inspect image "weaveworks/weave-kube:2.3.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311433     298 pod_workers.go:186] Error syncing pod 95d24908-5efa-11e8-b36b-b827ebccbe66 ("weave-net-hxgx5_kube-system(95d24908-5efa-11e8-b36b-b827ebccbe66)"), skipping: failed to "StartContainer" for "weave" with ImageInspectError: "Failed to inspect image \"weaveworks/weave-kube:2.3.0\": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:58 k8s-node-4 kubelet[298]: W0524 02:50:58.686588     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:50:58 k8s-node-4 kubelet[298]: E0524 02:50:58.689475     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:03 k8s-node-4 kubelet[298]: W0524 02:51:03.694277     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:03 k8s-node-4 kubelet[298]: E0524 02:51:03.695457     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:08 k8s-node-4 kubelet[298]: W0524 02:51:08.700522     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:08 k8s-node-4 kubelet[298]: E0524 02:51:08.701369     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Thanks!

@njohnsn I never got weave to work but have got flannel working. After looking into the issue I couldn't resolve the whole ImageInspectError so gave up. Out of 4 nodes I had two in the cluster and two wouldn't ever connect. Tried clean installs on them and still nothing. Strangely enough if I did a clean install of the cluster it would be different PIs that would fail :/

To get flannel to work I had to update /etc/kubernetes/manifests/kube-controller-manager.yaml and update the spec containers command with:
- --allocate-node-cidrs=true - --cluster-cidr=172.30.0.0/16
After reloading systemd (systemctl daemon-reload) and restarting k8s (systemctl restart kubelet.service) I was able to get flannel working.

I think you won't need that if you run kubeadm init with the flag --pod-network-cidr <network cidr> but not tried that since I got my cluster up and running. Will give it ago when I got more time (and update my playbook https://github.com/carlosroman/ansible-k8s-raspberry-playbook/) and see if flannel "just works".

I can report (eventual) success with the following configuration on my 4 Raspberry pis:

• Stretch Lite (2018-04-08)

• v.18.04.0-ce Docker-ce [sudo apt-get install docker-ce=18.04.0ce3-0~raspbian]

• Flannel (could not get my nodes to get to Ready state with Weave) [curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f - ]
  Note: no further editing of Kubernetes manifest files as suggested previously was required

• v1.10.2 of Kubernetes (like many before, had to back rev from 1.10.3) [ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address= --kubernetes-version v1.10.2 --pod-network-cidr= ]

And thanks Alex for this original post and others who commented. Hoping this helps someone else struggling ....

I can not have dashboard work using proxy. I got the error message:

Error: 'dial tcp 172.17.0.2:9090: getsockopt: connection refused'
Trying to reach: 'https://172.17.0.2:9090/'

I followed the steps and learned some rbac along the way, but still can not figure out where to look at to solve this.

Any suggestions?

I also found this from kubernetes/dashboard readme:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

From what I have so far, my cluster did not install Heapster. Is it necessary to mention that in this guide?

I am late to the fun :) has anyone followed this on the latest?
RASPBIAN STRETCH WITH DESKTOP
Image with desktop based on Debian Stretch
Version:April 2018
Release date:2018-04-18
Kernel version:4.14

I have not been able to progress beyond this
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.10.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-1] and IPs [192.168.0.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm:v1.10.3
- k8s.gcr.io/kube-controller-manager-arm:v1.10.3
- k8s.gcr.io/kube-scheduler-arm:v1.10.3
- k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Appreciate your time and help

@Creamen downgrading as you advised worked for me. Wasted 2 nights.
Anyone facing a similar issue here is how to downgrade (uninstall the latest version and install v1.9.700)

Step 1: uninstall kubernetes, follow the below given commands
$kubeadm reset
$sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$sudo apt-get autoremove
$sudo rm -rf ~/.kube

Step 2: reboot your pi
$sudo reboot

Step 3: Install v1.9.7-00, follow the below given commands
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

Step 4: Initiate your master node
$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

Still no luck.
I switched to flannel but now the dns pod doesn't work:
commands I used after downgrading to Docker 18.0.4 and k8s 10.1.2.

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.1.240 --pod-network-cidr=172.30.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:/var/log/containers $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    41m       v1.10.2
k8s-node-1     Ready     <none>    40m       v1.10.2
k8s-node-2     Ready     <none>    40m       v1.10.2
k8s-node-3     Ready     <none>    40m       v1.10.2
k8s-node-4     Ready     <none>    40m       v1.10.2
k8s-node-5     Ready     <none>    40m       v1.10.2

pi@k8s-master-1:/var/log/containers $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS             RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running            0          41m
kube-apiserver-k8s-master-1            1/1       Running            0          42m
kube-controller-manager-k8s-master-1   1/1       Running            0          42m
kube-dns-686d6fb9c-ms85s               1/3       CrashLoopBackOff   27         42m
kube-flannel-ds-2nfrv                  1/1       Running            0          36m
kube-flannel-ds-8j8vh                  1/1       Running            0          36m
kube-flannel-ds-b2dfb                  1/1       Running            0          36m
kube-flannel-ds-qxrt6                  1/1       Running            0          36m
kube-flannel-ds-rr4dr                  1/1       Running            0          36m
kube-flannel-ds-w92sm                  1/1       Running            0          36m
kube-proxy-6f7g5                       1/1       Running            0          41m
kube-proxy-7w57r                       1/1       Running            0          41m
kube-proxy-8lvkq                       1/1       Running            0          41m
kube-proxy-f4x4s                       1/1       Running            0          41m
kube-proxy-psll8                       1/1       Running            0          42m
kube-proxy-ttrdn                       1/1       Running            0          41m
kube-scheduler-k8s-master-1            1/1       Running            0          42m
pi@k8s-master-1:/var/log/containers $

Where do I find the logs for the dns pod?

Thanks!

Found the logs for the DNS pod"

pi@k8s-master-1:~ $ kubectl logs -f --namespace=kube-system kube-dns-686d6fb9c-ms85s -c kubedns
I0603 14:37:58.522134       1 dns.go:48] version: 1.14.8
I0603 14:37:58.526437       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0603 14:37:58.526863       1 server.go:119] FLAG: --alsologtostderr="false"
I0603 14:37:58.527113       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0603 14:37:58.527202       1 server.go:119] FLAG: --config-map=""
I0603 14:37:58.527393       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0603 14:37:58.527473       1 server.go:119] FLAG: --config-period="10s"
I0603 14:37:58.527712       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0603 14:37:58.527788       1 server.go:119] FLAG: --dns-port="10053"
I0603 14:37:58.528038       1 server.go:119] FLAG: --domain="cluster.local."
I0603 14:37:58.528123       1 server.go:119] FLAG: --federations=""
I0603 14:37:58.528348       1 server.go:119] FLAG: --healthz-port="8081"
I0603 14:37:58.528425       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0603 14:37:58.528658       1 server.go:119] FLAG: --kube-master-url=""
I0603 14:37:58.528740       1 server.go:119] FLAG: --kubecfg-file=""
I0603 14:37:58.528935       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0603 14:37:58.529036       1 server.go:119] FLAG: --log-dir=""
I0603 14:37:58.529226       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0603 14:37:58.529307       1 server.go:119] FLAG: --logtostderr="true"
I0603 14:37:58.529497       1 server.go:119] FLAG: --nameservers=""
I0603 14:37:58.529524       1 server.go:119] FLAG: --stderrthreshold="2"
I0603 14:37:58.529547       1 server.go:119] FLAG: --v="2"
I0603 14:37:58.529570       1 server.go:119] FLAG: --version="false"
I0603 14:37:58.529602       1 server.go:119] FLAG: --vmodule=""
I0603 14:37:58.530045       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0603 14:37:58.531256       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0603 14:37:58.531308       1 dns.go:146] Starting endpointsController
I0603 14:37:58.531334       1 dns.go:149] Starting serviceController
I0603 14:37:58.533533       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0603 14:37:58.533610       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0603 14:37:59.032154       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:37:59.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.032087       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.532194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.532223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.032227       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.032217       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.532270       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.032212       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.532256       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.032242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.532230       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.032174       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.532176       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.032166       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.032207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.532207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.032224       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.532219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.032115       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.532079       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.032109       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.532204       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.032178       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.532142       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.032191       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.532209       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.032232       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.032219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.532118       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.032194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.532202       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.032177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.532185       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.032170       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.532257       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.032189       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.532366       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.032243       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.532134       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.032122       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.032221       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.532175       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.032214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.532241       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.032198       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.532179       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.032288       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.532242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.032240       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.032140       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.532060       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

It appears the DNS process is trying to check on the api status on port 443 when the api server is running on port 6443.
I don't know where to change it from though.

-Neil

Turns out the answer was here
I had to bump the version of kube-dns to 1.14.10.

@njohnsn Logs are shown by kubectl logs. A kube-dns pod is made up of 3 containers but the one you need to watch is kubedns (the others are dnsmasq & sidecar). --follow is if you want to live follow the logs. In the above case you could use:

kubectl logs --follow pods/kube-dns-686d6fb9c-ms85s -c kubedns --namespace=kube-system

@kumardeepam Thanks for you work and instructions on how to downgrade (uninstall the latest version and install v1.9.700) when getting an error "[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2"

My master is running now!

Also had to set cgroup_memory=memory not cgroup_memory=1.

useful way of downgrading docker is after running --
$ curl -sSL get.docker.com | sh && \ sudo usermod pi -aG docker

run this to downgrade without having to go through a full uninstall --
sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian --allow-downgrades

Thanks for the tip @tomtom215!

Following the thread above i wonder if we need to downgrade both the k8s and docker versions? Or just downgrading docker is enough?

Hi there, did someone succeed to deploy it with docker 18.05 and k8s 1.11 ? I tried to but got issue I don't understand, I'm new to docker and k8s...

Got this when trying to join :

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

and the controller pod keep crashing on my masternode then :

pi@MasterNode:~ $ kubectl get all --all-namespaces -o wide
NAMESPACE     NAME                                     READY     STATUS             RESTARTS   AGE       IP              NODE
kube-system   pod/coredns-78fcdf6894-mwx54             0/1       Pending            0          18m       <none>          <none>
kube-system   pod/coredns-78fcdf6894-skpvr             0/1       Pending            0          17m       <none>          <none>
kube-system   pod/etcd-masternode                      1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-apiserver-masternode            1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-controller-manager-masternode   0/1       CrashLoopBackOff   7          19m       10.192.79.250   masternode
kube-system   pod/kube-proxy-44gcl                     1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-scheduler-masternode            1/1       Running            0          18m       10.192.79.250   masternode

There is a configuration that work well for someone ? Which versions and which CNI ?

Thanks by advance :)

Mine seems to be working fine . I'm using ce=18.04.0ce3-0~raspbian kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
with flannel as the CNI . As mentioned above , the version of kubedns does not place nice. One you have done the cluster installation then do
kubectl edit deploy kube-dns --namespace=kube-system to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors
I now have OpenFAAS and prometheus-operator (arm) running fine it , even using the kubernetes incuabtor nfs-client-provisioner so I can use a non-clustered pi as an NFS server for PV and PVC's

sudo  sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  # DO this on MASTER NODE ONLY

Now initialise the master node 

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.1.62
# note 192.168.1.62 is the ip of my master node```

docker-ce=18.04.0ce3-0~raspbian
kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
CNI using flannel
Once initilased do as njohnsn says above and change the version of kubeDNS to 1.14.10 (1.14.8 is broken)

kubectl edit deploy kube-dns --namespace=kube-system

to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors

my cluster is now happily running the armhf version of OpenFAAS and the arm version of prometheues-operator , and using the docker-incubator nfs-client-provisioner to handle PV and PVC,s served from an NFS Server which is not part of the cluster
Hope that help

Andy

I am unable to get past the CGROUPS_MEMORY: missing issue after running sudo kubeadm init --token-ttl=0

@gianlazz: same thing for me, had to revert to cgroup_memory=memory on my rpi2, rpi3 and rpi3+ with latest raspbian.

I'm still unable to run a cluster at all. I'm blocked at the init:

pi@rpi-k8s-master-1:~ $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.60 --pod-network-cidr=172.30.0.0/16 --kubernetes-version v1.10.2 --ignore-preflight-error
s=ALL
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.04.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [rpi-k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96
.0.1 192.168.0.60]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [rpi-k8s-master-1] and IPs [192.168.0.60]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - Either there is no internet connection, or imagePullPolicy is set to "Never",
          so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-arm:v1.10.2
                - k8s.gcr.io/kube-controller-manager-arm:v1.10.2
                - k8s.gcr.io/kube-scheduler-arm:v1.10.2
                - k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Image used:

RASPBIAN STRETCH LITE
Minimal image based on Debian Stretch
Version:June 2018
Release date:2018-06-27
Kernel version:4.14

Docker version:

Server:
 Engine:
  Version:      18.04.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   3d479c0
  Built:        Tue Apr 10 18:21:25 2018
  OS/Arch:      linux/arm
  Experimental: false

Kubernetes version: 1.10.2

@Gallouche Apparently the kube-controller-manager CrashLoopBackOff will be fixed in 1.11.1 according to kubernetes/kubernetes#65674
I also note that after the recent kernal update on the Pi, weavenet now works for me. I've had openfaas (and this gist) working nicely since then. Once 1.11.1 comes out and tests ok I'm going to freeze versions in my ansible script.

Thank you all for the answers, my bachelor degree work will maybe be done in time ! I will try, thanks a lot !

@deurk it looks like your kubelet service is not running, kubeadm has a dependancy on kubelet which does seem to be documented very well.
Try

sudo systemctl restart kubelet
sudo systemctl status kubelet

The problem seems to be that /etc/kubernetes/kubelet.conf is missing in the initial installation, after the first run of kubeadm init a copy of the kubelet.conf gets created.

alright so after 2 days of googling in multiple forums coming back to this tutorial many times and formating my varous microsd cards over and over again, i finally go it to work by replacing the step where it says "Add repo lists & install kubeadm" to:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

and where it said "Edit /boot/cmdline.txt"

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

hopefully it will help some tired soul... and yes i understand I'm running an old version of kubernetes, i don't care its working and i just wanna learn...

While I'd like to get this working with latest I was successful following @kumardeepam to downgrade version as well as of July 6, 2018. The dns pods weren't starting, and using a specific version of weave brought them online as per @haebler advice.

Thanks to all contributors and @alexellis for putting this together.

Thus, the steps for success with a slightly older version circa July 2018 are as follows:

changes in pre-req

sudo vi /boot/cmdline.txt
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

#changes in install

Install v1.9.7-00, follow the below given commands

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

install weave 1.6

kubectl apply -f https://git.io/weave-kube-1.6

no change for init

$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

I went through the process and am still having this issue:

[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Edit

I ran $sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL from the comment above however it looks like my IP address may not have been configured correctly so I've run $sudo kubeadm reset and and trying again to initialize Kubernetes with sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL

Edit

Running into another error now that if I understand correctly is due to the dns pods not having been started correctly however when I run kubectl apply -f https://git.io/weave-kube-1.6 it says The connection to the server localhost:8080 was refused - did you specify the right host or port?

So I'm still left with this below:

sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.6]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster```

Thanks @futurisma, this did not solve my problem so I reinstalled my cluster with kube* in 1.9.7 per @mrpaws instructions and that worked. Now I'm waiting for 1.11.1 to try again :)

I'm still having the same issue after trying every single one of the above suggestions :(

@mgazza Which issue is that?

@kumardeepam Solution works, Thanks! I had the healthz problem. I had to use 1.9.7-00
on master
sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.X.XXX --ignore-preflight-errors=ALL --kubernetes-version v1.9.7

Had luck with flannel. Weave was giving me the loopBackError

This post from @aaronkjones was important: sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

curl -sSL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm64/g" | kubectl apply -f -

This post also had something to say : kubectl annotate node pine1 flannel.alpha.coreos.com/public-ip=192.168.X.XXX --overwrite=true (not sure if did anything)

ps: in between attempts I cleared things: kubeadm reset && sudo rm -rf ~/.kube && sudo systemctl daemon-reload && sudo systemctl restart docker && sudo systemctl restart kubelet sudo reboot

Everything seemed good now, sudo netstat -tuplen | grep 6443 -> LISTEN. But the join command on the nodes was giving me this: Failed to request cluster info, will try again: ... dial tcp 192.168.1.163:6443: i/o timeout] . Solution was to disable firewall on master, i know it's ugly: sudo ufw disable

Now I can ping master from nodes: nc -vz 192.168.1.163 6443 & the join command works.

@deurk
after following @alexellis gist and having no joy and after swapping part of the original for @mrpaws and pretty much every other variation

history
    1  curl -sSL get.docker.com | sh && sudo usermod pi -aG docker
    2  sudo dphys-swapfile swapoff &&   sudo dphys-swapfile uninstall &&   sudo update-rc.d dphys-swapfile remove
    3  sudo nano /boot/cmdline.txt
    4  sudo reboot
    5  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00
    6  kubectl apply -f https://git.io/weave-kube-1.6
    7  sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.12 --ignore-preflight-errors=ALL

[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
	[WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.12]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
	- There is no internet connection, so the kubelet cannot pull the following control plane images:
		- gcr.io/google_containers/kube-apiserver-arm:v1.9.9
		- gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
		- gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

uname -a
Linux raspberrypi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux

@aaronkjones I had no joy with your repo either, using raspbian or Hypriot

Kubernetes 1.11.1 has been released! Following this gist (with the /boot/cmdline.txt change) I've got openfaas & faas-cli working! AND that's with weavenet rather than flannel. [Not sure where I got the heapster.yaml file to get the dashboard working as I lost my notes for that].

`pi@shepherd:~/faas-functions $ uname -a
Linux shepherd 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

pi@shepherd:~/faas-functions $ docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:19:46 2018
OS/Arch: linux/arm
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:15:34 2018
OS/Arch: linux/arm
Experimental: false

pi@shepherd:~/faas-functions $ sudo apt list kube*
Listing... Done
kubeadm/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubectl/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubelet/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubernetes-cni/kubernetes-xenial,now 0.6.0-00 armhf [installed]`

Off to buy more SD cards to do a backup.

Thanks Alex!

After a lot of trouble with Hypriot's tutorial and this gist as well, I was able to successfully deploy and run the markdownrender on two pis with @aaronkjones's guide. Much thanks to @alexellis and @aaronkjones for the development of these guides as well as the discussion in the comments.

hi All, I think I'm OK - can anyone tell me if I need to run Weave on the workers? If so, I'm getting a port 8080 error. Many TIAs ;-)

NAME            STATUS    ROLES     AGE       VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
bramble-1-pi3   Ready     master    4h        v1.11.1   172.17.12.134   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-2-pi3   Ready     <none>    48m       v1.11.1   172.17.12.135   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-3-pi3   Ready     <none>    48m       v1.11.1   172.17.12.127   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
pi@bramble-1-pi3:~ $ kubectl get pods -n kube-system
NAME                                        READY     STATUS             RESTARTS   AGE
coredns-78fcdf6894-nkjxz                    1/1       Running            0          4h
coredns-78fcdf6894-q8489                    1/1       Running            0          4h
etcd-bramble-1-pi3                          1/1       Running            0          4h
kube-apiserver-bramble-1-pi3                1/1       Running            0          4h
kube-controller-manager-bramble-1-pi3       1/1       Running            0          4h
kube-proxy-fr9mg                            1/1       Running            0          48m
kube-proxy-qnb9q                            1/1       Running            0          48m
kube-proxy-vm7gg                            1/1       Running            0          4h
kube-scheduler-bramble-1-pi3                1/1       Running            0          4h
kubernetes-dashboard-6948bdb78-ndfzk        0/1       CrashLoopBackOff   3          2m
kubernetes-dashboard-head-6b79997c9-mc5q5   1/1       Running            0          2m
weave-net-dmzl5                             2/2       Running            0          48m
weave-net-n7gf8                             2/2       Running            0          48m
weave-net-pjzhn                             2/2       Running            0          3h

pi@bramble-3-pi3:~ $ kubectl apply -f \
>  "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
error: unable to recognize "https://cloud.weave.works/k8s/net?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxMSIsIEdpdFZlcnNpb246InYxLjExLjEiLCBHaXRDb21taXQ6ImIxYjI5OTc4MjcwZGMyMmZlY2M1OTJhYzU1ZDkwMzM1MDQ1NDMxMGEiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE4LTA3LTE3VDE4OjUzOjIwWiIsIEdvVmVyc2lvbjoiZ28xLjEwLjMiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYXJtIn0K": Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused```

OK so it looks like Weave is running across the cluster so presumably only requires installing on the master:

weave-net-dmzl5                             2/2       Running   0          3d        172.17.12.135   bramble-2-pi3
weave-net-n7gf8                             2/2       Running   0          3d        172.17.12.127   bramble-3-pi3
weave-net-pjzhn                             2/2       Running   0          4d        172.17.12.134   bramble-1-pi3

@micedwards how did you get the latest 1.11.1 to even have a successful install?

OK, 1.11.2 is out. I will give it a spin tonight or tomorrow to see how it fares with this Gist.

Took me a bit longer than expected to get to test it but I managed to make it work 👍

root@rpi-k8s-master-1:~# kubectl get nodes -o wide
NAME               STATUS    ROLES     AGE       VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
rpi-k8s-master-1   Ready     master    11h       v1.11.2   192.168.0.60   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.52-v7+      docker://18.6.0
rpi-k8s-slave-1    Ready     <none>    3h        v1.11.2   192.168.0.61   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0

root@rpi-k8s-master-1:~# kubectl get pods --namespace=kube-system -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE               NOMINATED NODE
coredns-78fcdf6894-mrbcb                   1/1       Running   0          11h       172.19.0.2     rpi-k8s-master-1   <none>
coredns-78fcdf6894-vmzmw                   1/1       Running   0          11h       172.19.0.4     rpi-k8s-master-1   <none>
etcd-rpi-k8s-master-1                      1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-apiserver-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-controller-manager-rpi-k8s-master-1   1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-flannel-ds-9cllm                      1/1       Running   0          11m       192.168.0.61   rpi-k8s-slave-1    <none>
kube-flannel-ds-r8pf6                      1/1       Running   0          11m       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-4hjcn                           1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-n4pmb                           1/1       Running   0          3h        192.168.0.61   rpi-k8s-slave-1    <none>
kube-scheduler-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>

root@rpi-k8s-master-1:~# docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:19:46 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:15:34 2018
  OS/Arch:          linux/arm
  Experimental:     false

root@rpi-k8s-master-1:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}

root@rpi-k8s-master-1:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:14:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}

Flannel 0.9.1

Thank you very much for the write-up, very straight-forward and working like a charm. Just make sure to update the Raspbian images to the latest kernel version and to follow the instructions. On stretch light you don't have to create the dhcpcd.conf file yourself for static IP's, just adjust the existing file. And also sometimes it helps to wait for a bit, getting all the kube-system containers in ready and running state can take a while..now getting ready for OpenFaaS and maybe running some in-house Ghost-blogs.

@deurk How could you get it work? I init k8s 1.11.2 but I get this error: "failed to pull image [k8s.gcr.io/kube-proxy-arm:v1.11.2]: exit status 1". Can you help me fix it?
I used pi 3 model B

Just wanted to share here as I had been watching this thread off and on for a while as @aaronkjones directed me here. The latest deployment using Ansible for all of this now works again. I am using Weave BTW.

https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster

Has anyone been able to get their Rpi3 k8s cluster integrated with Gitlab-CE? I'm trying to integrate my cluster right now and it's failing to install Helm-Tiller through Gitlab CE - just failing to connect in general really.

figured i'd contribute some deviations from the instructions that helped me, i managed to get this running (28/08/2018) by specifying version 1.8.3 on kubelet, kubectl, kubeadm install, and using the flannel network.

Thank you for your example install commands for v1.9.7 @chito4! I was that tired sould you referenced that needed that info to get this working.

Got it running on Raspberry Pi 3B+, 16GB Micro SD card, Raspbian Stretch Lite OS, Kubernetes v1.9.7 (kubeadm, kubectl), Weave CNI, Docker 18.06.1-ce

I haven't attempted any nodes except the master node so far, but will update if I run into problems.

I was able to get a 7-node Raspberry Pi cluster running using:

• Raspbery Pi 3 B+
• Raspbian Stretch Lite (2018-06-27)
• Docker v18.06.1-ce (the latest apt-get installable as of 2018-09-07)
• Kubernetes v1.11.2
• Weave networking

Here is my exact system state:

$ cat /proc/device-tree/model
Raspberry Pi 3 Model B Plus Rev 1.3

$ uname -a
Linux pi-master 4.14.62-v7+ #1134 SMP Tue Aug 14 17:10:10 BST 2018 armv7l GNU/Linux

$ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=830d7945-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory

Note: Here's a really easy way to append the cgroup stuff to the /boot/cmdline.txt file:
sudo sed -i 's/ rootwait$/ rootwait cgroup_enable=cpuset cgroup_enable=memory/g' /boot/cmdline.txt

When I initialized the cluster, I used the following command:

$ sudo kubeadm init --token-ttl=0 --kubernetes-version v1.11.2 --apiserver-advertise-address=192.168.1.48

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:30:52 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:26:37 2018
  OS/Arch:          linux/arm
  Experimental:     false

$ kubeadm version # (formatted for readability)
kubeadm version: &version.Info{
  Major:"1",
  Minor:"11",
  GitVersion:"v1.11.2",
  GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239",
  GitTreeState:"clean",
  BuildDate:"2018-08-07T23:14:39Z",
  GoVersion:"go1.10.3",
  Compiler:"gc",
  Platform:"linux/arm",
}

$ kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
pi-master    Ready     master    13m       v1.11.2
pi-node-01   Ready     <none>    5m        v1.11.2
pi-node-02   Ready     <none>    5m        v1.11.2
pi-node-03   Ready     <none>    5m        v1.11.2
pi-node-04   Ready     <none>    5m        v1.11.2
pi-node-05   Ready     <none>    5m        v1.11.2
pi-node-06   Ready     <none>    5m        v1.11.2

$ kubectl get pods --namespace=kube-system
NAME                                READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-zl96s            1/1       Running   0          12m
coredns-78fcdf6894-zxc95            1/1       Running   0          12m
etcd-pi-master                      1/1       Running   0          11m
kube-apiserver-pi-master            1/1       Running   1          11m
kube-controller-manager-pi-master   1/1       Running   0          11m
kube-proxy-2zblw                    1/1       Running   0          6m
kube-proxy-5wb5l                    1/1       Running   0          6m
kube-proxy-6cngc                    1/1       Running   0          6m
kube-proxy-6sk8t                    1/1       Running   0          6m
kube-proxy-dczbt                    1/1       Running   0          6m
kube-proxy-rtvtm                    1/1       Running   0          12m
kube-proxy-zvwph                    1/1       Running   0          6m
kube-scheduler-pi-master            1/1       Running   0          11m
weave-net-49hfp                     2/2       Running   1          6m
weave-net-cz7pm                     2/2       Running   0          6m
weave-net-j75wt                     2/2       Running   0          6m
weave-net-jb7vp                     2/2       Running   0          6m
weave-net-kmzd8                     2/2       Running   0          6m
weave-net-kzspl                     2/2       Running   0          10m
weave-net-wr692                     2/2       Running   1          6m

$ dpkg -l | egrep "kube|docker"
ii  docker-ce                       18.06.1~ce~3-0~raspbian      armhf        Docker: the open-source application container engine
ii  kubeadm                         1.11.2-00                    armhf        Kubernetes Cluster Bootstrapping Tool
ii  kubectl                         1.11.2-00                    armhf        Kubernetes Command Line Tool
ii  kubelet                         1.11.2-00                    armhf        Kubernetes Node Agent
ii  kubernetes-cni                  0.6.0-00                     armhf        Kubernetes CNI

Still not joy using the scripts.

cat /proc/device-tree/model
Raspberry Pi 2 Model B Rev 1.1

uname -a
Linux node-1 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
>   echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
>   sudo apt-get update -q && \
>   sudo apt-get install -qy kubeadm
OK
deb http://apt.kubernetes.io/ kubernetes-xenial main
Hit:1 http://archive.raspberrypi.org/debian stretch InRelease
Get:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]
Hit:4 https://download.docker.com/linux/raspbian stretch InRelease
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf Packages [18.3 kB]
Fetched 42.3 kB in 3s (13.3 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
kubeadm is already the newest version (1.12.0-rc.1-00).

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.100 --ignore-preflight-errors=ALL
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.12.0-rc.1" Control plane version: "1.11.3"
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [node-1 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [node-1 localhost] and IPs [192.168.1.100 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

Hi @alexellis, thanks for your great work.
Just notice that the link inside quick.md hasn't been updated.

• From https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/b04f1e9250c61a8ff554bfe3475b6dd050062484/prep.sh
• To https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/125ad6eae27e40a235412c2b623285a089a08721/prep.sh
  The former link miss the setting of cgroup_enable=memory. Thanks again.

For those who want to save their SD cards running K8S like this (I know I spend a lot of them for fun and games) the quick and dirty fix and its really simple to add them to this howto :

I assume you have a raspbian / rsyslog / whatever NAS running somewhere on you're local network. DO NOT DO THIS OVER THE NET unless you have a hackwish ;)

On your NAS:

add before GLOBAL DIRECTIVES

sudo nano /etc/rsyslog.conf  

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

$template DynaFile,"/<YOUR NAS PATH>/%HOSTNAME%/%syslogfacility-text%.log"
*.* -?DynaFile

$ sudo systemctl reload rsyslog

(make sure your NAS firewall accepts 514 incoming).

Then on all "masters & workers" comment everything out in /etc/rsyslog.conf after the "RULES" section and add a little something like this:

$ sudo nano /etc/rsyslog.conf

# To remote syslog server
*.*      @@192.168.x:y:514

sudo systemctl reload rsyslog

I tried to keep it as simple as possible, again QUICK AND DIRTY adjust to your own situation. No need for complicated remote logging services and so on. And trust me your SD card will be a lot happier!!!

Thx again to that post. Unfortunately it does not work for me. So far the latest 1.9.x version is useable for me. More on that can be found here.

Running the get.docker.com script fails for me unless I disable swap first.

thx @denhamparry, worked for me without kubectl proxy

The issue with kubeadm init crashing on v1.12.x is due to the kube-apiserver container running out of memory (code 137). I've put up a bug report in kubernetes/kubeadm: kubernetes/kubeadm#1279

Hopefully, we can get a solution put together so we can run the latest kubernetes...

@kumardeepam @deurk @gianlazz @mgazza

To get the current flannel manifest (https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml) to work on 1.12.2 I had to apply the patch suggested here:

kubectl patch daemonset kube-flannel-ds-arm \
      --namespace=kube-system \
      --patch='{"spec":{"template":{"spec":{"tolerations":[{"key": "node-role.kubernetes.io/master", "operator": "Exists", "effect": 
      "NoSchedule"},{"effect":"NoSchedule","operator":"Exists"}]}}}}' ```

I just went through the entire process, and as of today (2018-12-03), it is very very unstable and fragile.

Recapping for anybody that is spending sleepless nights on it:

1. followed https://gist.github.com/alexellis/a7b6c8499d9e598a285669596e9cdfa2 - my nodes are called ocramius-k8s-pi-1 (192.168.1.110) and ocramius-k8s-pi-2 (192.168.1.111)

2. followed steps above until before kubeadm init (note: as I'm writing, I have v1.12.3 installed)

3. had to downgrade docker-ce to 18.06.0 on all hosts, due to kubernetes/minikube#3323. To do that, I followed:

   curl -sSL get.docker.com | sh && \
   sudo usermod pi -aG docker
   newgrp docker
   apt purge -y docker-ce && apt-autoremove -y
   apt install docker-ce=18.06.0~ce~3-0~raspbian

   Note that ignoring the preflight checks with --ignore-preflight-errors=SystemVerification won't work, since something changed in how dockerd handles temporary files. Make sure that docker version reports 18.06.0:

   pi@ocramius-k8s-pi-1:/home/pi# docker version
   Client:
    Version:           18.06.0-ce
    API version:       1.38
    Go version:        go1.10.3
    Git commit:        0ffa825
    Built:             Wed Jul 18 19:19:46 2018
    OS/Arch:           linux/arm
    Experimental:      false
   
   Server:
    Engine:
     Version:          18.06.0-ce
     API version:      1.38 (minimum version 1.12)
     Go version:       go1.10.3
     Git commit:       0ffa825
     Built:            Wed Jul 18 19:15:34 2018
     OS/Arch:          linux/arm
     Experimental:     false
   

4. went with the flannel setup (couldn't get weavenet to work):

   sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.1.110
   kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml
   sudo sysctl net.bridge.bridge-nf-call-iptables=1

5. kubectl get pods --namespace=kube-system will report something like following:

   pi@ocramius-k8s-pi-1:/home/pi# kubectl get pods --namespace=kube-system
   NAME                                        READY   STATUS              RESTARTS   AGE
   coredns-576cbf47c7-9bp9s                    0/1     ContainerCreating   0          2m29s
   coredns-576cbf47c7-jmgf5                    0/1     ContainerCreating   0          2m29s
   etcd-ocramius-k8s-pi-1                      1/1     Running             0          110s
   kube-apiserver-ocramius-k8s-pi-1            1/1     Running             1          95s
   kube-controller-manager-ocramius-k8s-pi-1   1/1     Running             0          106s
   kube-proxy-t4qc7                            1/1     Running             0          2m29s
   kube-scheduler-ocramius-k8s-pi-1            1/1     Running             0          2m44s
   

   Inspecting the pods that are in ContainerCreating status, you will get something like:

   kubectl describe pods coredns-576cbf47c7-9bp9s --namespace=kube-system
   <snip>
   Events:
      Type     Reason           Age                  From                        Message
    ----     ------           ----                 ----                        -------
    Normal   Scheduled        2m41s                default-scheduler           Successfully assigned kube-system/coredns-576cbf47c7-9bp9s to ocramius-k8s-pi-1
    Warning  NetworkNotReady  1s (x13 over 2m41s)  kubelet, ocramius-k8s-pi-1  network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
   

6. applied the patch suggested by @cglazner right above me:

   kubectl patch daemonset kube-flannel-ds-arm \
     --namespace=kube-system \
     --patch='{"spec":{"template":{"spec":{"tolerations":[{"key": "node-role.kubernetes.io/master", "operator": "Exists", "effect": 
     "NoSchedule"},{"effect":"NoSchedule","operator":"Exists"}]}}}}'

7. system will recover:

   pi@ocramius-k8s-pi-1:/home/pi# kubectl get pods --namespace=kube-system
   NAME                                        READY   STATUS    RESTARTS   AGE
   coredns-576cbf47c7-9bp9s                    1/1     Running   0          31m
   coredns-576cbf47c7-jmgf5                    1/1     Running   0          31m
   etcd-ocramius-k8s-pi-1                      1/1     Running   0          30m
   kube-apiserver-ocramius-k8s-pi-1            1/1     Running   1          30m
   kube-controller-manager-ocramius-k8s-pi-1   1/1     Running   0          30m
   kube-flannel-ds-arm-pznfc                   1/1     Running   0          19m
   kube-proxy-t4qc7                            1/1     Running   0          31m
   kube-scheduler-ocramius-k8s-pi-1            1/1     Running   0          31m
   pi@ocramius-k8s-pi-1:/home/pi# kubectl get nodes
   NAME                STATUS   ROLES    AGE   VERSION
   ocramius-k8s-pi-1   Ready    master   32m   v1.12.3
   

8. can now join other nodes (in my case ocramius-k8s-pi-2):

   sudo sysctl net.bridge.bridge-nf-call-iptables=1
   kubeadm join 192.168.1.110:6443 --token <snip> --discovery-token-ca-cert-hash sha256:<snip>

9. verify status:

   pi@ocramius-k8s-pi-1:/home/pi# kubectl get nodes
   NAME                STATUS   ROLES    AGE   VERSION
   ocramius-k8s-pi-1   Ready    master   37m   v1.12.3
   ocramius-k8s-pi-2   Ready    <none>   70s   v1.12.3
   

Very nice!
Indeed the patch mentionned by cglazner did fix the flannel issue.
I'm on k8s 1.13.0 with flannel 0.10-arm:

#kubectl get no
NAME STATUS ROLES AGE VERSION
pi-master Ready,SchedulingDisabled master 53m v1.13.0
pi3-slave-01 Ready worker 46m v1.13.0
pi3-slave-02 Ready worker 46m v1.13.0
pi3-slave-03 Ready worker 46m v1.13.0
pi3-slave-04 Ready worker 46m v1.13.0

I use the below code to create the cluster:

--On Master node--

root:
kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16
myuser :
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml| sed "s/amd64/arm/g" | kubectl create -f -

kubectl -n kube-system patch daemonset kube-flannel-ds
--patch='{"spec":{"template":{"spec":{"tolerations":[{"key": "node-role.kubernetes.io/master", "operator": "Exists", "effect": "NoSchedule"},{"effect":"NoSchedule","operator":"Exists"}]}}}}'

sysctl net.bridge.bridge-nf-call-iptables=1

--On Slaves node--

kubeadm join 192.168.x.x:6443 --token --discovery-token-ca-cert-hash sha256:

--On Master node--

kubectl cordon pi-master
kubectl label node pi3-slave-01 node-role.kubernetes.io/worker=
kubectl label node pi3-slave-02 node-role.kubernetes.io/worker=
kubectl label node pi3-slave-03 node-role.kubernetes.io/worker=
kubectl label node pi3-slave-04 node-role.kubernetes.io/worker=

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

* Edit `kubernetes-dashboard` service.

$ kubectl -n kube-system edit service kubernetes-dashboard

* You should the see yaml representation of the service. Change type: **ClusterIP** to type: **NodePort** and save file.

* Check port on which Dashboard was exposed.

$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m

* Create a proxy to view within your browser

$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

* Browse [localhost:8001](http://localhost:8001)

Thanks again Alex!

Many thanks Alex for a great post and thanks to Denham I incorporated your comments to my dashboard config with help from this post to get me up and running https://kubecloud.io/kubernetes-dashboard-on-arm-with-rbac-61309310a640 having experienced the same issues with RBAC permissions in the post. (from...

$ kubectl create serviceaccount dashboard....

I hope this helps anyone who is getting to grips with K8S RPI cluster!! ::D

New problem seems to have cropped up on flannel: coreos/flannel#1060

vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false E1218 03:12:12.719715 1 main.go:280] Error registering network: failed to configure interface flannel.1: failed to ensure address of interface flannel.1: link has incompatible addresses. Remove additional addresses and try again. &netlink.Vxlan{LinkAttrs:netlink.LinkAttrs{Index:5, MTU:1450, TxQLen:0, Name:"flannel.1", HardwareAddr:net.HardwareAddr{0x2, 0x2a, 0x1c, 0x2f, 0x61, 0x25}, Flags:0x13, RawFlags:0x11043, ParentIndex:0, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0x13a320e4), Promisc:0, Xdp:(*netlink.LinkXdp)(0x13812200), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x0}, VxlanId:1, VtepDevIndex:2, SrcAddr:net.IP{0xc0, 0xa8, 0x3, 0x32}, Group:net.IP(nil), TTL:0, TOS:0, Learning:false, Proxy:false, RSC:false, L2miss:false, L3miss:false, UDPCSum:true, NoAge:false, GBP:false, Age:300, Limit:0, Port:8472, PortLow:0, PortHigh:0}

To allow the pod to start successfully, SSH onto the worker and run sudo ip link delete flannel.1. Recreating the pod will then start successfully.

Install Kubernetes 1.13.1 on Raspberry Pi Cluster

This comment combines the knowledge of this gist and the many comments above plus the workings of https://github.com/aaronkjones/rpi-k8s-node-prep

Download Raspbian Stretch Lite (2018-11-13 4.14 kernel), flash to sd cards for your cluster (in my case 5 cards).Once flashed and BEFORE you boot the pis, set up the networking by mounting the sd card (you can of course just boot the Pis and setup networking on each machine, my personal preference is to do this in advance):

Turn on ssh: sudo touch <boot partition mount point>/ssh

Enable C-Groups: sudo vi <boot partition mount point>/cmdline.txt

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=7ee80803-02 rootfstype=ext4 elevator=deadline fsck.repair=yes group_enable=cpuset cgroup_memory=1 cgroup_enable=memory rootwait quiet init=/usr/lib/raspi-config/init_resize.sh

I'm using wired networking on a 192.168.2.xx subnet so setup the host entries: sudo vi <rootfs partition mount point>/etc/hosts

 	...add to bottom of file...
	192.168.2.31       node01
	192.168.2.32       node02
	192.168.2.33       node03
	192.168.2.34       node04
	192.168.2.35       node05

sudo vi <rootfs partition mount point>/etc/dhcpcd.conf

	...add to bottom of file...
    interface eth0
	static ip_address=192.168.2.XX/24   <--- change XX = 31,32,33,34,35
	static routers=192.168.2.1
	static domain_name_servers=192.168.2.1

Unmount the sd card, then in turn mount the other 4 sd cards repeating the steps above changing the ip address. Once you have setup all 5 sdcards put them into the pis and power on the cluster. SSH to each in turn and complete the configuration and install the software with the below steps.

ssh pi@192.168.2.XX

sudo -i
hostnamectl set-hostname nodeXX <- change to node01/02/03/04/05 as appropriate
apt-get update
apt-get upgrade -y

curl -s https://download.docker.com/linux/raspbian/gpg | sudo apt-key add -
echo "deb [arch=armhf] https://download.docker.com/linux/raspbian stretch edge" | sudo tee /etc/apt/sources.list.d/socker.list
apt-get update -q
apt-get install -y docker-ce=18.06.0~ce~3-0~raspbian --allow-downgrades
echo "docker-ce hold" | sudo dpkg --set-selections
usermod pi -aG docker

dphys-swapfile swapoff
dphys-swapfile uninstall
update-rc.d dphys-swapfile remove

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update -q
apt-get install -y kubeadm=1.13.1-00 kubectl=1.13.1-00 kubelet=1.13.1-00

reboot

Setup master node01

ssh pi@192.168.2.31

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.2.31 --kubernetes-version v1.13.1

Save the join token and token hash, it will be needed in "Setup slave nodes02-05"

Make local config for pi user, so login as pi on node01

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Check it's working (except the dns pods wont be ready)

kubectl get pods --all-namespaces

Setup kubernetes ovverlay networking

kubectl apply -f https://git.io/weave-kube-1.6
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Setup slave nodes02-05

Join the cluster using the join token and token hash when you ran kubeadm on node01

sudo kubeadm join 192.168.2.31:6443 --token xxxxxxxxxxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

Back on the master node01 check the nodes have joined the cluster and that pods are running:

kubectl get nodes
kubectl get pods --all-namespaces

Deploy dashboard

Deploy the tls disabled version of the dashboard

echo -n 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system' | kubectl apply -f -

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/alternative/kubernetes-dashboard-arm.yaml

To access the dashboard start the proxy on node01:
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'

Then from your pc point your browser at:
http://192.168.2.31:8001/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/

@andyburgin I followed your instructions, but I can't get the master node running...

The kubeadm init [...] did not finish:

[...]
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
        - 'docker ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

I waited for some time until all pod were "Running":

$ kubectl get pods --all-namespaces

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   etcd-msh-master                      1/1     Running   0          105s
kube-system   kube-apiserver-msh-master            1/1     Running   5          107s
kube-system   kube-controller-manager-msh-master   1/1     Running   0          115s
kube-system   kube-scheduler-msh-master            1/1     Running   0          76s

_{(is it possible that kube-dns and kube-proxy are missing?)}

Then I applied the two weave-net files you mentioned:

kubectl apply -f https://git.io/weave-kube-1.6
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

But the weave-net pod will not become "Running"...

ERROR: logging before flag.Parse: E0120 16:25:32.259195   11085 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:319: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0120 16:25:32.267598   11085 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0120 16:25:32.274948   11085 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:318: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

10.96.0.1 seems to be the kubernetes service IP:

$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17m

Oooookayy... I finally managed to get it working \o/

I wrote a small bash script that checks for /etc/kubernetes/manifests/kube-apiserver.yaml to update failureThreshold (new value: 100) and initialDelaySeconds (new value: 1080) as soon as the file exists. The new values are much bigger than they need to be, but they allowed my to get my master node up and running! Whenever I tried to change these values this by hand, the kubeadm init ... command failed.

I just set up a working cluster but couldn't get the master running on an RPI 2. Moved SD card over to a RPI 3 and then kubeadm init ran just fine. The worker node seem to run just fine on the RPI 2.

Wondering if anyone has gotten helm/tiller working in this configuration?

@rnbwkat I got it working but I had to specify a different tiller image, one which was compatible with ARM. The command I used was:

helm init --service-account tiller --tiler-image=jessestuart/tiller:v2.9.0

@janpieper. I’ve run into the “node not found”. looking through all the comments I was going to follow the save steps you did. I wonder what versions of k8s and docker you’ve installed

I tried to setup the cluster following the steps described but still didn't get a succesful kubeadm init. I tried different versions of k8s and docker. Is there somebody who has the steps to get 1.13-3 working with 18.09.0

@janpieper can you share the script?

@janpieper steps worked up until the point everyone mentioned, and rather than the script that polls and zaps the config, I found you can do the same (after the initial failure) by running these commands (lifted from this issue kubernetes/kubeadm#1380)

sudo kubeadm reset
sudo kubeadm init phase certs all
sudo kubeadm init phase kubeconfig all
sudo kubeadm init phase control-plane all --pod-network-cidr 10.244.0.0/16
sudo sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml
sudo sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g'             /etc/kubernetes/manifests/kube-apiserver.yaml
sudo sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g'            /etc/kubernetes/manifests/kube-apiserver.yaml
sudo kubeadm init --v=1 --skip-phases=certs,kubeconfig,control-plane --ignore-preflight-errors=all --pod-network-cidr 10.244.0.0/16

Then I installed flannel.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml

Something that threw me off was the shell demo that Kubernetes provides works fine (kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml) docs here:
https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

But it fails when doing a deployment of nginx from their example here:
https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/

Turns out the nginx image isn't compatible with ARM, once I changed the image to a pi supported image (tobi312/rpi-nginx
) it worked fine! Thanks to everyone here, I finally got my pi cluster going.