Create a gist now

Instantly share code, notes, and snippets.

Embed
K8s on Raspbian

Kubernetes on (vanilla) Raspbian Lite

Yes - you can create a Kubernetes cluster with Raspberry Pis with the default operating system called Raspbian. This means you can carry on using all the tools and packages you're used to with the officially-supported OS.

Pre-reqs:

  • You must use an RPi 2 or 3 for use with Kubernetes
  • I'm assuming you're using wired ethernet (Wi-Fi also works, but it's not recommended)

Master node setup

  • Flash Raspbian to a fresh SD card.

You can use Etcher.io to burn the SD card.

Before booting set up an empty file called ssh in /boot/ on the SD card.

Use Raspbian Stretch Lite

Update: I previously recommended downloading Raspbian Jessie instead of Stretch. At time of writing (3 Jan 2018) Stretch is now fully compatible.

https://www.raspberrypi.org/downloads/raspbian/

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-master-1 or similar and then reboot.

  • Set a static IP address

It's not fun when your cluste breaks because the IP of your master changed. Let's fix that problem ahead of time:

cat >> /etc/dhcpcd.conf

Paste this block:

profile static_eth0
static ip_address=192.168.0.100/24
static routers=192.168.0.1
static domain_name_servers=8.8.8.8

Hit Control + D.

Change 100 for 101, 102, 103 etc.

You may also need to make a reservation on your router's DHCP table so these addresses don't get given out to other devices on your network.

  • Install Docker

This installs 17.12 or newer.

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
  • Disable swap

For Kubernetes 1.7 and newer you will get an error if swap space is enabled.

Turn off swap:

$ sudo dphys-swapfile swapoff && \
  sudo dphys-swapfile uninstall && \
  sudo update-rc.d dphys-swapfile remove

This should now show no entries:

$ sudo swapon --summary
  • Edit /boot/cmdline.txt

Add this text at the end of the line, but don't create any new lines:

cgroup_enable=cpuset cgroup_enable=memory

Some people in the comments suggest cgroup_memory=memory should now be: cgroup_memory=1.

Now reboot - do not skip this step.

  • Add repo lists & install kubeadm
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubeadm

I realise this says 'xenial' in the apt listing, don't worry. It still works.

  • You now have two new commands installed:

  • kubeadm - used to create new clusters or join an existing one

  • kubectl - the CLI administration tool for Kubernetes

  • Initialize your master node:

$ sudo kubeadm init --token-ttl=0

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

Optionally also pass --apiserver-advertise-address=192.168.0.27 with the IP of the Pi.

Note: This step will take a long time, even up to 15 minutes.

After the init is complete run the snippet given to you on the command-line:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

This step takes the key generated for cluster administration and makes it available in a default location for use with kubectl.

  • Now save your join-token

Your join token is valid for 24 hours, so save it into a text file. Here's an example of mine:

$ kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea
  • Check everything worked:
$ kubectl get pods --namespace=kube-system
NAME                           READY     STATUS    RESTARTS   AGE                
etcd-of-2                      1/1       Running   0          12m                
kube-apiserver-of-2            1/1       Running   2          12m                
kube-controller-manager-of-2   1/1       Running   1          11m                
kube-dns-66ffd5c588-d8292      3/3       Running   0          11m                
kube-proxy-xcj5h               1/1       Running   0          11m                
kube-scheduler-of-2            1/1       Running   0          11m                
weave-net-zz9rz                2/2       Running   0          5m 

You should see the "READY" count showing as 1/1 for all services as above. DNS uses three pods, so you'll see 3/3 for that.

  • Setup networking

Install Weave network driver

$ kubectl apply -f \
 "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Join other nodes

On the other RPis, repeat everything apart from kubeadm init.

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-worker-1 or similar and then reboot.

  • Join the cluster

Replace the token / IP for the output you got from the master node:

$ sudo kubeadm join --token 1fd0d8.67e7083ed7ec08f3 192.168.0.27:6443

You can now run this on the master:

$ kubectl get nodes
NAME      STATUS     AGE       VERSION
k8s-1     Ready      5m        v1.7.4
k8s-2     Ready      10m       v1.7.4

Deploy a container

This container will expose a HTTP port and convert Markdown to HTML. Just post a body to it via curl - follow the instructions below.

function.yml

apiVersion: v1
kind: Service
metadata:
  name: markdownrender
  labels:
    app: markdownrender
spec:
  type: NodePort
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 31118
  selector:
    app: markdownrender
---
apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: markdownrender
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: markdownrender
    spec:
      containers:
      - name: markdownrender
        image: functions/markdownrender:latest-armhf
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP

Deploy and test:

$ kubectl create -f function.yml

Once the Docker image has been pulled from the hub and the Pod is running you can access it via curl:

$ curl -4 http://127.0.0.1:31118 -d "# test"
<p><h1>test</h1></p>

If you want to call the service from a remote machine such as your laptop then use the IP address of your Kubernetes master node and try the same again.

Start up the dashboard

The dashboard can be useful for visualising the state and health of your system but it does require the equivalent of "root" in the cluster. If you want to proceed you should first run in a ClusterRole from the docs.

echo -n 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system' | kubectl apply -f -

This is the development/alternative dashboard which has TLS disabled and is easier to use.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard-arm.yaml

You can then find the IP and port via kubectl get svc -n kube-system. To access this from your laptop you will need to use kubectl proxy and navigate to http://localhost:8001/ on the master, or tunnel to this address with ssh.

Remove the test deployment

Now on the Kubernetes master remove the test deployment:

$ kubectl delete -f function.yml

Moving on

Now head back over to the tutorial and deploy OpenFaaS to put the cluster through its paces.

#!/bin/sh
# This installs the base instructions up to the point of joining / creating a cluster
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update -q && \
sudo apt-get install -qy kubeadm
echo Adding " cgroup_enable=cpuset cgroup_memory=1" to /boot/cmdline.txt
sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_memory=1"
echo $orig | sudo tee /boot/cmdline.txt
echo Please reboot

Use this to setup quickly

# curl -sL \
 https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/b04f1e9250c61a8ff554bfe3475b6dd050062484/prep.sh \
 | sudo sh
@Lewiscowles1986

This comment has been minimized.

Show comment
Hide comment
@Lewiscowles1986

Lewiscowles1986 Oct 12, 2017

This is great. It'd be very cool to have this operate unattended. (or as unattended as possible)

Lewiscowles1986 commented Oct 12, 2017

This is great. It'd be very cool to have this operate unattended. (or as unattended as possible)

@shanselman

This comment has been minimized.

Show comment
Hide comment

shanselman commented Oct 25, 2017

@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman Oct 25, 2017

The swapfile turns back on when you reboot unless you

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

The swapfile turns back on when you reboot unless you

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove
@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman Oct 25, 2017

For this line curl localhost:31118 -d "# test" I had to use the full host name. Localhost is still 127.0.0.1 and it doesn't seem to be listening

For this line curl localhost:31118 -d "# test" I had to use the full host name. Localhost is still 127.0.0.1 and it doesn't seem to be listening

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Oct 25, 2017

Kubernetes please stop changing every other day 👎

Owner

alexellis commented Oct 25, 2017

Kubernetes please stop changing every other day 👎

@olavt

This comment has been minimized.

Show comment
Hide comment
@olavt

olavt Oct 29, 2017

I followed the instructions and got everything installed on a 2x Raspberry PI 3 cluster (1 master and 1 node). But, I have not been able to get the Dashboard up and running.

olavt@k8s-master-1:~ $ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 5h
kubernetes-dashboard ClusterIP 10.104.85.132 443/TCP 4h
olavt@k8s-master-1:~ $ kubectl proxy
Starting to serve on 127.0.0.1:8001

What is the Url I should use from my other computer to connect to the Dashboard?

olavt commented Oct 29, 2017

I followed the instructions and got everything installed on a 2x Raspberry PI 3 cluster (1 master and 1 node). But, I have not been able to get the Dashboard up and running.

olavt@k8s-master-1:~ $ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 5h
kubernetes-dashboard ClusterIP 10.104.85.132 443/TCP 4h
olavt@k8s-master-1:~ $ kubectl proxy
Starting to serve on 127.0.0.1:8001

What is the Url I should use from my other computer to connect to the Dashboard?

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Oct 30, 2017

OK for the dashboard you need to run kubectl on your own PC/laptop. Maybe an SSH tunnel would work?

ssh -L 8001:127.0.01:8001 pi@k8s-master-1.local

then try 127.0.0.1:8001 on your local machine

Owner

alexellis commented Oct 30, 2017

OK for the dashboard you need to run kubectl on your own PC/laptop. Maybe an SSH tunnel would work?

ssh -L 8001:127.0.01:8001 pi@k8s-master-1.local

then try 127.0.0.1:8001 on your local machine

@olavt

This comment has been minimized.

Show comment
Hide comment
@olavt

olavt Oct 30, 2017

That didn't work for me.

olavt commented Oct 30, 2017

That didn't work for me.

@steini

This comment has been minimized.

Show comment
Hide comment
@steini

steini Nov 3, 2017

First of all thanks for the detailed setup process.

After updating raspbian i ran into the problem that sudo kubeadm join raised the error CGROUPS_MEMORY: missing. The boot option is no longer cgroup_enable=memory but cgroup_memory=1

See https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086#p57035 and raspberrypi/linux@ba742b5

steini commented Nov 3, 2017

First of all thanks for the detailed setup process.

After updating raspbian i ran into the problem that sudo kubeadm join raised the error CGROUPS_MEMORY: missing. The boot option is no longer cgroup_enable=memory but cgroup_memory=1

See https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086#p57035 and raspberrypi/linux@ba742b5

@movingbytes

This comment has been minimized.

Show comment
Hide comment
@movingbytes

movingbytes Nov 5, 2017

after installation the status of all pods in namespace kube-system is pending except kube-proxy (NodeLost). Any ideas?
Using docker 17.10 and K8S 1.8.2

movingbytes commented Nov 5, 2017

after installation the status of all pods in namespace kube-system is pending except kube-proxy (NodeLost). Any ideas?
Using docker 17.10 and K8S 1.8.2

@borrillis

This comment has been minimized.

Show comment
Hide comment
@borrillis

borrillis Nov 15, 2017

My dashboard wouldn't work properly until I did:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccou nt=kube-system:kubernetes-dashboard

I could get to the dashboard using kubectl proxy and opened the url http://localhost:8001/ui in a browser, but it couldn't get any data from the api.

borrillis commented Nov 15, 2017

My dashboard wouldn't work properly until I did:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccou nt=kube-system:kubernetes-dashboard

I could get to the dashboard using kubectl proxy and opened the url http://localhost:8001/ui in a browser, but it couldn't get any data from the api.

@francis2211

This comment has been minimized.

Show comment
Hide comment
@francis2211

francis2211 Nov 22, 2017

@alexellis it should be cgroup_memory=1 not cgroup_enable=memory

@alexellis it should be cgroup_memory=1 not cgroup_enable=memory

@krystan

This comment has been minimized.

Show comment
Hide comment
@krystan

krystan Dec 12, 2017

cgroup_enable=memory seems to be fine under kernel 4.9.35-v7.

krystan commented Dec 12, 2017

cgroup_enable=memory seems to be fine under kernel 4.9.35-v7.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Dec 23, 2017

I've updated the instructions for the newer RPi kernel.

Owner

alexellis commented Dec 23, 2017

I've updated the instructions for the newer RPi kernel.

@charliesolomon

This comment has been minimized.

Show comment
Hide comment
@charliesolomon

charliesolomon Jan 1, 2018

I had to run the "set up networking" step (install weave) in order to get "Running" back from the 3 DNS pods. Before that, they reported "Pending"... move the "set up networking" step before "check everything worked" in your instructions?

charliesolomon commented Jan 1, 2018

I had to run the "set up networking" step (install weave) in order to get "Running" back from the 3 DNS pods. Before that, they reported "Pending"... move the "set up networking" step before "check everything worked" in your instructions?

@teekay

This comment has been minimized.

Show comment
Hide comment
@teekay

teekay Jan 4, 2018

I was also only able to get both Master and 1 "slave" node to the Ready status when I first installed the "weave" networking on the master, and only after that joined the worker. K8s version 1.9.

teekay commented Jan 4, 2018

I was also only able to get both Master and 1 "slave" node to the Ready status when I first installed the "weave" networking on the master, and only after that joined the worker. K8s version 1.9.

@evnsio

This comment has been minimized.

Show comment
Hide comment
@evnsio

evnsio Jan 8, 2018

Has anyone experienced an issue kubeadm? I'm getting Illegal instruction when I try to run it.

Running on Raspian Stretch 4.9.59+.

evnsio commented Jan 8, 2018

Has anyone experienced an issue kubeadm? I'm getting Illegal instruction when I try to run it.

Running on Raspian Stretch 4.9.59+.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Jan 8, 2018

@caedev - no, you are definitely using a Raspberry Pi 2 or 3?

Owner

alexellis commented Jan 8, 2018

@caedev - no, you are definitely using a Raspberry Pi 2 or 3?

@evnsio

This comment has been minimized.

Show comment
Hide comment
@evnsio

evnsio Jan 8, 2018

Sorry, just realised I was ssh'ing into the wrong pi; this works absolutely fine on my Pi 2. Thanks for writing this @alexellis - much appreciated.

evnsio commented Jan 8, 2018

Sorry, just realised I was ssh'ing into the wrong pi; this works absolutely fine on my Pi 2. Thanks for writing this @alexellis - much appreciated.

@haebler

This comment has been minimized.

Show comment
Hide comment
@haebler

haebler Jan 9, 2018

same experience as @charliesolomon, DNS doesn't come up until you install the weave network driver.

Basically change to below:

  • Install network driver kubectl apply -f https://git.io/weave-kube-1.6
  • Check status: kubectl get pods --namespace=kube-system

Note: Be patient on the 2nd step, the weave driver comes up first. Once it is Running DNS goes from Pending to ContainerCreating to Running.

haebler commented Jan 9, 2018

same experience as @charliesolomon, DNS doesn't come up until you install the weave network driver.

Basically change to below:

  • Install network driver kubectl apply -f https://git.io/weave-kube-1.6
  • Check status: kubectl get pods --namespace=kube-system

Note: Be patient on the 2nd step, the weave driver comes up first. Once it is Running DNS goes from Pending to ContainerCreating to Running.

@chris-short

This comment has been minimized.

Show comment
Hide comment
@chris-short

chris-short Jan 13, 2018

In the dashboard section, you might want to mention the need for rbac: https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

In the dashboard section, you might want to mention the need for rbac: https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

@DazWilkin

This comment has been minimized.

Show comment
Hide comment
@DazWilkin

DazWilkin Jan 20, 2018

An excellent guide, thank you!

The instructions are unclear for accessing the cluster remotely but are explained here:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master

Effectively make a copy on the local machine of the master's /etc/kubernetes/admin.conf perhaps named k8s_pi.conf

Then kubectl --kubeconfig ./k8s_pi.conf get nodes

Or, per your example to create a proxy: kubectl --kubeconfig ./k8s_pi.conf proxy &

To avoid specifying --kubeconfig repeatedly, you can merge the contents of k8s_pi.conf into the default config ~/.kube/config

DazWilkin commented Jan 20, 2018

An excellent guide, thank you!

The instructions are unclear for accessing the cluster remotely but are explained here:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master

Effectively make a copy on the local machine of the master's /etc/kubernetes/admin.conf perhaps named k8s_pi.conf

Then kubectl --kubeconfig ./k8s_pi.conf get nodes

Or, per your example to create a proxy: kubectl --kubeconfig ./k8s_pi.conf proxy &

To avoid specifying --kubeconfig repeatedly, you can merge the contents of k8s_pi.conf into the default config ~/.kube/config

@DazWilkin

This comment has been minimized.

Show comment
Hide comment
@DazWilkin

DazWilkin Jan 20, 2018

Follow-up (kubeadm) question: What's the process to shutdown and restart the cluster?

kubeadm reset seems more of a teardown.

What if you'd just like to shut the cluster down correctly to then shutdown the underlying Pis and restart subsequently?

Follow-up (kubeadm) question: What's the process to shutdown and restart the cluster?

kubeadm reset seems more of a teardown.

What if you'd just like to shut the cluster down correctly to then shutdown the underlying Pis and restart subsequently?

@denhamparry

This comment has been minimized.

Show comment
Hide comment
@denhamparry

denhamparry Jan 29, 2018

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

  • Edit kubernetes-dashboard service.
$ kubectl -n kube-system edit service kubernetes-dashboard
  • You should the see yaml representation of the service. Change type: ClusterIP to type: NodePort and save file.
  • Check port on which Dashboard was exposed.
$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m
  • Create a proxy to view within your browser
$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

Thanks again Alex!

denhamparry commented Jan 29, 2018

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

  • Edit kubernetes-dashboard service.
$ kubectl -n kube-system edit service kubernetes-dashboard
  • You should the see yaml representation of the service. Change type: ClusterIP to type: NodePort and save file.
  • Check port on which Dashboard was exposed.
$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m
  • Create a proxy to view within your browser
$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

Thanks again Alex!

@yosephbuitrago

This comment has been minimized.

Show comment
Hide comment
@yosephbuitrago

yosephbuitrago Mar 2, 2018

Hi, Alex, thank for share this tutorial. I builded a raspberry pi cluster and is running kubernetes and OpenFaas as expected it. the only thing is that the auto-scaling don't in OpenfaaS does work! on my computer works but it does work in the cluster!

Do I have to change something in the .yml files? I check them but they look the same.

Hi, Alex, thank for share this tutorial. I builded a raspberry pi cluster and is running kubernetes and OpenFaas as expected it. the only thing is that the auto-scaling don't in OpenfaaS does work! on my computer works but it does work in the cluster!

Do I have to change something in the .yml files? I check them but they look the same.

@johndcollins

This comment has been minimized.

Show comment
Hide comment
@johndcollins

johndcollins Mar 13, 2018

I had to add both cgroup_memory=memory AND cgroup_memory=1 to the cmdline.txt to get it to work.

I had to add both cgroup_memory=memory AND cgroup_memory=1 to the cmdline.txt to get it to work.

@bilalAchahbar

This comment has been minimized.

Show comment
Hide comment
@bilalAchahbar

bilalAchahbar Mar 21, 2018

Great and very understandable post !!
I've set the kubernetes dashboard through the Nodeport and can access it on my host but the certificates still give a lot of issues.
Is it possible to use Let's encrypt for the kubernetes dashboard ?
As i am new to the concept of certificates through websites can anyone point me how i can do this through an authomatic service like let's encrypt.

Great and very understandable post !!
I've set the kubernetes dashboard through the Nodeport and can access it on my host but the certificates still give a lot of issues.
Is it possible to use Let's encrypt for the kubernetes dashboard ?
As i am new to the concept of certificates through websites can anyone point me how i can do this through an authomatic service like let's encrypt.

@Jickelsen

This comment has been minimized.

Show comment
Hide comment
@Jickelsen

Jickelsen Apr 1, 2018

Thanks for the fantastic guide, I had great fun learning about all these topics in practice over a weekend. As a switch I'm having great success with the 5-port D-Link DGS-1005D, newer versions of which use mini-USB for power.

I had issues getting Weave to work on Raspbian Stretch and the Pi3 B+. Shortly after running
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
the master and connected nodes would reboot unexpectedly, and would leave the cluster in an error state.
I ended up using flannel:

  • Use --pod-network-cidr=10.244.0.0/16 when initializing the cluster
    $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=<internal master ip> --pod-network-cidr=10.244.0.0/16
  • Install flannel with
    $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

I also managed to set up the master as a router, with Wifi on the WAN side, using the steps in this particular post https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=132674&start=50#p1252309

Thanks for the fantastic guide, I had great fun learning about all these topics in practice over a weekend. As a switch I'm having great success with the 5-port D-Link DGS-1005D, newer versions of which use mini-USB for power.

I had issues getting Weave to work on Raspbian Stretch and the Pi3 B+. Shortly after running
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
the master and connected nodes would reboot unexpectedly, and would leave the cluster in an error state.
I ended up using flannel:

  • Use --pod-network-cidr=10.244.0.0/16 when initializing the cluster
    $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=<internal master ip> --pod-network-cidr=10.244.0.0/16
  • Install flannel with
    $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

I also managed to set up the master as a router, with Wifi on the WAN side, using the steps in this particular post https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=132674&start=50#p1252309

@DerfOh

This comment has been minimized.

Show comment
Hide comment
@DerfOh

DerfOh Apr 7, 2018

Thanks @Jickelsen I had to do the same.
In addition to that I also my nodes stuck in a not ready state due to the following error:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Fixed this by removing KUBELET_NETWORK_ARGS from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf then rebooting according to this issue: kubernetes/kubernetes#38653

I was then able to run
curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
without issue.

DerfOh commented Apr 7, 2018

Thanks @Jickelsen I had to do the same.
In addition to that I also my nodes stuck in a not ready state due to the following error:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Fixed this by removing KUBELET_NETWORK_ARGS from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf then rebooting according to this issue: kubernetes/kubernetes#38653

I was then able to run
curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
without issue.

@exp0nge

This comment has been minimized.

Show comment
Hide comment
@exp0nge

exp0nge Apr 7, 2018

I can't seem to get init past [init] This might take a minute or longer if the control plane images have to be pulled.. There's so many issues on kubernetes/kubadm about this. I used a fresh install of rasbian lite (march 3rd update). Anyone else get this or know a workaround?

exp0nge commented Apr 7, 2018

I can't seem to get init past [init] This might take a minute or longer if the control plane images have to be pulled.. There's so many issues on kubernetes/kubadm about this. I used a fresh install of rasbian lite (march 3rd update). Anyone else get this or know a workaround?

@rashray

This comment has been minimized.

Show comment
Hide comment
@rashray

rashray Apr 8, 2018

Thank You Alex. Very detailed steps. I am using a b plus Pi as a master. Any idea why the Pi goes dead slow on initiating the Kube master.

rashray commented Apr 8, 2018

Thank You Alex. Very detailed steps. I am using a b plus Pi as a master. Any idea why the Pi goes dead slow on initiating the Kube master.

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Apr 9, 2018

Thanks @Jickelsen & @DerfOh! I spent all my spare time in the last three weeks trying to get kubernetes to work again. The gist worked great at Xmas but now once you get weavenet up on the node & synced to the master, both crash with an oops:
kernel:[ 4286.584219] Internal error: Oops: 80000007 [#1] SMP ARM
kernel:[ 4287.037510] Process weaver (pid: 13327, stack limit = 0x9bb12210)
kernel:[ 4287.059886] Stack: (0x9bb139f0 to 0x9bb14000)
kernel:[ 4287.081698] 39e0: 00000000 00000000 5001a8c0 9bb13a88
kernel:[ 4287.125181] 3a00: 0000801a 0000db84 9bab4150 9bab4118 9bb13d2c 7f63bad0 00000001 9bb13a5c
Finally I can finish writing my ansible play-book to automate the whole thing.

Thanks @Jickelsen & @DerfOh! I spent all my spare time in the last three weeks trying to get kubernetes to work again. The gist worked great at Xmas but now once you get weavenet up on the node & synced to the master, both crash with an oops:
kernel:[ 4286.584219] Internal error: Oops: 80000007 [#1] SMP ARM
kernel:[ 4287.037510] Process weaver (pid: 13327, stack limit = 0x9bb12210)
kernel:[ 4287.059886] Stack: (0x9bb139f0 to 0x9bb14000)
kernel:[ 4287.081698] 39e0: 00000000 00000000 5001a8c0 9bb13a88
kernel:[ 4287.125181] 3a00: 0000801a 0000db84 9bab4150 9bab4118 9bb13d2c 7f63bad0 00000001 9bb13a5c
Finally I can finish writing my ansible play-book to automate the whole thing.

@carlosroman

This comment has been minimized.

Show comment
Hide comment
@carlosroman

carlosroman Apr 10, 2018

I've had strange issues with getting weavenet running

NAMESPACE     NAME                                      READY     STATUS              RESTARTS   AGE
kube-system   weave-net-8t7zd                           2/2       Running             494        1d
kube-system   weave-net-gpcnj                           1/2       CrashLoopBackOff    417        1d
kube-system   weave-net-m7tnn                           1/2       ImageInspectError   0          1d
kube-system   weave-net-qmjwk                           1/2       ImageInspectError   0          1d
kube-system   weave-net-rvwpj                           2/2       Running             534        1d

Still debuging it but it has been a fun learning experience getting K8s running on a Raspberry Pi cluster.

@micedwards, I ended up writting an ansible playbook as kept rebuilding my cluster to see why weave kept crashing. Wrote it after running kubeadm reset on the master accidently or on a node. Now have a playbook that sets up my cluster and adds nodes to it as well. Any improvements would be great, https://github.com/carlosroman/ansible-k8s-raspberry-playbook.

I've had strange issues with getting weavenet running

NAMESPACE     NAME                                      READY     STATUS              RESTARTS   AGE
kube-system   weave-net-8t7zd                           2/2       Running             494        1d
kube-system   weave-net-gpcnj                           1/2       CrashLoopBackOff    417        1d
kube-system   weave-net-m7tnn                           1/2       ImageInspectError   0          1d
kube-system   weave-net-qmjwk                           1/2       ImageInspectError   0          1d
kube-system   weave-net-rvwpj                           2/2       Running             534        1d

Still debuging it but it has been a fun learning experience getting K8s running on a Raspberry Pi cluster.

@micedwards, I ended up writting an ansible playbook as kept rebuilding my cluster to see why weave kept crashing. Wrote it after running kubeadm reset on the master accidently or on a node. Now have a playbook that sets up my cluster and adds nodes to it as well. Any improvements would be great, https://github.com/carlosroman/ansible-k8s-raspberry-playbook.

@ScubaJimmer

This comment has been minimized.

Show comment
Hide comment
@ScubaJimmer

ScubaJimmer Apr 12, 2018

Good Evening.

I have been having trouble getting kubernetes+docker running as a 2 RPI cluster. My master node continues to reboot. I followed all the steps above to configure two fresh nodes, except I used my router to establish a static IP for my master and worker node. Interestingly my worker node seems stable so far right now. In previous attempts, when I had set up 4 additional nodes they too became unstable.
The master node was stable before I joined my first worker node

Docker version: 18.03.0-ce, build 0520e24
Kubernetes version : 1.10

Master node:

pi@k8boss1:~ $ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE
etcd-k8boss1 1/1 Running 33 1d
kube-apiserver-k8boss1 1/1 Running 34 1d
kube-controller-manager-k8boss1 1/1 Running 34 1d
kube-dns-686d6fb9c-hwxxw 0/3 Error 0 1d
kube-proxy-8v8z7 0/1 Error 33 1d
kube-proxy-dgqxp 1/1 Running 0 1h
kube-scheduler-k8boss1 1/1 Running 34 1d
weave-net-ggxwp 2/2 Running 0 1h
weave-net-l7xsl 0/2 Error 71 1d

pi@k8boss1:~ $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8boss1 Ready master 1d v1.10.0
k8worker1 Ready 1h v1.10.0

pi@k8boss1:~ $ uptime
01:48:50 up 0 min, 1 user, load average: 1.37, 0.41, 0.14
pi@k8boss1:~ $

Worker:
pi@k8worker1:~ $ uptime
01:49:35 up 1:58, 1 user, load average: 0.11, 0.21, 0.19
pi@k8worker1:~ $

Any thoughts?

ScubaJimmer commented Apr 12, 2018

Good Evening.

I have been having trouble getting kubernetes+docker running as a 2 RPI cluster. My master node continues to reboot. I followed all the steps above to configure two fresh nodes, except I used my router to establish a static IP for my master and worker node. Interestingly my worker node seems stable so far right now. In previous attempts, when I had set up 4 additional nodes they too became unstable.
The master node was stable before I joined my first worker node

Docker version: 18.03.0-ce, build 0520e24
Kubernetes version : 1.10

Master node:

pi@k8boss1:~ $ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE
etcd-k8boss1 1/1 Running 33 1d
kube-apiserver-k8boss1 1/1 Running 34 1d
kube-controller-manager-k8boss1 1/1 Running 34 1d
kube-dns-686d6fb9c-hwxxw 0/3 Error 0 1d
kube-proxy-8v8z7 0/1 Error 33 1d
kube-proxy-dgqxp 1/1 Running 0 1h
kube-scheduler-k8boss1 1/1 Running 34 1d
weave-net-ggxwp 2/2 Running 0 1h
weave-net-l7xsl 0/2 Error 71 1d

pi@k8boss1:~ $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8boss1 Ready master 1d v1.10.0
k8worker1 Ready 1h v1.10.0

pi@k8boss1:~ $ uptime
01:48:50 up 0 min, 1 user, load average: 1.37, 0.41, 0.14
pi@k8boss1:~ $

Worker:
pi@k8worker1:~ $ uptime
01:49:35 up 1:58, 1 user, load average: 0.11, 0.21, 0.19
pi@k8worker1:~ $

Any thoughts?

@peterkingsbury

This comment has been minimized.

Show comment
Hide comment
@peterkingsbury

peterkingsbury Apr 19, 2018

On Raspbian Stretch Lite, the installation halts during the master setup phase (sudo kubeadm init --token-ttl=0) with the following output:

[init] This might take a minute or longer if the control plane images have to be pulled.

I found it necessary to install Kubernetes 1.9.6:

sudo apt-get install -y kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

Took 552.013509 seconds to complete, but it's up and running now!

Thanks for a great tutorial!

On Raspbian Stretch Lite, the installation halts during the master setup phase (sudo kubeadm init --token-ttl=0) with the following output:

[init] This might take a minute or longer if the control plane images have to be pulled.

I found it necessary to install Kubernetes 1.9.6:

sudo apt-get install -y kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

Took 552.013509 seconds to complete, but it's up and running now!

Thanks for a great tutorial!

@danielvaughan

This comment has been minimized.

Show comment
Hide comment
@danielvaughan

danielvaughan Apr 21, 2018

I am running into the same problems as @carlosroman and @micedwards after applying weave on a 4 RPi 3 cluster:

Raspbian GNU/Linux 9 (stretch)
Docker version 18.04.0-ce, build 3d479c0
Kubernetes v1.10.1

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS              RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running             22         10h
kube-apiserver-k8s-master-1            1/1       Running             39         10h
kube-controller-manager-k8s-master-1   1/1       Running             13         10h
kube-dns-686d6fb9c-qn2mp               0/3       Pending             0          10h
kube-proxy-6dlz4                       1/1       Running             11         9h
kube-proxy-7s977                       1/1       Running             2          9h
kube-proxy-q7jlh                       1/1       Running             11         10h
kube-proxy-qdmp7                       1/1       Running             2          9h
kube-scheduler-k8s-master-1            1/1       Running             13         10h
weave-net-5scxb                        2/2       Running             1          2m
weave-net-5vxzw                        1/2       CrashLoopBackOff    4          2m
weave-net-jmlzc                        1/2       ImageInspectError   0          2m
weave-net-xc2f8                        1/2       ImageInspectError   1          2m
pi@k8s-master-1:~ $
Message from syslogd@k8s-master-1 at Apr 22 08:04:14 ...
 kernel:[  155.252476] Internal error: Oops: 80000007 [#1] SMP ARM

I am having more luck with flannel

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.1.1.200 --pod-network-cidr=10.244.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          5m
kube-apiserver-k8s-master-1            1/1       Running   0          5m
kube-controller-manager-k8s-master-1   1/1       Running   0          5m
kube-dns-686d6fb9c-xxrbg               3/3       Running   0          5m
kube-flannel-ds-gxt4n                  1/1       Running   0          23s
kube-flannel-ds-hngfv                  1/1       Running   0          2m
kube-flannel-ds-mgxdn                  1/1       Running   0          1m
kube-flannel-ds-qb8ch                  1/1       Running   0          3m
kube-proxy-4kxr8                       1/1       Running   0          1m
kube-proxy-54q5g                       1/1       Running   0          5m
kube-proxy-7zb4p                       1/1       Running   0          23s
kube-proxy-rwvp4                       1/1       Running   0          2m
kube-scheduler-k8s-master-1            1/1       Running   0          5m

danielvaughan commented Apr 21, 2018

I am running into the same problems as @carlosroman and @micedwards after applying weave on a 4 RPi 3 cluster:

Raspbian GNU/Linux 9 (stretch)
Docker version 18.04.0-ce, build 3d479c0
Kubernetes v1.10.1

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS              RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running             22         10h
kube-apiserver-k8s-master-1            1/1       Running             39         10h
kube-controller-manager-k8s-master-1   1/1       Running             13         10h
kube-dns-686d6fb9c-qn2mp               0/3       Pending             0          10h
kube-proxy-6dlz4                       1/1       Running             11         9h
kube-proxy-7s977                       1/1       Running             2          9h
kube-proxy-q7jlh                       1/1       Running             11         10h
kube-proxy-qdmp7                       1/1       Running             2          9h
kube-scheduler-k8s-master-1            1/1       Running             13         10h
weave-net-5scxb                        2/2       Running             1          2m
weave-net-5vxzw                        1/2       CrashLoopBackOff    4          2m
weave-net-jmlzc                        1/2       ImageInspectError   0          2m
weave-net-xc2f8                        1/2       ImageInspectError   1          2m
pi@k8s-master-1:~ $
Message from syslogd@k8s-master-1 at Apr 22 08:04:14 ...
 kernel:[  155.252476] Internal error: Oops: 80000007 [#1] SMP ARM

I am having more luck with flannel

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.1.1.200 --pod-network-cidr=10.244.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          5m
kube-apiserver-k8s-master-1            1/1       Running   0          5m
kube-controller-manager-k8s-master-1   1/1       Running   0          5m
kube-dns-686d6fb9c-xxrbg               3/3       Running   0          5m
kube-flannel-ds-gxt4n                  1/1       Running   0          23s
kube-flannel-ds-hngfv                  1/1       Running   0          2m
kube-flannel-ds-mgxdn                  1/1       Running   0          1m
kube-flannel-ds-qb8ch                  1/1       Running   0          3m
kube-proxy-4kxr8                       1/1       Running   0          1m
kube-proxy-54q5g                       1/1       Running   0          5m
kube-proxy-7zb4p                       1/1       Running   0          23s
kube-proxy-rwvp4                       1/1       Running   0          2m
kube-scheduler-k8s-master-1            1/1       Running   0          5m
@peterska

This comment has been minimized.

Show comment
Hide comment
@peterska

peterska Apr 29, 2018

If you use flannel instead of weave networking the kernel oops does not occur. You can install flannel using
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
Make sure you install flannel before joining any nodes to the cluster.

If you use flannel instead of weave networking the kernel oops does not occur. You can install flannel using
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
Make sure you install flannel before joining any nodes to the cluster.

@mcglinn

This comment has been minimized.

Show comment
Hide comment
@mcglinn

mcglinn Apr 29, 2018

I have the same issue - hanging at 'sudo kubeadm init' on master at line:
'[init] This might take a minute or longer if the control plane images have to be pulled.'
Tried so many different versions and options but still no luck.

I am using a raspberry pi 2 B+

Have used various raspbians Wheezy/Stretch various kubernetes up to latest (inc. 1.9.6 as suggested by PeterKing above) and various docker versions.

Anyone with this running on raspberry Pi 2 with recent raspbian, able to share version of all components (raspbian + kubernetes + docker)?

Please im sick of reflashing my SD :)

mcglinn commented Apr 29, 2018

I have the same issue - hanging at 'sudo kubeadm init' on master at line:
'[init] This might take a minute or longer if the control plane images have to be pulled.'
Tried so many different versions and options but still no luck.

I am using a raspberry pi 2 B+

Have used various raspbians Wheezy/Stretch various kubernetes up to latest (inc. 1.9.6 as suggested by PeterKing above) and various docker versions.

Anyone with this running on raspberry Pi 2 with recent raspbian, able to share version of all components (raspbian + kubernetes + docker)?

Please im sick of reflashing my SD :)

@Creamen

This comment has been minimized.

Show comment
Hide comment
@Creamen

Creamen Apr 29, 2018

Many many thanks for this bootstrap introduction !

I was facing issues with the latest version ( v1.10.2 - 28-04-2018 ) and after loosing some (more) hair - kube-apiserver was dying in loop ultimately leading to fail of kudeadm init - , I tried to downgrade both kubeadm and kubelet to 1.9.7-00 and - for now as it's a fresh start - things are up on my RPI3 cluster ... Cross finger :)
My cluster is a result of this simple :

sudo kubeadm init

Creamen commented Apr 29, 2018

Many many thanks for this bootstrap introduction !

I was facing issues with the latest version ( v1.10.2 - 28-04-2018 ) and after loosing some (more) hair - kube-apiserver was dying in loop ultimately leading to fail of kudeadm init - , I tried to downgrade both kubeadm and kubelet to 1.9.7-00 and - for now as it's a fresh start - things are up on my RPI3 cluster ... Cross finger :)
My cluster is a result of this simple :

sudo kubeadm init

@petershaw

This comment has been minimized.

Show comment
Hide comment

Kudo for @Jickelsen

@bjornramberg

This comment has been minimized.

Show comment
Hide comment
@bjornramberg

bjornramberg May 2, 2018

@alexellis: Thanks for a great guide! 👍
@Creamen: was facing exact same issue with same version, running on rpi3b+. The apiserver was looping, dying and geting stuck in a timeout on kubeadm init. After a downgrade it went through.

@alexellis: Thanks for a great guide! 👍
@Creamen: was facing exact same issue with same version, running on rpi3b+. The apiserver was looping, dying and geting stuck in a timeout on kubeadm init. After a downgrade it went through.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis May 4, 2018

Im seeing Weavenet being mentioned but I can’t see that anyone has logged an issue with kubeadm or weave - I’d suggest doing that if you are seeing unexpected behaviour with newer versions of the components.

The init step can take several minutes.

Owner

alexellis commented May 4, 2018

Im seeing Weavenet being mentioned but I can’t see that anyone has logged an issue with kubeadm or weave - I’d suggest doing that if you are seeing unexpected behaviour with newer versions of the components.

The init step can take several minutes.

@aaronkjones

This comment has been minimized.

Show comment
Hide comment
@aaronkjones

aaronkjones May 5, 2018

I have attempted to get Weave Net to work with k8s v1.10.2 and exausted all options it seems. I could only get it working with flannel as the CNI. I also had to add and change some commands.

I forked your gist and made the modifications including changing the script (prep.sh)

Try it out
https://github.com/aaronkjones/rpi-k8s-node-prep

pi@k8s-master-1:~ $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    2h        v1.10.2
k8s-slave-1    Ready     <none>    2h        v1.10.2
k8s-slave-2    Ready     <none>    33m       v1.10.2
k8s-slave-3    Ready     <none>    33m       v1.10.2
pi@k8s-master-1:~ $ kubectl get pods -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          2h
kube-apiserver-k8s-master-1            1/1       Running   0          2h
kube-controller-manager-k8s-master-1   1/1       Running   0          2h
kube-dns-686d6fb9c-glz8x               3/3       Running   0          2h
kube-flannel-ds-5grwv                  1/1       Running   0          2h
kube-flannel-ds-756mt                  1/1       Running   0          33m
kube-flannel-ds-7hvdg                  1/1       Running   0          33m
kube-flannel-ds-k6hsn                  1/1       Running   0          2h
kube-proxy-hzpkw                       1/1       Running   0          33m
kube-proxy-wsj9v                       1/1       Running   0          2h
kube-proxy-xjvvp                       1/1       Running   0          2h
kube-proxy-z5ngl                       1/1       Running   0          33m
kube-scheduler-k8s-master-1            1/1       Running   0          2h
kubernetes-dashboard-64d66bcc8-vcc5v   1/1       Running   0          22m

aaronkjones commented May 5, 2018

I have attempted to get Weave Net to work with k8s v1.10.2 and exausted all options it seems. I could only get it working with flannel as the CNI. I also had to add and change some commands.

I forked your gist and made the modifications including changing the script (prep.sh)

Try it out
https://github.com/aaronkjones/rpi-k8s-node-prep

pi@k8s-master-1:~ $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    2h        v1.10.2
k8s-slave-1    Ready     <none>    2h        v1.10.2
k8s-slave-2    Ready     <none>    33m       v1.10.2
k8s-slave-3    Ready     <none>    33m       v1.10.2
pi@k8s-master-1:~ $ kubectl get pods -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          2h
kube-apiserver-k8s-master-1            1/1       Running   0          2h
kube-controller-manager-k8s-master-1   1/1       Running   0          2h
kube-dns-686d6fb9c-glz8x               3/3       Running   0          2h
kube-flannel-ds-5grwv                  1/1       Running   0          2h
kube-flannel-ds-756mt                  1/1       Running   0          33m
kube-flannel-ds-7hvdg                  1/1       Running   0          33m
kube-flannel-ds-k6hsn                  1/1       Running   0          2h
kube-proxy-hzpkw                       1/1       Running   0          33m
kube-proxy-wsj9v                       1/1       Running   0          2h
kube-proxy-xjvvp                       1/1       Running   0          2h
kube-proxy-z5ngl                       1/1       Running   0          33m
kube-scheduler-k8s-master-1            1/1       Running   0          2h
kubernetes-dashboard-64d66bcc8-vcc5v   1/1       Running   0          22m
@vieskees

This comment has been minimized.

Show comment
Hide comment
@vieskees

vieskees May 7, 2018

@alexellis : thanks for the guide, it really helped me. I've been trying to get it working with Weave for a couple of days, but in the end I gave up and went with @aaronkjones 's idea. I used flannel as the CNI and got it working on the first try.

vieskees commented May 7, 2018

@alexellis : thanks for the guide, it really helped me. I've been trying to get it working with Weave for a couple of days, but in the end I gave up and went with @aaronkjones 's idea. I used flannel as the CNI and got it working on the first try.

@elafargue

This comment has been minimized.

Show comment
Hide comment
@elafargue

elafargue May 10, 2018

Same here: @aaronkjones 's guide is what worked for me as well. I took the liberty of creating a variant of this gist for those who want to use Hypriot. It also covers networking setup a bit more in-depth (local ethernet for the cluster, wifi connection via the master to reach the outside world): https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f . Still a work in progress.

Same here: @aaronkjones 's guide is what worked for me as well. I took the liberty of creating a variant of this gist for those who want to use Hypriot. It also covers networking setup a bit more in-depth (local ethernet for the cluster, wifi connection via the master to reach the outside world): https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f . Still a work in progress.

@mgoodfellow

This comment has been minimized.

Show comment
Hide comment
@mgoodfellow

mgoodfellow May 13, 2018

Just as a heads up - @aaronkjones solution was working for me perfectly last week, but I added new worker nodes to my existing cluster today and the new nodes don't initialise flannel or kube-proxy:

flannel:

Error: failed to start container "install-cni": Error response from daemon: linux mounts: Could not find source mount of /var/lib/kubelet/pods/532b1489-569a-11e8-aed4-b827eb359bc2/volumes/kubernetes.io~configmap/flannel-cfg
Back-off restarting failed container

kube-proxy:

Error: failed to start container "kube-proxy": Error response from daemon: linux mounts: Could not find source mount of /lib/modules
Back-off restarting failed container

Turns out, as of the last couple of days the get-docker install script now installs docker-ce 18.05 as the latest version, and this seems to cause this error.

Step in @alexellis instructions above (which installs latest version of docker):

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

I just downgraded the docker version on my nodes to docker-ce 18.04, rebooted, and everything seems OK now and my nodes initialise correctly.

$ sudo apt-get install docker-ce=18.04.0~ce~3-0~raspbian

You can check out the packages added to the repo lists by using:

apt-cache madison docker-ce

Hope this helps someone! Massive thanks to @alexellis and everyone else in this thread who have got me a working K8s cluster on rPi's - learnt loads!

mgoodfellow commented May 13, 2018

Just as a heads up - @aaronkjones solution was working for me perfectly last week, but I added new worker nodes to my existing cluster today and the new nodes don't initialise flannel or kube-proxy:

flannel:

Error: failed to start container "install-cni": Error response from daemon: linux mounts: Could not find source mount of /var/lib/kubelet/pods/532b1489-569a-11e8-aed4-b827eb359bc2/volumes/kubernetes.io~configmap/flannel-cfg
Back-off restarting failed container

kube-proxy:

Error: failed to start container "kube-proxy": Error response from daemon: linux mounts: Could not find source mount of /lib/modules
Back-off restarting failed container

Turns out, as of the last couple of days the get-docker install script now installs docker-ce 18.05 as the latest version, and this seems to cause this error.

Step in @alexellis instructions above (which installs latest version of docker):

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

I just downgraded the docker version on my nodes to docker-ce 18.04, rebooted, and everything seems OK now and my nodes initialise correctly.

$ sudo apt-get install docker-ce=18.04.0~ce~3-0~raspbian

You can check out the packages added to the repo lists by using:

apt-cache madison docker-ce

Hope this helps someone! Massive thanks to @alexellis and everyone else in this thread who have got me a working K8s cluster on rPi's - learnt loads!

@toolboc

This comment has been minimized.

Show comment
Hide comment
@toolboc

toolboc May 20, 2018

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

toolboc commented May 20, 2018

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

@aaronkjones

This comment has been minimized.

Show comment
Hide comment
@aaronkjones

aaronkjones May 21, 2018

https://github.com/aaronkjones/rpi-k8s-node-prep

I modified the script to allow for a specific version of Docker and Kubeadm to be installed and also pinned to prevent upgrade.

I have 4 RPis, so i made two two-node clusters and tried different combinations of Kubeadm/Docker.

Docker 18.04 and Kubeadm 1.10.2-00 work for me. It has been running on Hypriot for a few days.

https://github.com/aaronkjones/rpi-k8s-node-prep

I modified the script to allow for a specific version of Docker and Kubeadm to be installed and also pinned to prevent upgrade.

I have 4 RPis, so i made two two-node clusters and tried different combinations of Kubeadm/Docker.

Docker 18.04 and Kubeadm 1.10.2-00 work for me. It has been running on Hypriot for a few days.

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn May 23, 2018

Where or how do we report the issue to?

njohnsn commented May 23, 2018

Where or how do we report the issue to?

@thapli88

This comment has been minimized.

Show comment
Hide comment
@thapli88

thapli88 May 23, 2018

I couldn't get 1.10.3-00 working either.

For 1.10.2-00, in addition to downloading, installing, and holding the right packages, when you init the master, you need to set the version there too (otherwise, it'll default download the latest stable control images, which are 1.10.3)

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.10.2

I couldn't get 1.10.3-00 working either.

For 1.10.2-00, in addition to downloading, installing, and holding the right packages, when you init the master, you need to set the version there too (otherwise, it'll default download the latest stable control images, which are 1.10.3)

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.10.2

@geerlingguy

This comment has been minimized.

Show comment
Hide comment
@geerlingguy

geerlingguy May 23, 2018

@njohnsn, probably an issue on the Kubernetes repository at https://github.com/kubernetes/kubernetes/issues

I ran into the same issue, was getting errors like the one in this comment with the latest version of kubelet: geerlingguy/raspberry-pi-dramble#100 (comment)

I uninstalled docker-ce then reinstalled with sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian.

The init command I used (after installing with sudo apt-get install -y kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00) was:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=PI_IP_HERE --kubernetes-version v1.10.2

I had to downgrade both Kubernetes (from 1.10.3 to 1.10.2) and Docker CE (from 18.05.0 to 18.04.0) to Kubernetes to boot and run on Debian Stretch (Raspbian Lite)... but I finally got to:

# kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                  STATUS     ROLES     AGE       VERSION
kube1.pidramble.com   NotReady   master    5m        v1.10.2

@njohnsn, probably an issue on the Kubernetes repository at https://github.com/kubernetes/kubernetes/issues

I ran into the same issue, was getting errors like the one in this comment with the latest version of kubelet: geerlingguy/raspberry-pi-dramble#100 (comment)

I uninstalled docker-ce then reinstalled with sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian.

The init command I used (after installing with sudo apt-get install -y kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00) was:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=PI_IP_HERE --kubernetes-version v1.10.2

I had to downgrade both Kubernetes (from 1.10.3 to 1.10.2) and Docker CE (from 18.05.0 to 18.04.0) to Kubernetes to boot and run on Debian Stretch (Raspbian Lite)... but I finally got to:

# kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                  STATUS     ROLES     AGE       VERSION
kube1.pidramble.com   NotReady   master    5m        v1.10.2
@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn May 24, 2018

For some reason I can't get my fourth node to go into the ready state.

I've blown the SD card a way and reinstalled everything from scratch twice, but unlike the other 3 nodes, it won't come up.

Here is the output from syslog:

May 24 02:50:56 k8s-node-4 dockerd[479]: time="2018-05-24T02:50:56.309407492Z" level=error msg="Handler for GET /v1.31/images/weaveworks/weave-kube:2.3.0/json returned error: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311048     298 remote_image.go:83] ImageStatus "weaveworks/weave-kube:2.3.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311172     298 kuberuntime_image.go:87] ImageStatus for image {"weaveworks/weave-kube:2.3.0"} failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311326     298 kuberuntime_manager.go:733] container start failed: ImageInspectError: Failed to inspect image "weaveworks/weave-kube:2.3.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311433     298 pod_workers.go:186] Error syncing pod 95d24908-5efa-11e8-b36b-b827ebccbe66 ("weave-net-hxgx5_kube-system(95d24908-5efa-11e8-b36b-b827ebccbe66)"), skipping: failed to "StartContainer" for "weave" with ImageInspectError: "Failed to inspect image \"weaveworks/weave-kube:2.3.0\": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:58 k8s-node-4 kubelet[298]: W0524 02:50:58.686588     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:50:58 k8s-node-4 kubelet[298]: E0524 02:50:58.689475     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:03 k8s-node-4 kubelet[298]: W0524 02:51:03.694277     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:03 k8s-node-4 kubelet[298]: E0524 02:51:03.695457     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:08 k8s-node-4 kubelet[298]: W0524 02:51:08.700522     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:08 k8s-node-4 kubelet[298]: E0524 02:51:08.701369     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Thanks!

njohnsn commented May 24, 2018

For some reason I can't get my fourth node to go into the ready state.

I've blown the SD card a way and reinstalled everything from scratch twice, but unlike the other 3 nodes, it won't come up.

Here is the output from syslog:

May 24 02:50:56 k8s-node-4 dockerd[479]: time="2018-05-24T02:50:56.309407492Z" level=error msg="Handler for GET /v1.31/images/weaveworks/weave-kube:2.3.0/json returned error: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311048     298 remote_image.go:83] ImageStatus "weaveworks/weave-kube:2.3.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311172     298 kuberuntime_image.go:87] ImageStatus for image {"weaveworks/weave-kube:2.3.0"} failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311326     298 kuberuntime_manager.go:733] container start failed: ImageInspectError: Failed to inspect image "weaveworks/weave-kube:2.3.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311433     298 pod_workers.go:186] Error syncing pod 95d24908-5efa-11e8-b36b-b827ebccbe66 ("weave-net-hxgx5_kube-system(95d24908-5efa-11e8-b36b-b827ebccbe66)"), skipping: failed to "StartContainer" for "weave" with ImageInspectError: "Failed to inspect image \"weaveworks/weave-kube:2.3.0\": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:58 k8s-node-4 kubelet[298]: W0524 02:50:58.686588     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:50:58 k8s-node-4 kubelet[298]: E0524 02:50:58.689475     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:03 k8s-node-4 kubelet[298]: W0524 02:51:03.694277     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:03 k8s-node-4 kubelet[298]: E0524 02:51:03.695457     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:08 k8s-node-4 kubelet[298]: W0524 02:51:08.700522     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:08 k8s-node-4 kubelet[298]: E0524 02:51:08.701369     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Thanks!

@carlosroman

This comment has been minimized.

Show comment
Hide comment
@carlosroman

carlosroman May 27, 2018

@njohnsn I never got weave to work but have got flannel working. After looking into the issue I couldn't resolve the whole ImageInspectError so gave up. Out of 4 nodes I had two in the cluster and two wouldn't ever connect. Tried clean installs on them and still nothing. Strangely enough if I did a clean install of the cluster it would be different PIs that would fail :/

To get flannel to work I had to update /etc/kubernetes/manifests/kube-controller-manager.yaml and update the spec containers command with:
- --allocate-node-cidrs=true - --cluster-cidr=172.30.0.0/16
After reloading systemd (systemctl daemon-reload) and restarting k8s (systemctl restart kubelet.service) I was able to get flannel working.

I think you won't need that if you run kubeadm init with the flag --pod-network-cidr <network cidr> but not tried that since I got my cluster up and running. Will give it ago when I got more time (and update my playbook https://github.com/carlosroman/ansible-k8s-raspberry-playbook/) and see if flannel "just works".

@njohnsn I never got weave to work but have got flannel working. After looking into the issue I couldn't resolve the whole ImageInspectError so gave up. Out of 4 nodes I had two in the cluster and two wouldn't ever connect. Tried clean installs on them and still nothing. Strangely enough if I did a clean install of the cluster it would be different PIs that would fail :/

To get flannel to work I had to update /etc/kubernetes/manifests/kube-controller-manager.yaml and update the spec containers command with:
- --allocate-node-cidrs=true - --cluster-cidr=172.30.0.0/16
After reloading systemd (systemctl daemon-reload) and restarting k8s (systemctl restart kubelet.service) I was able to get flannel working.

I think you won't need that if you run kubeadm init with the flag --pod-network-cidr <network cidr> but not tried that since I got my cluster up and running. Will give it ago when I got more time (and update my playbook https://github.com/carlosroman/ansible-k8s-raspberry-playbook/) and see if flannel "just works".

@grmhay

This comment has been minimized.

Show comment
Hide comment
@grmhay

grmhay May 28, 2018

I can report (eventual) success with the following configuration on my 4 Raspberry pis:

  • Stretch Lite (2018-04-08)

  • v.18.04.0-ce Docker-ce [sudo apt-get install docker-ce=18.04.0ce3-0~raspbian]

  • Flannel (could not get my nodes to get to Ready state with Weave) [curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f - ]
    Note: no further editing of Kubernetes manifest files as suggested previously was required

  • v1.10.2 of Kubernetes (like many before, had to back rev from 1.10.3) [ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address= --kubernetes-version v1.10.2 --pod-network-cidr= ]

And thanks Alex for this original post and others who commented. Hoping this helps someone else struggling ....

grmhay commented May 28, 2018

I can report (eventual) success with the following configuration on my 4 Raspberry pis:

  • Stretch Lite (2018-04-08)

  • v.18.04.0-ce Docker-ce [sudo apt-get install docker-ce=18.04.0ce3-0~raspbian]

  • Flannel (could not get my nodes to get to Ready state with Weave) [curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f - ]
    Note: no further editing of Kubernetes manifest files as suggested previously was required

  • v1.10.2 of Kubernetes (like many before, had to back rev from 1.10.3) [ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address= --kubernetes-version v1.10.2 --pod-network-cidr= ]

And thanks Alex for this original post and others who commented. Hoping this helps someone else struggling ....

@liweiz

This comment has been minimized.

Show comment
Hide comment
@liweiz

liweiz May 29, 2018

I can not have dashboard work using proxy. I got the error message:

Error: 'dial tcp 172.17.0.2:9090: getsockopt: connection refused'
Trying to reach: 'https://172.17.0.2:9090/'

I followed the steps and learned some rbac along the way, but still can not figure out where to look at to solve this.

Any suggestions?

I also found this from kubernetes/dashboard readme:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

From what I have so far, my cluster did not install Heapster. Is it necessary to mention that in this guide?

liweiz commented May 29, 2018

I can not have dashboard work using proxy. I got the error message:

Error: 'dial tcp 172.17.0.2:9090: getsockopt: connection refused'
Trying to reach: 'https://172.17.0.2:9090/'

I followed the steps and learned some rbac along the way, but still can not figure out where to look at to solve this.

Any suggestions?

I also found this from kubernetes/dashboard readme:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

From what I have so far, my cluster did not install Heapster. Is it necessary to mention that in this guide?

@kumardeepam

This comment has been minimized.

Show comment
Hide comment
@kumardeepam

kumardeepam May 30, 2018

I am late to the fun :) has anyone followed this on the latest?
RASPBIAN STRETCH WITH DESKTOP
Image with desktop based on Debian Stretch
Version:April 2018
Release date:2018-04-18
Kernel version:4.14

I have not been able to progress beyond this
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.10.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-1] and IPs [192.168.0.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm:v1.10.3
- k8s.gcr.io/kube-controller-manager-arm:v1.10.3
- k8s.gcr.io/kube-scheduler-arm:v1.10.3
- k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Appreciate your time and help

I am late to the fun :) has anyone followed this on the latest?
RASPBIAN STRETCH WITH DESKTOP
Image with desktop based on Debian Stretch
Version:April 2018
Release date:2018-04-18
Kernel version:4.14

I have not been able to progress beyond this
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.10.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-1] and IPs [192.168.0.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm:v1.10.3
- k8s.gcr.io/kube-controller-manager-arm:v1.10.3
- k8s.gcr.io/kube-scheduler-arm:v1.10.3
- k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Appreciate your time and help

@kumardeepam

This comment has been minimized.

Show comment
Hide comment
@kumardeepam

kumardeepam May 30, 2018

@Creamen downgrading as you advised worked for me. Wasted 2 nights.
Anyone facing a similar issue here is how to downgrade (uninstall the latest version and install v1.9.700)

Step 1: uninstall kubernetes, follow the below given commands
$kubeadm reset
$sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$sudo apt-get autoremove
$sudo rm -rf ~/.kube

Step 2: reboot your pi
$sudo reboot

Step 3: Install v1.9.7-00, follow the below given commands
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

Step 4: Initiate your master node
$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

@Creamen downgrading as you advised worked for me. Wasted 2 nights.
Anyone facing a similar issue here is how to downgrade (uninstall the latest version and install v1.9.700)

Step 1: uninstall kubernetes, follow the below given commands
$kubeadm reset
$sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$sudo apt-get autoremove
$sudo rm -rf ~/.kube

Step 2: reboot your pi
$sudo reboot

Step 3: Install v1.9.7-00, follow the below given commands
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

Step 4: Initiate your master node
$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 2, 2018

Still no luck.
I switched to flannel but now the dns pod doesn't work:
commands I used after downgrading to Docker 18.0.4 and k8s 10.1.2.

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.1.240 --pod-network-cidr=172.30.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:/var/log/containers $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    41m       v1.10.2
k8s-node-1     Ready     <none>    40m       v1.10.2
k8s-node-2     Ready     <none>    40m       v1.10.2
k8s-node-3     Ready     <none>    40m       v1.10.2
k8s-node-4     Ready     <none>    40m       v1.10.2
k8s-node-5     Ready     <none>    40m       v1.10.2

pi@k8s-master-1:/var/log/containers $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS             RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running            0          41m
kube-apiserver-k8s-master-1            1/1       Running            0          42m
kube-controller-manager-k8s-master-1   1/1       Running            0          42m
kube-dns-686d6fb9c-ms85s               1/3       CrashLoopBackOff   27         42m
kube-flannel-ds-2nfrv                  1/1       Running            0          36m
kube-flannel-ds-8j8vh                  1/1       Running            0          36m
kube-flannel-ds-b2dfb                  1/1       Running            0          36m
kube-flannel-ds-qxrt6                  1/1       Running            0          36m
kube-flannel-ds-rr4dr                  1/1       Running            0          36m
kube-flannel-ds-w92sm                  1/1       Running            0          36m
kube-proxy-6f7g5                       1/1       Running            0          41m
kube-proxy-7w57r                       1/1       Running            0          41m
kube-proxy-8lvkq                       1/1       Running            0          41m
kube-proxy-f4x4s                       1/1       Running            0          41m
kube-proxy-psll8                       1/1       Running            0          42m
kube-proxy-ttrdn                       1/1       Running            0          41m
kube-scheduler-k8s-master-1            1/1       Running            0          42m
pi@k8s-master-1:/var/log/containers $

Where do I find the logs for the dns pod?

Thanks!

njohnsn commented Jun 2, 2018

Still no luck.
I switched to flannel but now the dns pod doesn't work:
commands I used after downgrading to Docker 18.0.4 and k8s 10.1.2.

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.1.240 --pod-network-cidr=172.30.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:/var/log/containers $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    41m       v1.10.2
k8s-node-1     Ready     <none>    40m       v1.10.2
k8s-node-2     Ready     <none>    40m       v1.10.2
k8s-node-3     Ready     <none>    40m       v1.10.2
k8s-node-4     Ready     <none>    40m       v1.10.2
k8s-node-5     Ready     <none>    40m       v1.10.2

pi@k8s-master-1:/var/log/containers $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS             RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running            0          41m
kube-apiserver-k8s-master-1            1/1       Running            0          42m
kube-controller-manager-k8s-master-1   1/1       Running            0          42m
kube-dns-686d6fb9c-ms85s               1/3       CrashLoopBackOff   27         42m
kube-flannel-ds-2nfrv                  1/1       Running            0          36m
kube-flannel-ds-8j8vh                  1/1       Running            0          36m
kube-flannel-ds-b2dfb                  1/1       Running            0          36m
kube-flannel-ds-qxrt6                  1/1       Running            0          36m
kube-flannel-ds-rr4dr                  1/1       Running            0          36m
kube-flannel-ds-w92sm                  1/1       Running            0          36m
kube-proxy-6f7g5                       1/1       Running            0          41m
kube-proxy-7w57r                       1/1       Running            0          41m
kube-proxy-8lvkq                       1/1       Running            0          41m
kube-proxy-f4x4s                       1/1       Running            0          41m
kube-proxy-psll8                       1/1       Running            0          42m
kube-proxy-ttrdn                       1/1       Running            0          41m
kube-scheduler-k8s-master-1            1/1       Running            0          42m
pi@k8s-master-1:/var/log/containers $

Where do I find the logs for the dns pod?

Thanks!

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 3, 2018

Found the logs for the DNS pod"

pi@k8s-master-1:~ $ kubectl logs -f --namespace=kube-system kube-dns-686d6fb9c-ms85s -c kubedns
I0603 14:37:58.522134       1 dns.go:48] version: 1.14.8
I0603 14:37:58.526437       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0603 14:37:58.526863       1 server.go:119] FLAG: --alsologtostderr="false"
I0603 14:37:58.527113       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0603 14:37:58.527202       1 server.go:119] FLAG: --config-map=""
I0603 14:37:58.527393       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0603 14:37:58.527473       1 server.go:119] FLAG: --config-period="10s"
I0603 14:37:58.527712       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0603 14:37:58.527788       1 server.go:119] FLAG: --dns-port="10053"
I0603 14:37:58.528038       1 server.go:119] FLAG: --domain="cluster.local."
I0603 14:37:58.528123       1 server.go:119] FLAG: --federations=""
I0603 14:37:58.528348       1 server.go:119] FLAG: --healthz-port="8081"
I0603 14:37:58.528425       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0603 14:37:58.528658       1 server.go:119] FLAG: --kube-master-url=""
I0603 14:37:58.528740       1 server.go:119] FLAG: --kubecfg-file=""
I0603 14:37:58.528935       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0603 14:37:58.529036       1 server.go:119] FLAG: --log-dir=""
I0603 14:37:58.529226       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0603 14:37:58.529307       1 server.go:119] FLAG: --logtostderr="true"
I0603 14:37:58.529497       1 server.go:119] FLAG: --nameservers=""
I0603 14:37:58.529524       1 server.go:119] FLAG: --stderrthreshold="2"
I0603 14:37:58.529547       1 server.go:119] FLAG: --v="2"
I0603 14:37:58.529570       1 server.go:119] FLAG: --version="false"
I0603 14:37:58.529602       1 server.go:119] FLAG: --vmodule=""
I0603 14:37:58.530045       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0603 14:37:58.531256       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0603 14:37:58.531308       1 dns.go:146] Starting endpointsController
I0603 14:37:58.531334       1 dns.go:149] Starting serviceController
I0603 14:37:58.533533       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0603 14:37:58.533610       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0603 14:37:59.032154       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:37:59.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.032087       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.532194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.532223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.032227       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.032217       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.532270       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.032212       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.532256       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.032242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.532230       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.032174       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.532176       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.032166       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.032207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.532207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.032224       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.532219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.032115       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.532079       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.032109       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.532204       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.032178       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.532142       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.032191       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.532209       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.032232       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.032219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.532118       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.032194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.532202       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.032177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.532185       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.032170       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.532257       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.032189       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.532366       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.032243       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.532134       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.032122       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.032221       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.532175       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.032214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.532241       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.032198       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.532179       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.032288       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.532242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.032240       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.032140       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.532060       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

It appears the DNS process is trying to check on the api status on port 443 when the api server is running on port 6443.
I don't know where to change it from though.

-Neil

njohnsn commented Jun 3, 2018

Found the logs for the DNS pod"

pi@k8s-master-1:~ $ kubectl logs -f --namespace=kube-system kube-dns-686d6fb9c-ms85s -c kubedns
I0603 14:37:58.522134       1 dns.go:48] version: 1.14.8
I0603 14:37:58.526437       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0603 14:37:58.526863       1 server.go:119] FLAG: --alsologtostderr="false"
I0603 14:37:58.527113       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0603 14:37:58.527202       1 server.go:119] FLAG: --config-map=""
I0603 14:37:58.527393       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0603 14:37:58.527473       1 server.go:119] FLAG: --config-period="10s"
I0603 14:37:58.527712       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0603 14:37:58.527788       1 server.go:119] FLAG: --dns-port="10053"
I0603 14:37:58.528038       1 server.go:119] FLAG: --domain="cluster.local."
I0603 14:37:58.528123       1 server.go:119] FLAG: --federations=""
I0603 14:37:58.528348       1 server.go:119] FLAG: --healthz-port="8081"
I0603 14:37:58.528425       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0603 14:37:58.528658       1 server.go:119] FLAG: --kube-master-url=""
I0603 14:37:58.528740       1 server.go:119] FLAG: --kubecfg-file=""
I0603 14:37:58.528935       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0603 14:37:58.529036       1 server.go:119] FLAG: --log-dir=""
I0603 14:37:58.529226       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0603 14:37:58.529307       1 server.go:119] FLAG: --logtostderr="true"
I0603 14:37:58.529497       1 server.go:119] FLAG: --nameservers=""
I0603 14:37:58.529524       1 server.go:119] FLAG: --stderrthreshold="2"
I0603 14:37:58.529547       1 server.go:119] FLAG: --v="2"
I0603 14:37:58.529570       1 server.go:119] FLAG: --version="false"
I0603 14:37:58.529602       1 server.go:119] FLAG: --vmodule=""
I0603 14:37:58.530045       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0603 14:37:58.531256       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0603 14:37:58.531308       1 dns.go:146] Starting endpointsController
I0603 14:37:58.531334       1 dns.go:149] Starting serviceController
I0603 14:37:58.533533       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0603 14:37:58.533610       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0603 14:37:59.032154       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:37:59.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.032087       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.532194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.532223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.032227       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.032217       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.532270       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.032212       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.532256       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.032242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.532230       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.032174       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.532176       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.032166       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.032207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.532207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.032224       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.532219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.032115       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.532079       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.032109       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.532204       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.032178       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.532142       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.032191       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.532209       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.032232       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.032219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.532118       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.032194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.532202       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.032177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.532185       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.032170       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.532257       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.032189       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.532366       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.032243       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.532134       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.032122       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.032221       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.532175       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.032214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.532241       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.032198       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.532179       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.032288       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.532242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.032240       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.032140       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.532060       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

It appears the DNS process is trying to check on the api status on port 443 when the api server is running on port 6443.
I don't know where to change it from though.

-Neil

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 4, 2018

Turns out the answer was here
I had to bump the version of kube-dns to 1.14.10.

njohnsn commented Jun 4, 2018

Turns out the answer was here
I had to bump the version of kube-dns to 1.14.10.

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Jun 4, 2018

@njohnsn Logs are shown by kubectl logs. A kube-dns pod is made up of 3 containers but the one you need to watch is kubedns(the others are dnsmasq& sidecar). --follow is if you want to live follow the logs. In the above case you could use:

kubectl logs --follow pods/kube-dns-686d6fb9c-ms85s -c kubedns --namespace=kube-system

@njohnsn Logs are shown by kubectl logs. A kube-dns pod is made up of 3 containers but the one you need to watch is kubedns(the others are dnsmasq& sidecar). --follow is if you want to live follow the logs. In the above case you could use:

kubectl logs --follow pods/kube-dns-686d6fb9c-ms85s -c kubedns --namespace=kube-system

@shreyasnaik

This comment has been minimized.

Show comment
Hide comment
@shreyasnaik

shreyasnaik Jun 5, 2018

@kumardeepam Thanks for you work and instructions on how to downgrade (uninstall the latest version and install v1.9.700) when getting an error "[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2"

My master is running now!

Also had to set cgroup_memory=memory not cgroup_memory=1.

@kumardeepam Thanks for you work and instructions on how to downgrade (uninstall the latest version and install v1.9.700) when getting an error "[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2"

My master is running now!

Also had to set cgroup_memory=memory not cgroup_memory=1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment