Instantly share code, notes, and snippets.

Embed
What would you like to do?
K8s on Raspbian

Kubernetes on (vanilla) Raspbian Lite

Yes - you can create a Kubernetes cluster with Raspberry Pis with the default operating system called Raspbian. This means you can carry on using all the tools and packages you're used to with the officially-supported OS.

This is part of a blog post Serverless Kubernetes home-lab with your Raspberry Pis written by Alex Ellis.

Copyright disclaimer: Please provide a link to the post and give attribution to the author if you plan to use this content in your own materials.

Pre-reqs:

  • You must use an RPi 2 or 3 for use with Kubernetes
  • I'm assuming you're using wired ethernet (Wi-Fi also works, but it's not recommended)

Master node setup

You can either follow the steps below, or use my flashing script which automates the below. The automated flashing script must be run on a Linux computer with an SD card writer or an RPi.

Flash with a Linux host

Provision a Raspberry Pi SD card

Then run:

curl -sLSf https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/125ad6eae27e40a235412c2b623285a089a08721/prep.sh | sudo sh

Continue to flash manually

  • Flash Raspbian to a fresh SD card.

You can use Etcher.io to burn the SD card.

Before booting set up an empty file called ssh in /boot/ on the SD card.

Use Raspbian Stretch Lite

Update: I previously recommended downloading Raspbian Jessie instead of Stretch. At time of writing (3 Jan 2018) Stretch is now fully compatible.

https://www.raspberrypi.org/downloads/raspbian/

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-master-1 or similar and then reboot.

  • Set a static IP address

It's not fun when your cluster breaks because the IP of your master changed. The master's certificates will be bound to the IP address, so let's fix that problem ahead of time:

cat >> /etc/dhcpcd.conf

Paste this block:

profile static_eth0
static ip_address=192.168.0.100/24
static routers=192.168.0.1
static domain_name_servers=8.8.8.8

Hit Control + D.

Change 100 for 101, 102, 103 etc.

You may also need to make a reservation on your router's DHCP table so these addresses don't get given out to other devices on your network.

  • Install Docker

This installs 17.12 or newer.

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
newgrp docker
  • Disable swap

For Kubernetes 1.7 and onwards you will get an error if swap space is enabled.

Turn off swap:

$ sudo dphys-swapfile swapoff && \
  sudo dphys-swapfile uninstall && \
  sudo update-rc.d dphys-swapfile remove

This should now show no entries:

$ sudo swapon --summary
  • Edit /boot/cmdline.txt

Add this text at the end of the line, but don't create any new lines:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Now reboot - do not skip this step.

  • Add repo lists & install kubeadm
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubeadm

I realise this says 'xenial' in the apt listing, don't worry. It still works.

Initialize your master node

  • You now have two new commands installed:
  • kubeadm - used to create new clusters or join an existing one
  • kubectl - the CLI administration tool for Kubernetes

If using Weave Net

  • Initialize your master node:
$ sudo kubeadm init --token-ttl=0

If using Flannel:

  • Initialize your master node with a Pod network CIDR:
$ sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

Optionally also pass --apiserver-advertise-address=192.168.0.27 with the IP of the Pi as found by typing ifconfig.

Note: This step can take a long time, even up to 15 minutes.

Sometimes this stage can fail, if it does then you should patch the API Server to allow for a higher failure threshold during initialization around the time you see [controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"

sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml

After the init is complete run the snippet given to you on the command-line:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

This step takes the key generated for cluster administration and makes it available in a default location for use with kubectl.

  • Now save your join-token

Your join token is valid for 24 hours, so save it into a text file. Here's an example of mine:

$ kubeadm join --token 9e700f.7dc97f5e3a45c9e5 192.168.0.27:6443 --discovery-token-ca-cert-hash sha256:95cbb9ee5536aa61ec0239d6edd8598af68758308d0a0425848ae1af28859bea
  • Check everything worked:
$ kubectl get pods --namespace=kube-system
NAME                           READY     STATUS    RESTARTS   AGE                
etcd-of-2                      1/1       Running   0          12m                
kube-apiserver-of-2            1/1       Running   2          12m                
kube-controller-manager-of-2   1/1       Running   1          11m                
kube-dns-66ffd5c588-d8292      3/3       Running   0          11m                
kube-proxy-xcj5h               1/1       Running   0          11m                
kube-scheduler-of-2            1/1       Running   0          11m                
weave-net-zz9rz                2/2       Running   0          5m 

You should see the "READY" count showing as 1/1 for all services as above. DNS uses three pods, so you'll see 3/3 for that.

Setup networking with Weave Net or Flannel

Some users have reported stability issues with Weave Net on ARMHF. These issues do not appear to affect x86_64 (regular PCs/VMs). You may want to try Flannel instead of Weave Net for your RPi cluster.

Weave Net

Install Weave Net network driver

$ kubectl apply -f \
 "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

If you run into any issues with Weaveworks' networking then flannel is also a popular choice for the ARM platform.

Flannel (alternative)

Apply the Flannel driver on the master:

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml

On each node that joins including the master:

$ sudo sysctl net.bridge.bridge-nf-call-iptables=1

Join other nodes

On the other RPis, repeat everything apart from kubeadm init.

  • Change hostname

Use the raspi-config utility to change the hostname to k8s-worker-1 or similar and then reboot.

  • Join the cluster

Replace the token / IP for the output you got from the master node, for example:

$ sudo kubeadm join --token 1fd0d8.67e7083ed7ec08f3 192.168.0.27:6443

You can now run this on the master:

$ kubectl get nodes
NAME      STATUS     AGE       VERSION
k8s-1     Ready      5m        v1.7.4
k8s-2     Ready      10m       v1.7.4

Deploy a container

This container will expose a HTTP port and convert Markdown to HTML. Just post a body to it via curl - follow the instructions below.

function.yml

apiVersion: v1
kind: Service
metadata:
  name: markdownrender
  labels:
    app: markdownrender
spec:
  type: NodePort
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 31118
  selector:
    app: markdownrender
---
apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: markdownrender
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: markdownrender
    spec:
      containers:
      - name: markdownrender
        image: functions/markdownrender:latest-armhf
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP

Deploy and test:

$ kubectl create -f function.yml

Once the Docker image has been pulled from the hub and the Pod is running you can access it via curl:

$ curl -4 http://127.0.0.1:31118 -d "# test"
<p><h1>test</h1></p>

If you want to call the service from a remote machine such as your laptop then use the IP address of your Kubernetes master node and try the same again.

Start up the Kubernetes dashboard

The dashboard can be useful for visualising the state and health of your system, but it does require the equivalent of "root" in the cluster. If you want to proceed you should first run in a ClusterRole from the docs.

echo -n 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system' | kubectl apply -f -

This is the development/alternative dashboard which has TLS disabled and is easier to use.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard-arm.yaml

You can then find the IP and port via kubectl get svc -n kube-system. To access this from your laptop you will need to use kubectl proxy and navigate to http://localhost:8001/ on the master, or tunnel to this address with ssh.

See also: Kubernetes Dashboard docs.

Remove the test deployment

Now on the Kubernetes master remove the test deployment:

$ kubectl delete -f function.yml

Wrapping up

You should now have an operational Kubernetes master and several worker nodes ready to accept workloads.

Now let's head back over to the tutorial and deploy OpenFaaS to put the cluster through its paces with Serverless functions.

See also: Kubernetes documentation

#!/bin/sh
# This installs the base instructions up to the point of joining / creating a cluster
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update -q && \
sudo apt-get install -qy kubeadm
echo Adding " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory" to /boot/cmdline.txt
sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"
echo $orig | sudo tee /boot/cmdline.txt
echo Please reboot

Use this to setup quickly

# curl -sL \
 https://gist.githubusercontent.com/alexellis/fdbc90de7691a1b9edb545c17da2d975/raw/b04f1e9250c61a8ff554bfe3475b6dd050062484/prep.sh \
 | sudo sh
@Lewiscowles1986

This comment has been minimized.

Show comment
Hide comment
@Lewiscowles1986

Lewiscowles1986 Oct 12, 2017

This is great. It'd be very cool to have this operate unattended. (or as unattended as possible)

Lewiscowles1986 commented Oct 12, 2017

This is great. It'd be very cool to have this operate unattended. (or as unattended as possible)

@shanselman

This comment has been minimized.

Show comment
Hide comment

shanselman commented Oct 25, 2017

@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman Oct 25, 2017

The swapfile turns back on when you reboot unless you

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

shanselman commented Oct 25, 2017

The swapfile turns back on when you reboot unless you

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove
@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman Oct 25, 2017

For this line curl localhost:31118 -d "# test" I had to use the full host name. Localhost is still 127.0.0.1 and it doesn't seem to be listening

shanselman commented Oct 25, 2017

For this line curl localhost:31118 -d "# test" I had to use the full host name. Localhost is still 127.0.0.1 and it doesn't seem to be listening

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Oct 25, 2017

Kubernetes please stop changing every other day 👎

Owner

alexellis commented Oct 25, 2017

Kubernetes please stop changing every other day 👎

@olavt

This comment has been minimized.

Show comment
Hide comment
@olavt

olavt Oct 29, 2017

I followed the instructions and got everything installed on a 2x Raspberry PI 3 cluster (1 master and 1 node). But, I have not been able to get the Dashboard up and running.

olavt@k8s-master-1:~ $ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 5h
kubernetes-dashboard ClusterIP 10.104.85.132 443/TCP 4h
olavt@k8s-master-1:~ $ kubectl proxy
Starting to serve on 127.0.0.1:8001

What is the Url I should use from my other computer to connect to the Dashboard?

olavt commented Oct 29, 2017

I followed the instructions and got everything installed on a 2x Raspberry PI 3 cluster (1 master and 1 node). But, I have not been able to get the Dashboard up and running.

olavt@k8s-master-1:~ $ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 5h
kubernetes-dashboard ClusterIP 10.104.85.132 443/TCP 4h
olavt@k8s-master-1:~ $ kubectl proxy
Starting to serve on 127.0.0.1:8001

What is the Url I should use from my other computer to connect to the Dashboard?

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Oct 30, 2017

OK for the dashboard you need to run kubectl on your own PC/laptop. Maybe an SSH tunnel would work?

ssh -L 8001:127.0.01:8001 pi@k8s-master-1.local

then try 127.0.0.1:8001 on your local machine

Owner

alexellis commented Oct 30, 2017

OK for the dashboard you need to run kubectl on your own PC/laptop. Maybe an SSH tunnel would work?

ssh -L 8001:127.0.01:8001 pi@k8s-master-1.local

then try 127.0.0.1:8001 on your local machine

@olavt

This comment has been minimized.

Show comment
Hide comment
@olavt

olavt Oct 30, 2017

That didn't work for me.

olavt commented Oct 30, 2017

That didn't work for me.

@steini

This comment has been minimized.

Show comment
Hide comment
@steini

steini Nov 3, 2017

First of all thanks for the detailed setup process.

After updating raspbian i ran into the problem that sudo kubeadm join raised the error CGROUPS_MEMORY: missing. The boot option is no longer cgroup_enable=memory but cgroup_memory=1

See https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086#p57035 and raspberrypi/linux@ba742b5

steini commented Nov 3, 2017

First of all thanks for the detailed setup process.

After updating raspbian i ran into the problem that sudo kubeadm join raised the error CGROUPS_MEMORY: missing. The boot option is no longer cgroup_enable=memory but cgroup_memory=1

See https://archlinuxarm.org/forum/viewtopic.php?f=15&t=12086#p57035 and raspberrypi/linux@ba742b5

@movingbytes

This comment has been minimized.

Show comment
Hide comment
@movingbytes

movingbytes Nov 5, 2017

after installation the status of all pods in namespace kube-system is pending except kube-proxy (NodeLost). Any ideas?
Using docker 17.10 and K8S 1.8.2

movingbytes commented Nov 5, 2017

after installation the status of all pods in namespace kube-system is pending except kube-proxy (NodeLost). Any ideas?
Using docker 17.10 and K8S 1.8.2

@borrillis

This comment has been minimized.

Show comment
Hide comment
@borrillis

borrillis Nov 15, 2017

My dashboard wouldn't work properly until I did:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccou nt=kube-system:kubernetes-dashboard

I could get to the dashboard using kubectl proxy and opened the url http://localhost:8001/ui in a browser, but it couldn't get any data from the api.

borrillis commented Nov 15, 2017

My dashboard wouldn't work properly until I did:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccou nt=kube-system:kubernetes-dashboard

I could get to the dashboard using kubectl proxy and opened the url http://localhost:8001/ui in a browser, but it couldn't get any data from the api.

@francis2211

This comment has been minimized.

Show comment
Hide comment
@francis2211

francis2211 Nov 22, 2017

@alexellis it should be cgroup_memory=1 not cgroup_enable=memory

francis2211 commented Nov 22, 2017

@alexellis it should be cgroup_memory=1 not cgroup_enable=memory

@krystan

This comment has been minimized.

Show comment
Hide comment
@krystan

krystan Dec 12, 2017

cgroup_enable=memory seems to be fine under kernel 4.9.35-v7.

krystan commented Dec 12, 2017

cgroup_enable=memory seems to be fine under kernel 4.9.35-v7.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Dec 23, 2017

I've updated the instructions for the newer RPi kernel.

Owner

alexellis commented Dec 23, 2017

I've updated the instructions for the newer RPi kernel.

@charliesolomon

This comment has been minimized.

Show comment
Hide comment
@charliesolomon

charliesolomon Jan 1, 2018

I had to run the "set up networking" step (install weave) in order to get "Running" back from the 3 DNS pods. Before that, they reported "Pending"... move the "set up networking" step before "check everything worked" in your instructions?

charliesolomon commented Jan 1, 2018

I had to run the "set up networking" step (install weave) in order to get "Running" back from the 3 DNS pods. Before that, they reported "Pending"... move the "set up networking" step before "check everything worked" in your instructions?

@teekay

This comment has been minimized.

Show comment
Hide comment
@teekay

teekay Jan 4, 2018

I was also only able to get both Master and 1 "slave" node to the Ready status when I first installed the "weave" networking on the master, and only after that joined the worker. K8s version 1.9.

teekay commented Jan 4, 2018

I was also only able to get both Master and 1 "slave" node to the Ready status when I first installed the "weave" networking on the master, and only after that joined the worker. K8s version 1.9.

@evnsio

This comment has been minimized.

Show comment
Hide comment
@evnsio

evnsio Jan 8, 2018

Has anyone experienced an issue kubeadm? I'm getting Illegal instruction when I try to run it.

Running on Raspian Stretch 4.9.59+.

evnsio commented Jan 8, 2018

Has anyone experienced an issue kubeadm? I'm getting Illegal instruction when I try to run it.

Running on Raspian Stretch 4.9.59+.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Jan 8, 2018

@caedev - no, you are definitely using a Raspberry Pi 2 or 3?

Owner

alexellis commented Jan 8, 2018

@caedev - no, you are definitely using a Raspberry Pi 2 or 3?

@evnsio

This comment has been minimized.

Show comment
Hide comment
@evnsio

evnsio Jan 8, 2018

Sorry, just realised I was ssh'ing into the wrong pi; this works absolutely fine on my Pi 2. Thanks for writing this @alexellis - much appreciated.

evnsio commented Jan 8, 2018

Sorry, just realised I was ssh'ing into the wrong pi; this works absolutely fine on my Pi 2. Thanks for writing this @alexellis - much appreciated.

@haebler

This comment has been minimized.

Show comment
Hide comment
@haebler

haebler Jan 9, 2018

same experience as @charliesolomon, DNS doesn't come up until you install the weave network driver.

Basically change to below:

  • Install network driver kubectl apply -f https://git.io/weave-kube-1.6
  • Check status: kubectl get pods --namespace=kube-system

Note: Be patient on the 2nd step, the weave driver comes up first. Once it is Running DNS goes from Pending to ContainerCreating to Running.

haebler commented Jan 9, 2018

same experience as @charliesolomon, DNS doesn't come up until you install the weave network driver.

Basically change to below:

  • Install network driver kubectl apply -f https://git.io/weave-kube-1.6
  • Check status: kubectl get pods --namespace=kube-system

Note: Be patient on the 2nd step, the weave driver comes up first. Once it is Running DNS goes from Pending to ContainerCreating to Running.

@chris-short

This comment has been minimized.

Show comment
Hide comment
@chris-short

chris-short Jan 13, 2018

In the dashboard section, you might want to mention the need for rbac: https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

chris-short commented Jan 13, 2018

In the dashboard section, you might want to mention the need for rbac: https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

@DazWilkin

This comment has been minimized.

Show comment
Hide comment
@DazWilkin

DazWilkin Jan 20, 2018

An excellent guide, thank you!

The instructions are unclear for accessing the cluster remotely but are explained here:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master

Effectively make a copy on the local machine of the master's /etc/kubernetes/admin.conf perhaps named k8s_pi.conf

Then kubectl --kubeconfig ./k8s_pi.conf get nodes

Or, per your example to create a proxy: kubectl --kubeconfig ./k8s_pi.conf proxy &

To avoid specifying --kubeconfig repeatedly, you can merge the contents of k8s_pi.conf into the default config ~/.kube/config

DazWilkin commented Jan 20, 2018

An excellent guide, thank you!

The instructions are unclear for accessing the cluster remotely but are explained here:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master

Effectively make a copy on the local machine of the master's /etc/kubernetes/admin.conf perhaps named k8s_pi.conf

Then kubectl --kubeconfig ./k8s_pi.conf get nodes

Or, per your example to create a proxy: kubectl --kubeconfig ./k8s_pi.conf proxy &

To avoid specifying --kubeconfig repeatedly, you can merge the contents of k8s_pi.conf into the default config ~/.kube/config

@DazWilkin

This comment has been minimized.

Show comment
Hide comment
@DazWilkin

DazWilkin Jan 20, 2018

Follow-up (kubeadm) question: What's the process to shutdown and restart the cluster?

kubeadm reset seems more of a teardown.

What if you'd just like to shut the cluster down correctly to then shutdown the underlying Pis and restart subsequently?

DazWilkin commented Jan 20, 2018

Follow-up (kubeadm) question: What's the process to shutdown and restart the cluster?

kubeadm reset seems more of a teardown.

What if you'd just like to shut the cluster down correctly to then shutdown the underlying Pis and restart subsequently?

@denhamparry

This comment has been minimized.

Show comment
Hide comment
@denhamparry

denhamparry Jan 29, 2018

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

  • Edit kubernetes-dashboard service.
$ kubectl -n kube-system edit service kubernetes-dashboard
  • You should the see yaml representation of the service. Change type: ClusterIP to type: NodePort and save file.
  • Check port on which Dashboard was exposed.
$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m
  • Create a proxy to view within your browser
$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

Thanks again Alex!

denhamparry commented Jan 29, 2018

Have been playing around with this over the weekend, really enjoying the project!

I hit a block with Kubernetes Dashboard, and realised that I couldn't connect to it via proxy due to it being set as a ClusterIP rather than a NodeIP.

  • Edit kubernetes-dashboard service.
$ kubectl -n kube-system edit service kubernetes-dashboard
  • You should the see yaml representation of the service. Change type: ClusterIP to type: NodePort and save file.
  • Check port on which Dashboard was exposed.
$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.108.252.18   <none>        80:30294/TCP   23m
  • Create a proxy to view within your browser
$ ssh -L 8001:127.0.0.1:31707 pi@k8s-master-1.local

Thanks again Alex!

@yosephbuitrago

This comment has been minimized.

Show comment
Hide comment
@yosephbuitrago

yosephbuitrago Mar 2, 2018

Hi, Alex, thank for share this tutorial. I builded a raspberry pi cluster and is running kubernetes and OpenFaas as expected it. the only thing is that the auto-scaling don't in OpenfaaS does work! on my computer works but it does work in the cluster!

Do I have to change something in the .yml files? I check them but they look the same.

yosephbuitrago commented Mar 2, 2018

Hi, Alex, thank for share this tutorial. I builded a raspberry pi cluster and is running kubernetes and OpenFaas as expected it. the only thing is that the auto-scaling don't in OpenfaaS does work! on my computer works but it does work in the cluster!

Do I have to change something in the .yml files? I check them but they look the same.

@johndcollins

This comment has been minimized.

Show comment
Hide comment
@johndcollins

johndcollins Mar 13, 2018

I had to add both cgroup_memory=memory AND cgroup_memory=1 to the cmdline.txt to get it to work.

johndcollins commented Mar 13, 2018

I had to add both cgroup_memory=memory AND cgroup_memory=1 to the cmdline.txt to get it to work.

@bilalAchahbar

This comment has been minimized.

Show comment
Hide comment
@bilalAchahbar

bilalAchahbar Mar 21, 2018

Great and very understandable post !!
I've set the kubernetes dashboard through the Nodeport and can access it on my host but the certificates still give a lot of issues.
Is it possible to use Let's encrypt for the kubernetes dashboard ?
As i am new to the concept of certificates through websites can anyone point me how i can do this through an authomatic service like let's encrypt.

bilalAchahbar commented Mar 21, 2018

Great and very understandable post !!
I've set the kubernetes dashboard through the Nodeport and can access it on my host but the certificates still give a lot of issues.
Is it possible to use Let's encrypt for the kubernetes dashboard ?
As i am new to the concept of certificates through websites can anyone point me how i can do this through an authomatic service like let's encrypt.

@Jickelsen

This comment has been minimized.

Show comment
Hide comment
@Jickelsen

Jickelsen Apr 1, 2018

Thanks for the fantastic guide, I had great fun learning about all these topics in practice over a weekend. As a switch I'm having great success with the 5-port D-Link DGS-1005D, newer versions of which use mini-USB for power.

I had issues getting Weave to work on Raspbian Stretch and the Pi3 B+. Shortly after running
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
the master and connected nodes would reboot unexpectedly, and would leave the cluster in an error state.
I ended up using flannel:

  • Use --pod-network-cidr=10.244.0.0/16 when initializing the cluster
    $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=<internal master ip> --pod-network-cidr=10.244.0.0/16
  • Install flannel with
    $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

I also managed to set up the master as a router, with Wifi on the WAN side, using the steps in this particular post https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=132674&start=50#p1252309

Jickelsen commented Apr 1, 2018

Thanks for the fantastic guide, I had great fun learning about all these topics in practice over a weekend. As a switch I'm having great success with the 5-port D-Link DGS-1005D, newer versions of which use mini-USB for power.

I had issues getting Weave to work on Raspbian Stretch and the Pi3 B+. Shortly after running
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
the master and connected nodes would reboot unexpectedly, and would leave the cluster in an error state.
I ended up using flannel:

  • Use --pod-network-cidr=10.244.0.0/16 when initializing the cluster
    $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=<internal master ip> --pod-network-cidr=10.244.0.0/16
  • Install flannel with
    $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

I also managed to set up the master as a router, with Wifi on the WAN side, using the steps in this particular post https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=132674&start=50#p1252309

@DerfOh

This comment has been minimized.

Show comment
Hide comment
@DerfOh

DerfOh Apr 7, 2018

Thanks @Jickelsen I had to do the same.
In addition to that I also my nodes stuck in a not ready state due to the following error:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Fixed this by removing KUBELET_NETWORK_ARGS from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf then rebooting according to this issue: kubernetes/kubernetes#38653

I was then able to run
curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
without issue.

DerfOh commented Apr 7, 2018

Thanks @Jickelsen I had to do the same.
In addition to that I also my nodes stuck in a not ready state due to the following error:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Fixed this by removing KUBELET_NETWORK_ARGS from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf then rebooting according to this issue: kubernetes/kubernetes#38653

I was then able to run
curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
without issue.

@exp0nge

This comment has been minimized.

Show comment
Hide comment
@exp0nge

exp0nge Apr 7, 2018

I can't seem to get init past [init] This might take a minute or longer if the control plane images have to be pulled.. There's so many issues on kubernetes/kubadm about this. I used a fresh install of rasbian lite (march 3rd update). Anyone else get this or know a workaround?

exp0nge commented Apr 7, 2018

I can't seem to get init past [init] This might take a minute or longer if the control plane images have to be pulled.. There's so many issues on kubernetes/kubadm about this. I used a fresh install of rasbian lite (march 3rd update). Anyone else get this or know a workaround?

@rashray

This comment has been minimized.

Show comment
Hide comment
@rashray

rashray Apr 8, 2018

Thank You Alex. Very detailed steps. I am using a b plus Pi as a master. Any idea why the Pi goes dead slow on initiating the Kube master.

rashray commented Apr 8, 2018

Thank You Alex. Very detailed steps. I am using a b plus Pi as a master. Any idea why the Pi goes dead slow on initiating the Kube master.

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Apr 9, 2018

Thanks @Jickelsen & @DerfOh! I spent all my spare time in the last three weeks trying to get kubernetes to work again. The gist worked great at Xmas but now once you get weavenet up on the node & synced to the master, both crash with an oops:
kernel:[ 4286.584219] Internal error: Oops: 80000007 [#1] SMP ARM
kernel:[ 4287.037510] Process weaver (pid: 13327, stack limit = 0x9bb12210)
kernel:[ 4287.059886] Stack: (0x9bb139f0 to 0x9bb14000)
kernel:[ 4287.081698] 39e0: 00000000 00000000 5001a8c0 9bb13a88
kernel:[ 4287.125181] 3a00: 0000801a 0000db84 9bab4150 9bab4118 9bb13d2c 7f63bad0 00000001 9bb13a5c
Finally I can finish writing my ansible play-book to automate the whole thing.

micedwards commented Apr 9, 2018

Thanks @Jickelsen & @DerfOh! I spent all my spare time in the last three weeks trying to get kubernetes to work again. The gist worked great at Xmas but now once you get weavenet up on the node & synced to the master, both crash with an oops:
kernel:[ 4286.584219] Internal error: Oops: 80000007 [#1] SMP ARM
kernel:[ 4287.037510] Process weaver (pid: 13327, stack limit = 0x9bb12210)
kernel:[ 4287.059886] Stack: (0x9bb139f0 to 0x9bb14000)
kernel:[ 4287.081698] 39e0: 00000000 00000000 5001a8c0 9bb13a88
kernel:[ 4287.125181] 3a00: 0000801a 0000db84 9bab4150 9bab4118 9bb13d2c 7f63bad0 00000001 9bb13a5c
Finally I can finish writing my ansible play-book to automate the whole thing.

@carlosroman

This comment has been minimized.

Show comment
Hide comment
@carlosroman

carlosroman Apr 10, 2018

I've had strange issues with getting weavenet running

NAMESPACE     NAME                                      READY     STATUS              RESTARTS   AGE
kube-system   weave-net-8t7zd                           2/2       Running             494        1d
kube-system   weave-net-gpcnj                           1/2       CrashLoopBackOff    417        1d
kube-system   weave-net-m7tnn                           1/2       ImageInspectError   0          1d
kube-system   weave-net-qmjwk                           1/2       ImageInspectError   0          1d
kube-system   weave-net-rvwpj                           2/2       Running             534        1d

Still debuging it but it has been a fun learning experience getting K8s running on a Raspberry Pi cluster.

@micedwards, I ended up writting an ansible playbook as kept rebuilding my cluster to see why weave kept crashing. Wrote it after running kubeadm reset on the master accidently or on a node. Now have a playbook that sets up my cluster and adds nodes to it as well. Any improvements would be great, https://github.com/carlosroman/ansible-k8s-raspberry-playbook.

carlosroman commented Apr 10, 2018

I've had strange issues with getting weavenet running

NAMESPACE     NAME                                      READY     STATUS              RESTARTS   AGE
kube-system   weave-net-8t7zd                           2/2       Running             494        1d
kube-system   weave-net-gpcnj                           1/2       CrashLoopBackOff    417        1d
kube-system   weave-net-m7tnn                           1/2       ImageInspectError   0          1d
kube-system   weave-net-qmjwk                           1/2       ImageInspectError   0          1d
kube-system   weave-net-rvwpj                           2/2       Running             534        1d

Still debuging it but it has been a fun learning experience getting K8s running on a Raspberry Pi cluster.

@micedwards, I ended up writting an ansible playbook as kept rebuilding my cluster to see why weave kept crashing. Wrote it after running kubeadm reset on the master accidently or on a node. Now have a playbook that sets up my cluster and adds nodes to it as well. Any improvements would be great, https://github.com/carlosroman/ansible-k8s-raspberry-playbook.

@ScubaJimmer

This comment has been minimized.

Show comment
Hide comment
@ScubaJimmer

ScubaJimmer Apr 12, 2018

Good Evening.

I have been having trouble getting kubernetes+docker running as a 2 RPI cluster. My master node continues to reboot. I followed all the steps above to configure two fresh nodes, except I used my router to establish a static IP for my master and worker node. Interestingly my worker node seems stable so far right now. In previous attempts, when I had set up 4 additional nodes they too became unstable.
The master node was stable before I joined my first worker node

Docker version: 18.03.0-ce, build 0520e24
Kubernetes version : 1.10

Master node:

pi@k8boss1:~ $ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE
etcd-k8boss1 1/1 Running 33 1d
kube-apiserver-k8boss1 1/1 Running 34 1d
kube-controller-manager-k8boss1 1/1 Running 34 1d
kube-dns-686d6fb9c-hwxxw 0/3 Error 0 1d
kube-proxy-8v8z7 0/1 Error 33 1d
kube-proxy-dgqxp 1/1 Running 0 1h
kube-scheduler-k8boss1 1/1 Running 34 1d
weave-net-ggxwp 2/2 Running 0 1h
weave-net-l7xsl 0/2 Error 71 1d

pi@k8boss1:~ $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8boss1 Ready master 1d v1.10.0
k8worker1 Ready 1h v1.10.0

pi@k8boss1:~ $ uptime
01:48:50 up 0 min, 1 user, load average: 1.37, 0.41, 0.14
pi@k8boss1:~ $

Worker:
pi@k8worker1:~ $ uptime
01:49:35 up 1:58, 1 user, load average: 0.11, 0.21, 0.19
pi@k8worker1:~ $

Any thoughts?

ScubaJimmer commented Apr 12, 2018

Good Evening.

I have been having trouble getting kubernetes+docker running as a 2 RPI cluster. My master node continues to reboot. I followed all the steps above to configure two fresh nodes, except I used my router to establish a static IP for my master and worker node. Interestingly my worker node seems stable so far right now. In previous attempts, when I had set up 4 additional nodes they too became unstable.
The master node was stable before I joined my first worker node

Docker version: 18.03.0-ce, build 0520e24
Kubernetes version : 1.10

Master node:

pi@k8boss1:~ $ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE
etcd-k8boss1 1/1 Running 33 1d
kube-apiserver-k8boss1 1/1 Running 34 1d
kube-controller-manager-k8boss1 1/1 Running 34 1d
kube-dns-686d6fb9c-hwxxw 0/3 Error 0 1d
kube-proxy-8v8z7 0/1 Error 33 1d
kube-proxy-dgqxp 1/1 Running 0 1h
kube-scheduler-k8boss1 1/1 Running 34 1d
weave-net-ggxwp 2/2 Running 0 1h
weave-net-l7xsl 0/2 Error 71 1d

pi@k8boss1:~ $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8boss1 Ready master 1d v1.10.0
k8worker1 Ready 1h v1.10.0

pi@k8boss1:~ $ uptime
01:48:50 up 0 min, 1 user, load average: 1.37, 0.41, 0.14
pi@k8boss1:~ $

Worker:
pi@k8worker1:~ $ uptime
01:49:35 up 1:58, 1 user, load average: 0.11, 0.21, 0.19
pi@k8worker1:~ $

Any thoughts?

@peterkingsbury

This comment has been minimized.

Show comment
Hide comment
@peterkingsbury

peterkingsbury Apr 19, 2018

On Raspbian Stretch Lite, the installation halts during the master setup phase (sudo kubeadm init --token-ttl=0) with the following output:

[init] This might take a minute or longer if the control plane images have to be pulled.

I found it necessary to install Kubernetes 1.9.6:

sudo apt-get install -y kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

Took 552.013509 seconds to complete, but it's up and running now!

Thanks for a great tutorial!

peterkingsbury commented Apr 19, 2018

On Raspbian Stretch Lite, the installation halts during the master setup phase (sudo kubeadm init --token-ttl=0) with the following output:

[init] This might take a minute or longer if the control plane images have to be pulled.

I found it necessary to install Kubernetes 1.9.6:

sudo apt-get install -y kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

Took 552.013509 seconds to complete, but it's up and running now!

Thanks for a great tutorial!

@danielvaughan

This comment has been minimized.

Show comment
Hide comment
@danielvaughan

danielvaughan Apr 21, 2018

I am running into the same problems as @carlosroman and @micedwards after applying weave on a 4 RPi 3 cluster:

Raspbian GNU/Linux 9 (stretch)
Docker version 18.04.0-ce, build 3d479c0
Kubernetes v1.10.1

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS              RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running             22         10h
kube-apiserver-k8s-master-1            1/1       Running             39         10h
kube-controller-manager-k8s-master-1   1/1       Running             13         10h
kube-dns-686d6fb9c-qn2mp               0/3       Pending             0          10h
kube-proxy-6dlz4                       1/1       Running             11         9h
kube-proxy-7s977                       1/1       Running             2          9h
kube-proxy-q7jlh                       1/1       Running             11         10h
kube-proxy-qdmp7                       1/1       Running             2          9h
kube-scheduler-k8s-master-1            1/1       Running             13         10h
weave-net-5scxb                        2/2       Running             1          2m
weave-net-5vxzw                        1/2       CrashLoopBackOff    4          2m
weave-net-jmlzc                        1/2       ImageInspectError   0          2m
weave-net-xc2f8                        1/2       ImageInspectError   1          2m
pi@k8s-master-1:~ $
Message from syslogd@k8s-master-1 at Apr 22 08:04:14 ...
 kernel:[  155.252476] Internal error: Oops: 80000007 [#1] SMP ARM

I am having more luck with flannel

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.1.1.200 --pod-network-cidr=10.244.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          5m
kube-apiserver-k8s-master-1            1/1       Running   0          5m
kube-controller-manager-k8s-master-1   1/1       Running   0          5m
kube-dns-686d6fb9c-xxrbg               3/3       Running   0          5m
kube-flannel-ds-gxt4n                  1/1       Running   0          23s
kube-flannel-ds-hngfv                  1/1       Running   0          2m
kube-flannel-ds-mgxdn                  1/1       Running   0          1m
kube-flannel-ds-qb8ch                  1/1       Running   0          3m
kube-proxy-4kxr8                       1/1       Running   0          1m
kube-proxy-54q5g                       1/1       Running   0          5m
kube-proxy-7zb4p                       1/1       Running   0          23s
kube-proxy-rwvp4                       1/1       Running   0          2m
kube-scheduler-k8s-master-1            1/1       Running   0          5m

danielvaughan commented Apr 21, 2018

I am running into the same problems as @carlosroman and @micedwards after applying weave on a 4 RPi 3 cluster:

Raspbian GNU/Linux 9 (stretch)
Docker version 18.04.0-ce, build 3d479c0
Kubernetes v1.10.1

pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS              RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running             22         10h
kube-apiserver-k8s-master-1            1/1       Running             39         10h
kube-controller-manager-k8s-master-1   1/1       Running             13         10h
kube-dns-686d6fb9c-qn2mp               0/3       Pending             0          10h
kube-proxy-6dlz4                       1/1       Running             11         9h
kube-proxy-7s977                       1/1       Running             2          9h
kube-proxy-q7jlh                       1/1       Running             11         10h
kube-proxy-qdmp7                       1/1       Running             2          9h
kube-scheduler-k8s-master-1            1/1       Running             13         10h
weave-net-5scxb                        2/2       Running             1          2m
weave-net-5vxzw                        1/2       CrashLoopBackOff    4          2m
weave-net-jmlzc                        1/2       ImageInspectError   0          2m
weave-net-xc2f8                        1/2       ImageInspectError   1          2m
pi@k8s-master-1:~ $
Message from syslogd@k8s-master-1 at Apr 22 08:04:14 ...
 kernel:[  155.252476] Internal error: Oops: 80000007 [#1] SMP ARM

I am having more luck with flannel

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.1.1.200 --pod-network-cidr=10.244.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
pi@k8s-master-1:~ $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          5m
kube-apiserver-k8s-master-1            1/1       Running   0          5m
kube-controller-manager-k8s-master-1   1/1       Running   0          5m
kube-dns-686d6fb9c-xxrbg               3/3       Running   0          5m
kube-flannel-ds-gxt4n                  1/1       Running   0          23s
kube-flannel-ds-hngfv                  1/1       Running   0          2m
kube-flannel-ds-mgxdn                  1/1       Running   0          1m
kube-flannel-ds-qb8ch                  1/1       Running   0          3m
kube-proxy-4kxr8                       1/1       Running   0          1m
kube-proxy-54q5g                       1/1       Running   0          5m
kube-proxy-7zb4p                       1/1       Running   0          23s
kube-proxy-rwvp4                       1/1       Running   0          2m
kube-scheduler-k8s-master-1            1/1       Running   0          5m
@peterska

This comment has been minimized.

Show comment
Hide comment
@peterska

peterska Apr 29, 2018

If you use flannel instead of weave networking the kernel oops does not occur. You can install flannel using
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
Make sure you install flannel before joining any nodes to the cluster.

peterska commented Apr 29, 2018

If you use flannel instead of weave networking the kernel oops does not occur. You can install flannel using
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
Make sure you install flannel before joining any nodes to the cluster.

@mcglinn

This comment has been minimized.

Show comment
Hide comment
@mcglinn

mcglinn Apr 29, 2018

I have the same issue - hanging at 'sudo kubeadm init' on master at line:
'[init] This might take a minute or longer if the control plane images have to be pulled.'
Tried so many different versions and options but still no luck.

I am using a raspberry pi 2 B+

Have used various raspbians Wheezy/Stretch various kubernetes up to latest (inc. 1.9.6 as suggested by PeterKing above) and various docker versions.

Anyone with this running on raspberry Pi 2 with recent raspbian, able to share version of all components (raspbian + kubernetes + docker)?

Please im sick of reflashing my SD :)

mcglinn commented Apr 29, 2018

I have the same issue - hanging at 'sudo kubeadm init' on master at line:
'[init] This might take a minute or longer if the control plane images have to be pulled.'
Tried so many different versions and options but still no luck.

I am using a raspberry pi 2 B+

Have used various raspbians Wheezy/Stretch various kubernetes up to latest (inc. 1.9.6 as suggested by PeterKing above) and various docker versions.

Anyone with this running on raspberry Pi 2 with recent raspbian, able to share version of all components (raspbian + kubernetes + docker)?

Please im sick of reflashing my SD :)

@Creamen

This comment has been minimized.

Show comment
Hide comment
@Creamen

Creamen Apr 29, 2018

Many many thanks for this bootstrap introduction !

I was facing issues with the latest version ( v1.10.2 - 28-04-2018 ) and after loosing some (more) hair - kube-apiserver was dying in loop ultimately leading to fail of kudeadm init - , I tried to downgrade both kubeadm and kubelet to 1.9.7-00 and - for now as it's a fresh start - things are up on my RPI3 cluster ... Cross finger :)
My cluster is a result of this simple :

sudo kubeadm init

Creamen commented Apr 29, 2018

Many many thanks for this bootstrap introduction !

I was facing issues with the latest version ( v1.10.2 - 28-04-2018 ) and after loosing some (more) hair - kube-apiserver was dying in loop ultimately leading to fail of kudeadm init - , I tried to downgrade both kubeadm and kubelet to 1.9.7-00 and - for now as it's a fresh start - things are up on my RPI3 cluster ... Cross finger :)
My cluster is a result of this simple :

sudo kubeadm init

@petershaw

This comment has been minimized.

Show comment
Hide comment
@petershaw

petershaw commented Apr 29, 2018

Kudo for @Jickelsen

@bjornramberg

This comment has been minimized.

Show comment
Hide comment
@bjornramberg

bjornramberg May 2, 2018

@alexellis: Thanks for a great guide! 👍
@Creamen: was facing exact same issue with same version, running on rpi3b+. The apiserver was looping, dying and geting stuck in a timeout on kubeadm init. After a downgrade it went through.

bjornramberg commented May 2, 2018

@alexellis: Thanks for a great guide! 👍
@Creamen: was facing exact same issue with same version, running on rpi3b+. The apiserver was looping, dying and geting stuck in a timeout on kubeadm init. After a downgrade it went through.

@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis May 4, 2018

Im seeing Weavenet being mentioned but I can’t see that anyone has logged an issue with kubeadm or weave - I’d suggest doing that if you are seeing unexpected behaviour with newer versions of the components.

The init step can take several minutes.

Owner

alexellis commented May 4, 2018

Im seeing Weavenet being mentioned but I can’t see that anyone has logged an issue with kubeadm or weave - I’d suggest doing that if you are seeing unexpected behaviour with newer versions of the components.

The init step can take several minutes.

@aaronkjones

This comment has been minimized.

Show comment
Hide comment
@aaronkjones

aaronkjones May 5, 2018

I have attempted to get Weave Net to work with k8s v1.10.2 and exausted all options it seems. I could only get it working with flannel as the CNI. I also had to add and change some commands.

I forked your gist and made the modifications including changing the script (prep.sh)

Try it out
https://github.com/aaronkjones/rpi-k8s-node-prep

pi@k8s-master-1:~ $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    2h        v1.10.2
k8s-slave-1    Ready     <none>    2h        v1.10.2
k8s-slave-2    Ready     <none>    33m       v1.10.2
k8s-slave-3    Ready     <none>    33m       v1.10.2
pi@k8s-master-1:~ $ kubectl get pods -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          2h
kube-apiserver-k8s-master-1            1/1       Running   0          2h
kube-controller-manager-k8s-master-1   1/1       Running   0          2h
kube-dns-686d6fb9c-glz8x               3/3       Running   0          2h
kube-flannel-ds-5grwv                  1/1       Running   0          2h
kube-flannel-ds-756mt                  1/1       Running   0          33m
kube-flannel-ds-7hvdg                  1/1       Running   0          33m
kube-flannel-ds-k6hsn                  1/1       Running   0          2h
kube-proxy-hzpkw                       1/1       Running   0          33m
kube-proxy-wsj9v                       1/1       Running   0          2h
kube-proxy-xjvvp                       1/1       Running   0          2h
kube-proxy-z5ngl                       1/1       Running   0          33m
kube-scheduler-k8s-master-1            1/1       Running   0          2h
kubernetes-dashboard-64d66bcc8-vcc5v   1/1       Running   0          22m

aaronkjones commented May 5, 2018

I have attempted to get Weave Net to work with k8s v1.10.2 and exausted all options it seems. I could only get it working with flannel as the CNI. I also had to add and change some commands.

I forked your gist and made the modifications including changing the script (prep.sh)

Try it out
https://github.com/aaronkjones/rpi-k8s-node-prep

pi@k8s-master-1:~ $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    2h        v1.10.2
k8s-slave-1    Ready     <none>    2h        v1.10.2
k8s-slave-2    Ready     <none>    33m       v1.10.2
k8s-slave-3    Ready     <none>    33m       v1.10.2
pi@k8s-master-1:~ $ kubectl get pods -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running   0          2h
kube-apiserver-k8s-master-1            1/1       Running   0          2h
kube-controller-manager-k8s-master-1   1/1       Running   0          2h
kube-dns-686d6fb9c-glz8x               3/3       Running   0          2h
kube-flannel-ds-5grwv                  1/1       Running   0          2h
kube-flannel-ds-756mt                  1/1       Running   0          33m
kube-flannel-ds-7hvdg                  1/1       Running   0          33m
kube-flannel-ds-k6hsn                  1/1       Running   0          2h
kube-proxy-hzpkw                       1/1       Running   0          33m
kube-proxy-wsj9v                       1/1       Running   0          2h
kube-proxy-xjvvp                       1/1       Running   0          2h
kube-proxy-z5ngl                       1/1       Running   0          33m
kube-scheduler-k8s-master-1            1/1       Running   0          2h
kubernetes-dashboard-64d66bcc8-vcc5v   1/1       Running   0          22m
@vieskees

This comment has been minimized.

Show comment
Hide comment
@vieskees

vieskees May 7, 2018

@alexellis : thanks for the guide, it really helped me. I've been trying to get it working with Weave for a couple of days, but in the end I gave up and went with @aaronkjones 's idea. I used flannel as the CNI and got it working on the first try.

vieskees commented May 7, 2018

@alexellis : thanks for the guide, it really helped me. I've been trying to get it working with Weave for a couple of days, but in the end I gave up and went with @aaronkjones 's idea. I used flannel as the CNI and got it working on the first try.

@elafargue

This comment has been minimized.

Show comment
Hide comment
@elafargue

elafargue May 10, 2018

Same here: @aaronkjones 's guide is what worked for me as well. I took the liberty of creating a variant of this gist for those who want to use Hypriot. It also covers networking setup a bit more in-depth (local ethernet for the cluster, wifi connection via the master to reach the outside world): https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f . Still a work in progress.

elafargue commented May 10, 2018

Same here: @aaronkjones 's guide is what worked for me as well. I took the liberty of creating a variant of this gist for those who want to use Hypriot. It also covers networking setup a bit more in-depth (local ethernet for the cluster, wifi connection via the master to reach the outside world): https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f . Still a work in progress.

@mgoodfellow

This comment has been minimized.

Show comment
Hide comment
@mgoodfellow

mgoodfellow May 13, 2018

Just as a heads up - @aaronkjones solution was working for me perfectly last week, but I added new worker nodes to my existing cluster today and the new nodes don't initialise flannel or kube-proxy:

flannel:

Error: failed to start container "install-cni": Error response from daemon: linux mounts: Could not find source mount of /var/lib/kubelet/pods/532b1489-569a-11e8-aed4-b827eb359bc2/volumes/kubernetes.io~configmap/flannel-cfg
Back-off restarting failed container

kube-proxy:

Error: failed to start container "kube-proxy": Error response from daemon: linux mounts: Could not find source mount of /lib/modules
Back-off restarting failed container

Turns out, as of the last couple of days the get-docker install script now installs docker-ce 18.05 as the latest version, and this seems to cause this error.

Step in @alexellis instructions above (which installs latest version of docker):

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

I just downgraded the docker version on my nodes to docker-ce 18.04, rebooted, and everything seems OK now and my nodes initialise correctly.

$ sudo apt-get install docker-ce=18.04.0~ce~3-0~raspbian

You can check out the packages added to the repo lists by using:

apt-cache madison docker-ce

Hope this helps someone! Massive thanks to @alexellis and everyone else in this thread who have got me a working K8s cluster on rPi's - learnt loads!

mgoodfellow commented May 13, 2018

Just as a heads up - @aaronkjones solution was working for me perfectly last week, but I added new worker nodes to my existing cluster today and the new nodes don't initialise flannel or kube-proxy:

flannel:

Error: failed to start container "install-cni": Error response from daemon: linux mounts: Could not find source mount of /var/lib/kubelet/pods/532b1489-569a-11e8-aed4-b827eb359bc2/volumes/kubernetes.io~configmap/flannel-cfg
Back-off restarting failed container

kube-proxy:

Error: failed to start container "kube-proxy": Error response from daemon: linux mounts: Could not find source mount of /lib/modules
Back-off restarting failed container

Turns out, as of the last couple of days the get-docker install script now installs docker-ce 18.05 as the latest version, and this seems to cause this error.

Step in @alexellis instructions above (which installs latest version of docker):

$ curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

I just downgraded the docker version on my nodes to docker-ce 18.04, rebooted, and everything seems OK now and my nodes initialise correctly.

$ sudo apt-get install docker-ce=18.04.0~ce~3-0~raspbian

You can check out the packages added to the repo lists by using:

apt-cache madison docker-ce

Hope this helps someone! Massive thanks to @alexellis and everyone else in this thread who have got me a working K8s cluster on rPi's - learnt loads!

@toolboc

This comment has been minimized.

Show comment
Hide comment
@toolboc

toolboc May 20, 2018

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

toolboc commented May 20, 2018

Hangs @ [init] This might take a minute or longer if the control plane images have to be pulled. on Raspberry Pi 3 B with Docker 18.05 and kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00

Fixed by downgrading kubeadm, kubectl, and kubelet to 1.9.6:
sudo apt-get install -qy kubeadm=1.9.6-00 kubectl=1.9.6-00 kubelet=1.9.6-00

AND

Downgrading to Docker 18.04:
sudo aptitude install -qy docker-ce=18.04.0~ce~3-0~raspbian

@aaronkjones

This comment has been minimized.

Show comment
Hide comment
@aaronkjones

aaronkjones May 21, 2018

https://github.com/aaronkjones/rpi-k8s-node-prep

I modified the script to allow for a specific version of Docker and Kubeadm to be installed and also pinned to prevent upgrade.

I have 4 RPis, so i made two two-node clusters and tried different combinations of Kubeadm/Docker.

Docker 18.04 and Kubeadm 1.10.2-00 work for me. It has been running on Hypriot for a few days.

aaronkjones commented May 21, 2018

https://github.com/aaronkjones/rpi-k8s-node-prep

I modified the script to allow for a specific version of Docker and Kubeadm to be installed and also pinned to prevent upgrade.

I have 4 RPis, so i made two two-node clusters and tried different combinations of Kubeadm/Docker.

Docker 18.04 and Kubeadm 1.10.2-00 work for me. It has been running on Hypriot for a few days.

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn May 23, 2018

Where or how do we report the issue to?

njohnsn commented May 23, 2018

Where or how do we report the issue to?

@thapli88

This comment has been minimized.

Show comment
Hide comment
@thapli88

thapli88 May 23, 2018

I couldn't get 1.10.3-00 working either.

For 1.10.2-00, in addition to downloading, installing, and holding the right packages, when you init the master, you need to set the version there too (otherwise, it'll default download the latest stable control images, which are 1.10.3)

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.10.2

thapli88 commented May 23, 2018

I couldn't get 1.10.3-00 working either.

For 1.10.2-00, in addition to downloading, installing, and holding the right packages, when you init the master, you need to set the version there too (otherwise, it'll default download the latest stable control images, which are 1.10.3)

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.10.2

@geerlingguy

This comment has been minimized.

Show comment
Hide comment
@geerlingguy

geerlingguy May 23, 2018

@njohnsn, probably an issue on the Kubernetes repository at https://github.com/kubernetes/kubernetes/issues

I ran into the same issue, was getting errors like the one in this comment with the latest version of kubelet: geerlingguy/raspberry-pi-dramble#100 (comment)

I uninstalled docker-ce then reinstalled with sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian.

The init command I used (after installing with sudo apt-get install -y kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00) was:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=PI_IP_HERE --kubernetes-version v1.10.2

I had to downgrade both Kubernetes (from 1.10.3 to 1.10.2) and Docker CE (from 18.05.0 to 18.04.0) to Kubernetes to boot and run on Debian Stretch (Raspbian Lite)... but I finally got to:

# kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                  STATUS     ROLES     AGE       VERSION
kube1.pidramble.com   NotReady   master    5m        v1.10.2

geerlingguy commented May 23, 2018

@njohnsn, probably an issue on the Kubernetes repository at https://github.com/kubernetes/kubernetes/issues

I ran into the same issue, was getting errors like the one in this comment with the latest version of kubelet: geerlingguy/raspberry-pi-dramble#100 (comment)

I uninstalled docker-ce then reinstalled with sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian.

The init command I used (after installing with sudo apt-get install -y kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00) was:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=PI_IP_HERE --kubernetes-version v1.10.2

I had to downgrade both Kubernetes (from 1.10.3 to 1.10.2) and Docker CE (from 18.05.0 to 18.04.0) to Kubernetes to boot and run on Debian Stretch (Raspbian Lite)... but I finally got to:

# kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                  STATUS     ROLES     AGE       VERSION
kube1.pidramble.com   NotReady   master    5m        v1.10.2
@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn May 24, 2018

For some reason I can't get my fourth node to go into the ready state.

I've blown the SD card a way and reinstalled everything from scratch twice, but unlike the other 3 nodes, it won't come up.

Here is the output from syslog:

May 24 02:50:56 k8s-node-4 dockerd[479]: time="2018-05-24T02:50:56.309407492Z" level=error msg="Handler for GET /v1.31/images/weaveworks/weave-kube:2.3.0/json returned error: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311048     298 remote_image.go:83] ImageStatus "weaveworks/weave-kube:2.3.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311172     298 kuberuntime_image.go:87] ImageStatus for image {"weaveworks/weave-kube:2.3.0"} failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311326     298 kuberuntime_manager.go:733] container start failed: ImageInspectError: Failed to inspect image "weaveworks/weave-kube:2.3.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311433     298 pod_workers.go:186] Error syncing pod 95d24908-5efa-11e8-b36b-b827ebccbe66 ("weave-net-hxgx5_kube-system(95d24908-5efa-11e8-b36b-b827ebccbe66)"), skipping: failed to "StartContainer" for "weave" with ImageInspectError: "Failed to inspect image \"weaveworks/weave-kube:2.3.0\": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:58 k8s-node-4 kubelet[298]: W0524 02:50:58.686588     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:50:58 k8s-node-4 kubelet[298]: E0524 02:50:58.689475     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:03 k8s-node-4 kubelet[298]: W0524 02:51:03.694277     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:03 k8s-node-4 kubelet[298]: E0524 02:51:03.695457     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:08 k8s-node-4 kubelet[298]: W0524 02:51:08.700522     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:08 k8s-node-4 kubelet[298]: E0524 02:51:08.701369     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Thanks!

njohnsn commented May 24, 2018

For some reason I can't get my fourth node to go into the ready state.

I've blown the SD card a way and reinstalled everything from scratch twice, but unlike the other 3 nodes, it won't come up.

Here is the output from syslog:

May 24 02:50:56 k8s-node-4 dockerd[479]: time="2018-05-24T02:50:56.309407492Z" level=error msg="Handler for GET /v1.31/images/weaveworks/weave-kube:2.3.0/json returned error: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311048     298 remote_image.go:83] ImageStatus "weaveworks/weave-kube:2.3.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311172     298 kuberuntime_image.go:87] ImageStatus for image {"weaveworks/weave-kube:2.3.0"} failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311326     298 kuberuntime_manager.go:733] container start failed: ImageInspectError: Failed to inspect image "weaveworks/weave-kube:2.3.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
May 24 02:50:56 k8s-node-4 kubelet[298]: E0524 02:50:56.311433     298 pod_workers.go:186] Error syncing pod 95d24908-5efa-11e8-b36b-b827ebccbe66 ("weave-net-hxgx5_kube-system(95d24908-5efa-11e8-b36b-b827ebccbe66)"), skipping: failed to "StartContainer" for "weave" with ImageInspectError: "Failed to inspect image \"weaveworks/weave-kube:2.3.0\": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument"
May 24 02:50:58 k8s-node-4 kubelet[298]: W0524 02:50:58.686588     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:50:58 k8s-node-4 kubelet[298]: E0524 02:50:58.689475     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:03 k8s-node-4 kubelet[298]: W0524 02:51:03.694277     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:03 k8s-node-4 kubelet[298]: E0524 02:51:03.695457     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 24 02:51:08 k8s-node-4 kubelet[298]: W0524 02:51:08.700522     298 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 24 02:51:08 k8s-node-4 kubelet[298]: E0524 02:51:08.701369     298 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Thanks!

@carlosroman

This comment has been minimized.

Show comment
Hide comment
@carlosroman

carlosroman May 27, 2018

@njohnsn I never got weave to work but have got flannel working. After looking into the issue I couldn't resolve the whole ImageInspectError so gave up. Out of 4 nodes I had two in the cluster and two wouldn't ever connect. Tried clean installs on them and still nothing. Strangely enough if I did a clean install of the cluster it would be different PIs that would fail :/

To get flannel to work I had to update /etc/kubernetes/manifests/kube-controller-manager.yaml and update the spec containers command with:
- --allocate-node-cidrs=true - --cluster-cidr=172.30.0.0/16
After reloading systemd (systemctl daemon-reload) and restarting k8s (systemctl restart kubelet.service) I was able to get flannel working.

I think you won't need that if you run kubeadm init with the flag --pod-network-cidr <network cidr> but not tried that since I got my cluster up and running. Will give it ago when I got more time (and update my playbook https://github.com/carlosroman/ansible-k8s-raspberry-playbook/) and see if flannel "just works".

carlosroman commented May 27, 2018

@njohnsn I never got weave to work but have got flannel working. After looking into the issue I couldn't resolve the whole ImageInspectError so gave up. Out of 4 nodes I had two in the cluster and two wouldn't ever connect. Tried clean installs on them and still nothing. Strangely enough if I did a clean install of the cluster it would be different PIs that would fail :/

To get flannel to work I had to update /etc/kubernetes/manifests/kube-controller-manager.yaml and update the spec containers command with:
- --allocate-node-cidrs=true - --cluster-cidr=172.30.0.0/16
After reloading systemd (systemctl daemon-reload) and restarting k8s (systemctl restart kubelet.service) I was able to get flannel working.

I think you won't need that if you run kubeadm init with the flag --pod-network-cidr <network cidr> but not tried that since I got my cluster up and running. Will give it ago when I got more time (and update my playbook https://github.com/carlosroman/ansible-k8s-raspberry-playbook/) and see if flannel "just works".

@grmhay

This comment has been minimized.

Show comment
Hide comment
@grmhay

grmhay May 28, 2018

I can report (eventual) success with the following configuration on my 4 Raspberry pis:

  • Stretch Lite (2018-04-08)

  • v.18.04.0-ce Docker-ce [sudo apt-get install docker-ce=18.04.0ce3-0~raspbian]

  • Flannel (could not get my nodes to get to Ready state with Weave) [curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f - ]
    Note: no further editing of Kubernetes manifest files as suggested previously was required

  • v1.10.2 of Kubernetes (like many before, had to back rev from 1.10.3) [ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address= --kubernetes-version v1.10.2 --pod-network-cidr= ]

And thanks Alex for this original post and others who commented. Hoping this helps someone else struggling ....

grmhay commented May 28, 2018

I can report (eventual) success with the following configuration on my 4 Raspberry pis:

  • Stretch Lite (2018-04-08)

  • v.18.04.0-ce Docker-ce [sudo apt-get install docker-ce=18.04.0ce3-0~raspbian]

  • Flannel (could not get my nodes to get to Ready state with Weave) [curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f - ]
    Note: no further editing of Kubernetes manifest files as suggested previously was required

  • v1.10.2 of Kubernetes (like many before, had to back rev from 1.10.3) [ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address= --kubernetes-version v1.10.2 --pod-network-cidr= ]

And thanks Alex for this original post and others who commented. Hoping this helps someone else struggling ....

@liweiz

This comment has been minimized.

Show comment
Hide comment
@liweiz

liweiz May 29, 2018

I can not have dashboard work using proxy. I got the error message:

Error: 'dial tcp 172.17.0.2:9090: getsockopt: connection refused'
Trying to reach: 'https://172.17.0.2:9090/'

I followed the steps and learned some rbac along the way, but still can not figure out where to look at to solve this.

Any suggestions?

I also found this from kubernetes/dashboard readme:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

From what I have so far, my cluster did not install Heapster. Is it necessary to mention that in this guide?

liweiz commented May 29, 2018

I can not have dashboard work using proxy. I got the error message:

Error: 'dial tcp 172.17.0.2:9090: getsockopt: connection refused'
Trying to reach: 'https://172.17.0.2:9090/'

I followed the steps and learned some rbac along the way, but still can not figure out where to look at to solve this.

Any suggestions?

I also found this from kubernetes/dashboard readme:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

From what I have so far, my cluster did not install Heapster. Is it necessary to mention that in this guide?

@kumardeepam

This comment has been minimized.

Show comment
Hide comment
@kumardeepam

kumardeepam May 30, 2018

I am late to the fun :) has anyone followed this on the latest?
RASPBIAN STRETCH WITH DESKTOP
Image with desktop based on Debian Stretch
Version:April 2018
Release date:2018-04-18
Kernel version:4.14

I have not been able to progress beyond this
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.10.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-1] and IPs [192.168.0.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm:v1.10.3
- k8s.gcr.io/kube-controller-manager-arm:v1.10.3
- k8s.gcr.io/kube-scheduler-arm:v1.10.3
- k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Appreciate your time and help

kumardeepam commented May 30, 2018

I am late to the fun :) has anyone followed this on the latest?
RASPBIAN STRETCH WITH DESKTOP
Image with desktop based on Debian Stretch
Version:April 2018
Release date:2018-04-18
Kernel version:4.14

I have not been able to progress beyond this
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.10.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-1] and IPs [192.168.0.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm:v1.10.3
- k8s.gcr.io/kube-controller-manager-arm:v1.10.3
- k8s.gcr.io/kube-scheduler-arm:v1.10.3
- k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Appreciate your time and help

@kumardeepam

This comment has been minimized.

Show comment
Hide comment
@kumardeepam

kumardeepam May 30, 2018

@Creamen downgrading as you advised worked for me. Wasted 2 nights.
Anyone facing a similar issue here is how to downgrade (uninstall the latest version and install v1.9.700)

Step 1: uninstall kubernetes, follow the below given commands
$kubeadm reset
$sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$sudo apt-get autoremove
$sudo rm -rf ~/.kube

Step 2: reboot your pi
$sudo reboot

Step 3: Install v1.9.7-00, follow the below given commands
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

Step 4: Initiate your master node
$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

kumardeepam commented May 30, 2018

@Creamen downgrading as you advised worked for me. Wasted 2 nights.
Anyone facing a similar issue here is how to downgrade (uninstall the latest version and install v1.9.700)

Step 1: uninstall kubernetes, follow the below given commands
$kubeadm reset
$sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$sudo apt-get autoremove
$sudo rm -rf ~/.kube

Step 2: reboot your pi
$sudo reboot

Step 3: Install v1.9.7-00, follow the below given commands
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

Step 4: Initiate your master node
$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 2, 2018

Still no luck.
I switched to flannel but now the dns pod doesn't work:
commands I used after downgrading to Docker 18.0.4 and k8s 10.1.2.

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.1.240 --pod-network-cidr=172.30.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:/var/log/containers $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    41m       v1.10.2
k8s-node-1     Ready     <none>    40m       v1.10.2
k8s-node-2     Ready     <none>    40m       v1.10.2
k8s-node-3     Ready     <none>    40m       v1.10.2
k8s-node-4     Ready     <none>    40m       v1.10.2
k8s-node-5     Ready     <none>    40m       v1.10.2

pi@k8s-master-1:/var/log/containers $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS             RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running            0          41m
kube-apiserver-k8s-master-1            1/1       Running            0          42m
kube-controller-manager-k8s-master-1   1/1       Running            0          42m
kube-dns-686d6fb9c-ms85s               1/3       CrashLoopBackOff   27         42m
kube-flannel-ds-2nfrv                  1/1       Running            0          36m
kube-flannel-ds-8j8vh                  1/1       Running            0          36m
kube-flannel-ds-b2dfb                  1/1       Running            0          36m
kube-flannel-ds-qxrt6                  1/1       Running            0          36m
kube-flannel-ds-rr4dr                  1/1       Running            0          36m
kube-flannel-ds-w92sm                  1/1       Running            0          36m
kube-proxy-6f7g5                       1/1       Running            0          41m
kube-proxy-7w57r                       1/1       Running            0          41m
kube-proxy-8lvkq                       1/1       Running            0          41m
kube-proxy-f4x4s                       1/1       Running            0          41m
kube-proxy-psll8                       1/1       Running            0          42m
kube-proxy-ttrdn                       1/1       Running            0          41m
kube-scheduler-k8s-master-1            1/1       Running            0          42m
pi@k8s-master-1:/var/log/containers $

Where do I find the logs for the dns pod?

Thanks!

njohnsn commented Jun 2, 2018

Still no luck.
I switched to flannel but now the dns pod doesn't work:
commands I used after downgrading to Docker 18.0.4 and k8s 10.1.2.

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.1.240 --pod-network-cidr=172.30.0.0/16
curl -sSL https://rawgit.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

pi@k8s-master-1:/var/log/containers $ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
k8s-master-1   Ready     master    41m       v1.10.2
k8s-node-1     Ready     <none>    40m       v1.10.2
k8s-node-2     Ready     <none>    40m       v1.10.2
k8s-node-3     Ready     <none>    40m       v1.10.2
k8s-node-4     Ready     <none>    40m       v1.10.2
k8s-node-5     Ready     <none>    40m       v1.10.2

pi@k8s-master-1:/var/log/containers $ kubectl get pods --namespace=kube-system
NAME                                   READY     STATUS             RESTARTS   AGE
etcd-k8s-master-1                      1/1       Running            0          41m
kube-apiserver-k8s-master-1            1/1       Running            0          42m
kube-controller-manager-k8s-master-1   1/1       Running            0          42m
kube-dns-686d6fb9c-ms85s               1/3       CrashLoopBackOff   27         42m
kube-flannel-ds-2nfrv                  1/1       Running            0          36m
kube-flannel-ds-8j8vh                  1/1       Running            0          36m
kube-flannel-ds-b2dfb                  1/1       Running            0          36m
kube-flannel-ds-qxrt6                  1/1       Running            0          36m
kube-flannel-ds-rr4dr                  1/1       Running            0          36m
kube-flannel-ds-w92sm                  1/1       Running            0          36m
kube-proxy-6f7g5                       1/1       Running            0          41m
kube-proxy-7w57r                       1/1       Running            0          41m
kube-proxy-8lvkq                       1/1       Running            0          41m
kube-proxy-f4x4s                       1/1       Running            0          41m
kube-proxy-psll8                       1/1       Running            0          42m
kube-proxy-ttrdn                       1/1       Running            0          41m
kube-scheduler-k8s-master-1            1/1       Running            0          42m
pi@k8s-master-1:/var/log/containers $

Where do I find the logs for the dns pod?

Thanks!

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 3, 2018

Found the logs for the DNS pod"

pi@k8s-master-1:~ $ kubectl logs -f --namespace=kube-system kube-dns-686d6fb9c-ms85s -c kubedns
I0603 14:37:58.522134       1 dns.go:48] version: 1.14.8
I0603 14:37:58.526437       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0603 14:37:58.526863       1 server.go:119] FLAG: --alsologtostderr="false"
I0603 14:37:58.527113       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0603 14:37:58.527202       1 server.go:119] FLAG: --config-map=""
I0603 14:37:58.527393       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0603 14:37:58.527473       1 server.go:119] FLAG: --config-period="10s"
I0603 14:37:58.527712       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0603 14:37:58.527788       1 server.go:119] FLAG: --dns-port="10053"
I0603 14:37:58.528038       1 server.go:119] FLAG: --domain="cluster.local."
I0603 14:37:58.528123       1 server.go:119] FLAG: --federations=""
I0603 14:37:58.528348       1 server.go:119] FLAG: --healthz-port="8081"
I0603 14:37:58.528425       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0603 14:37:58.528658       1 server.go:119] FLAG: --kube-master-url=""
I0603 14:37:58.528740       1 server.go:119] FLAG: --kubecfg-file=""
I0603 14:37:58.528935       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0603 14:37:58.529036       1 server.go:119] FLAG: --log-dir=""
I0603 14:37:58.529226       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0603 14:37:58.529307       1 server.go:119] FLAG: --logtostderr="true"
I0603 14:37:58.529497       1 server.go:119] FLAG: --nameservers=""
I0603 14:37:58.529524       1 server.go:119] FLAG: --stderrthreshold="2"
I0603 14:37:58.529547       1 server.go:119] FLAG: --v="2"
I0603 14:37:58.529570       1 server.go:119] FLAG: --version="false"
I0603 14:37:58.529602       1 server.go:119] FLAG: --vmodule=""
I0603 14:37:58.530045       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0603 14:37:58.531256       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0603 14:37:58.531308       1 dns.go:146] Starting endpointsController
I0603 14:37:58.531334       1 dns.go:149] Starting serviceController
I0603 14:37:58.533533       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0603 14:37:58.533610       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0603 14:37:59.032154       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:37:59.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.032087       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.532194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.532223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.032227       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.032217       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.532270       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.032212       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.532256       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.032242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.532230       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.032174       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.532176       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.032166       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.032207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.532207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.032224       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.532219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.032115       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.532079       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.032109       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.532204       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.032178       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.532142       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.032191       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.532209       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.032232       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.032219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.532118       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.032194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.532202       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.032177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.532185       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.032170       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.532257       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.032189       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.532366       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.032243       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.532134       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.032122       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.032221       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.532175       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.032214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.532241       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.032198       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.532179       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.032288       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.532242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.032240       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.032140       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.532060       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

It appears the DNS process is trying to check on the api status on port 443 when the api server is running on port 6443.
I don't know where to change it from though.

-Neil

njohnsn commented Jun 3, 2018

Found the logs for the DNS pod"

pi@k8s-master-1:~ $ kubectl logs -f --namespace=kube-system kube-dns-686d6fb9c-ms85s -c kubedns
I0603 14:37:58.522134       1 dns.go:48] version: 1.14.8
I0603 14:37:58.526437       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0603 14:37:58.526863       1 server.go:119] FLAG: --alsologtostderr="false"
I0603 14:37:58.527113       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0603 14:37:58.527202       1 server.go:119] FLAG: --config-map=""
I0603 14:37:58.527393       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0603 14:37:58.527473       1 server.go:119] FLAG: --config-period="10s"
I0603 14:37:58.527712       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0603 14:37:58.527788       1 server.go:119] FLAG: --dns-port="10053"
I0603 14:37:58.528038       1 server.go:119] FLAG: --domain="cluster.local."
I0603 14:37:58.528123       1 server.go:119] FLAG: --federations=""
I0603 14:37:58.528348       1 server.go:119] FLAG: --healthz-port="8081"
I0603 14:37:58.528425       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0603 14:37:58.528658       1 server.go:119] FLAG: --kube-master-url=""
I0603 14:37:58.528740       1 server.go:119] FLAG: --kubecfg-file=""
I0603 14:37:58.528935       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0603 14:37:58.529036       1 server.go:119] FLAG: --log-dir=""
I0603 14:37:58.529226       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0603 14:37:58.529307       1 server.go:119] FLAG: --logtostderr="true"
I0603 14:37:58.529497       1 server.go:119] FLAG: --nameservers=""
I0603 14:37:58.529524       1 server.go:119] FLAG: --stderrthreshold="2"
I0603 14:37:58.529547       1 server.go:119] FLAG: --v="2"
I0603 14:37:58.529570       1 server.go:119] FLAG: --version="false"
I0603 14:37:58.529602       1 server.go:119] FLAG: --vmodule=""
I0603 14:37:58.530045       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0603 14:37:58.531256       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0603 14:37:58.531308       1 dns.go:146] Starting endpointsController
I0603 14:37:58.531334       1 dns.go:149] Starting serviceController
I0603 14:37:58.533533       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0603 14:37:58.533610       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0603 14:37:59.032154       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:37:59.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.032087       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:00.532194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:01.532223       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.032227       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:02.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.032217       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:03.532270       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.032212       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:04.532256       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.032242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:05.532230       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.032199       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:06.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.032174       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:07.532176       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.032166       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:08.532177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.032207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:09.532207       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.032224       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:10.532219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.032115       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:11.532079       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.032109       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:12.532204       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.032178       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:13.532142       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.032191       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:14.532209       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.032232       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:15.532214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.032219       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:16.532118       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.032194       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:17.532202       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.032177       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:18.532185       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.032170       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:19.532257       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.032189       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:20.532366       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.032243       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:21.532134       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.032122       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:22.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.032221       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:23.532175       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.032214       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:24.532241       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.032198       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:25.532179       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.032288       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:26.532242       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.032240       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:27.532260       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.032140       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0603 14:38:28.532060       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0603 14:38:28.533618       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

It appears the DNS process is trying to check on the api status on port 443 when the api server is running on port 6443.
I don't know where to change it from though.

-Neil

@njohnsn

This comment has been minimized.

Show comment
Hide comment
@njohnsn

njohnsn Jun 4, 2018

Turns out the answer was here
I had to bump the version of kube-dns to 1.14.10.

njohnsn commented Jun 4, 2018

Turns out the answer was here
I had to bump the version of kube-dns to 1.14.10.

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Jun 4, 2018

@njohnsn Logs are shown by kubectl logs. A kube-dns pod is made up of 3 containers but the one you need to watch is kubedns(the others are dnsmasq& sidecar). --follow is if you want to live follow the logs. In the above case you could use:

kubectl logs --follow pods/kube-dns-686d6fb9c-ms85s -c kubedns --namespace=kube-system

micedwards commented Jun 4, 2018

@njohnsn Logs are shown by kubectl logs. A kube-dns pod is made up of 3 containers but the one you need to watch is kubedns(the others are dnsmasq& sidecar). --follow is if you want to live follow the logs. In the above case you could use:

kubectl logs --follow pods/kube-dns-686d6fb9c-ms85s -c kubedns --namespace=kube-system

@shreyasnaik

This comment has been minimized.

Show comment
Hide comment
@shreyasnaik

shreyasnaik Jun 5, 2018

@kumardeepam Thanks for you work and instructions on how to downgrade (uninstall the latest version and install v1.9.700) when getting an error "[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2"

My master is running now!

Also had to set cgroup_memory=memory not cgroup_memory=1.

shreyasnaik commented Jun 5, 2018

@kumardeepam Thanks for you work and instructions on how to downgrade (uninstall the latest version and install v1.9.700) when getting an error "[WARNING KubeletVersion]: couldn't get kubelet version: exit status 2"

My master is running now!

Also had to set cgroup_memory=memory not cgroup_memory=1.

@tomtom215

This comment has been minimized.

Show comment
Hide comment
@tomtom215

tomtom215 Jun 27, 2018

useful way of downgrading docker is after running --
$ curl -sSL get.docker.com | sh && \ sudo usermod pi -aG docker

run this to downgrade without having to go through a full uninstall --
sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian --allow-downgrades

tomtom215 commented Jun 27, 2018

useful way of downgrading docker is after running --
$ curl -sSL get.docker.com | sh && \ sudo usermod pi -aG docker

run this to downgrade without having to go through a full uninstall --
sudo apt-get install -y docker-ce=18.04.0~ce~3-0~raspbian --allow-downgrades

@tzolov

This comment has been minimized.

Show comment
Hide comment
@tzolov

tzolov Jun 28, 2018

Thanks for the tip @tomtom215!

Following the thread above i wonder if we need to downgrade both the k8s and docker versions? Or just downgrading docker is enough?

tzolov commented Jun 28, 2018

Thanks for the tip @tomtom215!

Following the thread above i wonder if we need to downgrade both the k8s and docker versions? Or just downgrading docker is enough?

@Gallouche

This comment has been minimized.

Show comment
Hide comment
@Gallouche

Gallouche Jun 29, 2018

Hi there, did someone succeed to deploy it with docker 18.05 and k8s 1.11 ? I tried to but got issue I don't understand, I'm new to docker and k8s...

Got this when trying to join :

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

and the controller pod keep crashing on my masternode then :

pi@MasterNode:~ $ kubectl get all --all-namespaces -o wide
NAMESPACE     NAME                                     READY     STATUS             RESTARTS   AGE       IP              NODE
kube-system   pod/coredns-78fcdf6894-mwx54             0/1       Pending            0          18m       <none>          <none>
kube-system   pod/coredns-78fcdf6894-skpvr             0/1       Pending            0          17m       <none>          <none>
kube-system   pod/etcd-masternode                      1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-apiserver-masternode            1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-controller-manager-masternode   0/1       CrashLoopBackOff   7          19m       10.192.79.250   masternode
kube-system   pod/kube-proxy-44gcl                     1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-scheduler-masternode            1/1       Running            0          18m       10.192.79.250   masternode

There is a configuration that work well for someone ? Which versions and which CNI ?

Thanks by advance :)

Gallouche commented Jun 29, 2018

Hi there, did someone succeed to deploy it with docker 18.05 and k8s 1.11 ? I tried to but got issue I don't understand, I'm new to docker and k8s...

Got this when trying to join :

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

and the controller pod keep crashing on my masternode then :

pi@MasterNode:~ $ kubectl get all --all-namespaces -o wide
NAMESPACE     NAME                                     READY     STATUS             RESTARTS   AGE       IP              NODE
kube-system   pod/coredns-78fcdf6894-mwx54             0/1       Pending            0          18m       <none>          <none>
kube-system   pod/coredns-78fcdf6894-skpvr             0/1       Pending            0          17m       <none>          <none>
kube-system   pod/etcd-masternode                      1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-apiserver-masternode            1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-controller-manager-masternode   0/1       CrashLoopBackOff   7          19m       10.192.79.250   masternode
kube-system   pod/kube-proxy-44gcl                     1/1       Running            0          18m       10.192.79.250   masternode
kube-system   pod/kube-scheduler-masternode            1/1       Running            0          18m       10.192.79.250   masternode

There is a configuration that work well for someone ? Which versions and which CNI ?

Thanks by advance :)

@BathAndy

This comment has been minimized.

Show comment
Hide comment
@BathAndy

BathAndy Jun 30, 2018

Mine seems to be working fine . I'm using ce=18.04.0ce3-0~raspbian kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
with flannel as the CNI . As mentioned above , the version of kubedns does not place nice. One you have done the cluster installation then do
kubectl edit deploy kube-dns --namespace=kube-system to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors
I now have OpenFAAS and prometheus-operator (arm) running fine it , even using the kubernetes incuabtor nfs-client-provisioner so I can use a non-clustered pi as an NFS server for PV and PVC's

sudo  sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  # DO this on MASTER NODE ONLY

Now initialise the master node 

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.1.62
# note 192.168.1.62 is the ip of my master node```

BathAndy commented Jun 30, 2018

Mine seems to be working fine . I'm using ce=18.04.0ce3-0~raspbian kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
with flannel as the CNI . As mentioned above , the version of kubedns does not place nice. One you have done the cluster installation then do
kubectl edit deploy kube-dns --namespace=kube-system to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors
I now have OpenFAAS and prometheus-operator (arm) running fine it , even using the kubernetes incuabtor nfs-client-provisioner so I can use a non-clustered pi as an NFS server for PV and PVC's

sudo  sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  # DO this on MASTER NODE ONLY

Now initialise the master node 

We pass in --token-ttl=0 so that the token never expires - do not use this setting in production. The UX for kubeadm means it's currently very hard to get a join token later on after the initial token has expired.

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.1.62
# note 192.168.1.62 is the ip of my master node```

@BathAndy

This comment has been minimized.

Show comment
Hide comment
@BathAndy

BathAndy Jun 30, 2018

docker-ce=18.04.0ce3-0~raspbian
kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
CNI using flannel
Once initilased do as njohnsn says above and change the version of kubeDNS to 1.14.10 (1.14.8 is broken)

kubectl edit deploy kube-dns --namespace=kube-system

to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors

my cluster is now happily running the armhf version of OpenFAAS and the arm version of prometheues-operator , and using the docker-incubator nfs-client-provisioner to handle PV and PVC,s served from an NFS Server which is not part of the cluster
Hope that help

Andy

BathAndy commented Jun 30, 2018

docker-ce=18.04.0ce3-0~raspbian
kubeadm=1.10.2-00 kubectl=1.10.2-00 kubelet=1.10.2-00
CNI using flannel
Once initilased do as njohnsn says above and change the version of kubeDNS to 1.14.10 (1.14.8 is broken)

kubectl edit deploy kube-dns --namespace=kube-system

to upgrade the image to 1.14.10 from 1.14.8 fixed it. If you dont get the framwork correct you get some weird errors

my cluster is now happily running the armhf version of OpenFAAS and the arm version of prometheues-operator , and using the docker-incubator nfs-client-provisioner to handle PV and PVC,s served from an NFS Server which is not part of the cluster
Hope that help

Andy

@gianlazz

This comment has been minimized.

Show comment
Hide comment
@gianlazz

gianlazz Jul 1, 2018

I am unable to get past the CGROUPS_MEMORY: missing issue after running sudo kubeadm init --token-ttl=0

gianlazz commented Jul 1, 2018

I am unable to get past the CGROUPS_MEMORY: missing issue after running sudo kubeadm init --token-ttl=0

@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Jul 2, 2018

@gianlazz: same thing for me, had to revert to cgroup_memory=memory on my rpi2, rpi3 and rpi3+ with latest raspbian.

deurk commented Jul 2, 2018

@gianlazz: same thing for me, had to revert to cgroup_memory=memory on my rpi2, rpi3 and rpi3+ with latest raspbian.

@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Jul 2, 2018

I'm still unable to run a cluster at all. I'm blocked at the init:

pi@rpi-k8s-master-1:~ $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.60 --pod-network-cidr=172.30.0.0/16 --kubernetes-version v1.10.2 --ignore-preflight-error
s=ALL
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.04.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [rpi-k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96
.0.1 192.168.0.60]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [rpi-k8s-master-1] and IPs [192.168.0.60]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - Either there is no internet connection, or imagePullPolicy is set to "Never",
          so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-arm:v1.10.2
                - k8s.gcr.io/kube-controller-manager-arm:v1.10.2
                - k8s.gcr.io/kube-scheduler-arm:v1.10.2
                - k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Image used:

RASPBIAN STRETCH LITE
Minimal image based on Debian Stretch
Version:June 2018
Release date:2018-06-27
Kernel version:4.14

Docker version:

Server:
 Engine:
  Version:      18.04.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   3d479c0
  Built:        Tue Apr 10 18:21:25 2018
  OS/Arch:      linux/arm
  Experimental: false

Kubernetes version: 1.10.2

deurk commented Jul 2, 2018

I'm still unable to run a cluster at all. I'm blocked at the init:

pi@rpi-k8s-master-1:~ $ sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.60 --pod-network-cidr=172.30.0.0/16 --kubernetes-version v1.10.2 --ignore-preflight-error
s=ALL
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.04.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [rpi-k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96
.0.1 192.168.0.60]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [rpi-k8s-master-1] and IPs [192.168.0.60]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - Either there is no internet connection, or imagePullPolicy is set to "Never",
          so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-arm:v1.10.2
                - k8s.gcr.io/kube-controller-manager-arm:v1.10.2
                - k8s.gcr.io/kube-scheduler-arm:v1.10.2
                - k8s.gcr.io/etcd-arm:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Image used:

RASPBIAN STRETCH LITE
Minimal image based on Debian Stretch
Version:June 2018
Release date:2018-06-27
Kernel version:4.14

Docker version:

Server:
 Engine:
  Version:      18.04.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   3d479c0
  Built:        Tue Apr 10 18:21:25 2018
  OS/Arch:      linux/arm
  Experimental: false

Kubernetes version: 1.10.2

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Jul 2, 2018

@Gallouche Apparently the kube-controller-manager CrashLoopBackOff will be fixed in 1.11.1 according to kubernetes/kubernetes#65674
I also note that after the recent kernal update on the Pi, weavenet now works for me. I've had openfaas (and this gist) working nicely since then. Once 1.11.1 comes out and tests ok I'm going to freeze versions in my ansible script.

micedwards commented Jul 2, 2018

@Gallouche Apparently the kube-controller-manager CrashLoopBackOff will be fixed in 1.11.1 according to kubernetes/kubernetes#65674
I also note that after the recent kernal update on the Pi, weavenet now works for me. I've had openfaas (and this gist) working nicely since then. Once 1.11.1 comes out and tests ok I'm going to freeze versions in my ansible script.

@Gallouche

This comment has been minimized.

Show comment
Hide comment
@Gallouche

Gallouche Jul 3, 2018

Thank you all for the answers, my bachelor degree work will maybe be done in time ! I will try, thanks a lot !

Gallouche commented Jul 3, 2018

Thank you all for the answers, my bachelor degree work will maybe be done in time ! I will try, thanks a lot !

@futurisma

This comment has been minimized.

Show comment
Hide comment
@futurisma

futurisma Jul 4, 2018

@deurk it looks like your kubelet service is not running, kubeadm has a dependancy on kubelet which does seem to be documented very well.
Try

sudo systemctl restart kubelet
sudo systemctl status kubelet

The problem seems to be that /etc/kubernetes/kubelet.conf is missing in the initial installation, after the first run of kubeadm init a copy of the kubelet.conf gets created.

futurisma commented Jul 4, 2018

@deurk it looks like your kubelet service is not running, kubeadm has a dependancy on kubelet which does seem to be documented very well.
Try

sudo systemctl restart kubelet
sudo systemctl status kubelet

The problem seems to be that /etc/kubernetes/kubelet.conf is missing in the initial installation, after the first run of kubeadm init a copy of the kubelet.conf gets created.

@chito4

This comment has been minimized.

Show comment
Hide comment
@chito4

chito4 Jul 6, 2018

alright so after 2 days of googling in multiple forums coming back to this tutorial many times and formating my varous microsd cards over and over again, i finally go it to work by replacing the step where it says "Add repo lists & install kubeadm" to:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

and where it said "Edit /boot/cmdline.txt"

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

hopefully it will help some tired soul... and yes i understand I'm running an old version of kubernetes, i don't care its working and i just wanna learn...

chito4 commented Jul 6, 2018

alright so after 2 days of googling in multiple forums coming back to this tutorial many times and formating my varous microsd cards over and over again, i finally go it to work by replacing the step where it says "Add repo lists & install kubeadm" to:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

and where it said "Edit /boot/cmdline.txt"

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

hopefully it will help some tired soul... and yes i understand I'm running an old version of kubernetes, i don't care its working and i just wanna learn...

@mrpaws

This comment has been minimized.

Show comment
Hide comment
@mrpaws

mrpaws Jul 6, 2018

While I'd like to get this working with latest I was successful following @kumardeepam to downgrade version as well as of July 6, 2018. The dns pods weren't starting, and using a specific version of weave brought them online as per @haebler advice.

Thanks to all contributors and @alexellis for putting this together.

Thus, the steps for success with a slightly older version circa July 2018 are as follows:

changes in pre-req

sudo vi /boot/cmdline.txt
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

#changes in install

Install v1.9.7-00, follow the below given commands

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

install weave 1.6

kubectl apply -f https://git.io/weave-kube-1.6

no change for init

$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

mrpaws commented Jul 6, 2018

While I'd like to get this working with latest I was successful following @kumardeepam to downgrade version as well as of July 6, 2018. The dns pods weren't starting, and using a specific version of weave brought them online as per @haebler advice.

Thanks to all contributors and @alexellis for putting this together.

Thus, the steps for success with a slightly older version circa July 2018 are as follows:

changes in pre-req

sudo vi /boot/cmdline.txt
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

#changes in install

Install v1.9.7-00, follow the below given commands

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00

install weave 1.6

kubectl apply -f https://git.io/weave-kube-1.6

no change for init

$sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL should work as expected

@gianlazz

This comment has been minimized.

Show comment
Hide comment
@gianlazz

gianlazz Jul 8, 2018

I went through the process and am still having this issue:

[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
Edit

I ran $sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL from the comment above however it looks like my IP address may not have been configured correctly so I've run $sudo kubeadm reset and and trying again to initialize Kubernetes with sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL

Edit

Running into another error now that if I understand correctly is due to the dns pods not having been started correctly however when I run kubectl apply -f https://git.io/weave-kube-1.6 it says The connection to the server localhost:8080 was refused - did you specify the right host or port?

So I'm still left with this below:

sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.6]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster```

gianlazz commented Jul 8, 2018

I went through the process and am still having this issue:

[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
Edit

I ran $sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.0.100 --ignore-preflight-errors=ALL from the comment above however it looks like my IP address may not have been configured correctly so I've run $sudo kubeadm reset and and trying again to initialize Kubernetes with sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL

Edit

Running into another error now that if I understand correctly is due to the dns pods not having been started correctly however when I run kubectl apply -f https://git.io/weave-kube-1.6 it says The connection to the server localhost:8080 was refused - did you specify the right host or port?

So I'm still left with this below:

sudo kubeadm init --token-ttl=0 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.6]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
        - There is no internet connection, so the kubelet cannot pull the following control plane images:
                - gcr.io/google_containers/kube-apiserver-arm:v1.9.9
                - gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
                - gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster```
@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Jul 9, 2018

Thanks @futurisma, this did not solve my problem so I reinstalled my cluster with kube* in 1.9.7 per @mrpaws instructions and that worked. Now I'm waiting for 1.11.1 to try again :)

deurk commented Jul 9, 2018

Thanks @futurisma, this did not solve my problem so I reinstalled my cluster with kube* in 1.9.7 per @mrpaws instructions and that worked. Now I'm waiting for 1.11.1 to try again :)

@mgazza

This comment has been minimized.

Show comment
Hide comment
@mgazza

mgazza Jul 10, 2018

I'm still having the same issue after trying every single one of the above suggestions :(

mgazza commented Jul 10, 2018

I'm still having the same issue after trying every single one of the above suggestions :(

@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Jul 12, 2018

@mgazza Which issue is that?

deurk commented Jul 12, 2018

@mgazza Which issue is that?

@samelie

This comment has been minimized.

Show comment
Hide comment
@samelie

samelie Jul 13, 2018

@kumardeepam Solution works, Thanks! I had the healthz problem. I had to use 1.9.7-00
on master
sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.X.XXX --ignore-preflight-errors=ALL --kubernetes-version v1.9.7

Had luck with flannel. Weave was giving me the loopBackError

This post from @aaronkjones was important: sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

curl -sSL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm64/g" | kubectl apply -f -

This post also had something to say :kubectl annotate node pine1 flannel.alpha.coreos.com/public-ip=192.168.X.XXX --overwrite=true (not sure if did anything)

ps: in between attempts I cleared things: kubeadm reset && sudo rm -rf ~/.kube && sudo systemctl daemon-reload && sudo systemctl restart docker && sudo systemctl restart kubelet sudo reboot

Everything seemed good now, sudo netstat -tuplen | grep 6443 -> LISTEN. But the join command on the nodes was giving me this: Failed to request cluster info, will try again: ... dial tcp 192.168.1.163:6443: i/o timeout]. Solution was to disable firewall on master, i know it's ugly: sudo ufw disable

Now I can ping master from nodes: nc -vz 192.168.1.163 6443 & the join command works.

samelie commented Jul 13, 2018

@kumardeepam Solution works, Thanks! I had the healthz problem. I had to use 1.9.7-00
on master
sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.X.XXX --ignore-preflight-errors=ALL --kubernetes-version v1.9.7

Had luck with flannel. Weave was giving me the loopBackError

This post from @aaronkjones was important: sudo sed -i '/KUBELET_NETWORK_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

curl -sSL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm64/g" | kubectl apply -f -

This post also had something to say :kubectl annotate node pine1 flannel.alpha.coreos.com/public-ip=192.168.X.XXX --overwrite=true (not sure if did anything)

ps: in between attempts I cleared things: kubeadm reset && sudo rm -rf ~/.kube && sudo systemctl daemon-reload && sudo systemctl restart docker && sudo systemctl restart kubelet sudo reboot

Everything seemed good now, sudo netstat -tuplen | grep 6443 -> LISTEN. But the join command on the nodes was giving me this: Failed to request cluster info, will try again: ... dial tcp 192.168.1.163:6443: i/o timeout]. Solution was to disable firewall on master, i know it's ugly: sudo ufw disable

Now I can ping master from nodes: nc -vz 192.168.1.163 6443 & the join command works.

@mgazza

This comment has been minimized.

Show comment
Hide comment
@mgazza

mgazza Jul 16, 2018

@deurk
after following @alexellis gist and having no joy and after swapping part of the original for @mrpaws and pretty much every other variation

history
    1  curl -sSL get.docker.com | sh && sudo usermod pi -aG docker
    2  sudo dphys-swapfile swapoff &&   sudo dphys-swapfile uninstall &&   sudo update-rc.d dphys-swapfile remove
    3  sudo nano /boot/cmdline.txt
    4  sudo reboot
    5  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00
    6  kubectl apply -f https://git.io/weave-kube-1.6
    7  sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.12 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
	[WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.12]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
	- There is no internet connection, so the kubelet cannot pull the following control plane images:
		- gcr.io/google_containers/kube-apiserver-arm:v1.9.9
		- gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
		- gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
uname -a
Linux raspberrypi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux

@aaronkjones I had no joy with your repo either, using raspbian or Hypriot

mgazza commented Jul 16, 2018

@deurk
after following @alexellis gist and having no joy and after swapping part of the original for @mrpaws and pretty much every other variation

history
    1  curl -sSL get.docker.com | sh && sudo usermod pi -aG docker
    2  sudo dphys-swapfile swapoff &&   sudo dphys-swapfile uninstall &&   sudo update-rc.d dphys-swapfile remove
    3  sudo nano /boot/cmdline.txt
    4  sudo reboot
    5  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00
    6  kubectl apply -f https://git.io/weave-kube-1.6
    7  sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.12 --ignore-preflight-errors=ALL
[init] Using Kubernetes version: v1.9.9
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03
	[WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [raspberrypi kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.12]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
	- There is no internet connection, so the kubelet cannot pull the following control plane images:
		- gcr.io/google_containers/kube-apiserver-arm:v1.9.9
		- gcr.io/google_containers/kube-controller-manager-arm:v1.9.9
		- gcr.io/google_containers/kube-scheduler-arm:v1.9.9

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
uname -a
Linux raspberrypi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux

@aaronkjones I had no joy with your repo either, using raspbian or Hypriot

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Jul 19, 2018

Kubernetes 1.11.1 has been released! Following this gist (with the /boot/cmdline.txt change) I've got openfaas & faas-cli working! AND that's with weavenet rather than flannel. [Not sure where I got the heapster.yaml file to get the dashboard working as I lost my notes for that].

`pi@shepherd:~/faas-functions $ uname -a
Linux shepherd 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

pi@shepherd:~/faas-functions $ docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:19:46 2018
OS/Arch: linux/arm
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:15:34 2018
OS/Arch: linux/arm
Experimental: false

pi@shepherd:~/faas-functions $ sudo apt list kube*
Listing... Done
kubeadm/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubectl/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubelet/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubernetes-cni/kubernetes-xenial,now 0.6.0-00 armhf [installed]`

Off to buy more SD cards to do a backup.

micedwards commented Jul 19, 2018

Kubernetes 1.11.1 has been released! Following this gist (with the /boot/cmdline.txt change) I've got openfaas & faas-cli working! AND that's with weavenet rather than flannel. [Not sure where I got the heapster.yaml file to get the dashboard working as I lost my notes for that].

`pi@shepherd:~/faas-functions $ uname -a
Linux shepherd 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

pi@shepherd:~/faas-functions $ docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:19:46 2018
OS/Arch: linux/arm
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:15:34 2018
OS/Arch: linux/arm
Experimental: false

pi@shepherd:~/faas-functions $ sudo apt list kube*
Listing... Done
kubeadm/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubectl/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubelet/kubernetes-xenial,now 1.11.1-00 armhf [installed]
kubernetes-cni/kubernetes-xenial,now 0.6.0-00 armhf [installed]`

Off to buy more SD cards to do a backup.

@micedwards

This comment has been minimized.

Show comment
Hide comment
@micedwards

micedwards Jul 19, 2018

Thanks Alex!

micedwards commented Jul 19, 2018

Thanks Alex!

@scottnuma

This comment has been minimized.

Show comment
Hide comment
@scottnuma

scottnuma Jul 19, 2018

After a lot of trouble with Hypriot's tutorial and this gist as well, I was able to successfully deploy and run the markdownrender on two pis with @aaronkjones's guide. Much thanks to @alexellis and @aaronkjones for the development of these guides as well as the discussion in the comments.

scottnuma commented Jul 19, 2018

After a lot of trouble with Hypriot's tutorial and this gist as well, I was able to successfully deploy and run the markdownrender on two pis with @aaronkjones's guide. Much thanks to @alexellis and @aaronkjones for the development of these guides as well as the discussion in the comments.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jul 24, 2018

hi All, I think I'm OK - can anyone tell me if I need to run Weave on the workers? If so, I'm getting a port 8080 error. Many TIAs ;-)

NAME            STATUS    ROLES     AGE       VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
bramble-1-pi3   Ready     master    4h        v1.11.1   172.17.12.134   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-2-pi3   Ready     <none>    48m       v1.11.1   172.17.12.135   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-3-pi3   Ready     <none>    48m       v1.11.1   172.17.12.127   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
pi@bramble-1-pi3:~ $ kubectl get pods -n kube-system
NAME                                        READY     STATUS             RESTARTS   AGE
coredns-78fcdf6894-nkjxz                    1/1       Running            0          4h
coredns-78fcdf6894-q8489                    1/1       Running            0          4h
etcd-bramble-1-pi3                          1/1       Running            0          4h
kube-apiserver-bramble-1-pi3                1/1       Running            0          4h
kube-controller-manager-bramble-1-pi3       1/1       Running            0          4h
kube-proxy-fr9mg                            1/1       Running            0          48m
kube-proxy-qnb9q                            1/1       Running            0          48m
kube-proxy-vm7gg                            1/1       Running            0          4h
kube-scheduler-bramble-1-pi3                1/1       Running            0          4h
kubernetes-dashboard-6948bdb78-ndfzk        0/1       CrashLoopBackOff   3          2m
kubernetes-dashboard-head-6b79997c9-mc5q5   1/1       Running            0          2m
weave-net-dmzl5                             2/2       Running            0          48m
weave-net-n7gf8                             2/2       Running            0          48m
weave-net-pjzhn                             2/2       Running            0          3h

pi@bramble-3-pi3:~ $ kubectl apply -f \
>  "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
error: unable to recognize "https://cloud.weave.works/k8s/net?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxMSIsIEdpdFZlcnNpb246InYxLjExLjEiLCBHaXRDb21taXQ6ImIxYjI5OTc4MjcwZGMyMmZlY2M1OTJhYzU1ZDkwMzM1MDQ1NDMxMGEiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE4LTA3LTE3VDE4OjUzOjIwWiIsIEdvVmVyc2lvbjoiZ28xLjEwLjMiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYXJtIn0K": Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused```

OK so it looks like Weave is running across the cluster so presumably only requires installing on the master:

weave-net-dmzl5                             2/2       Running   0          3d        172.17.12.135   bramble-2-pi3
weave-net-n7gf8                             2/2       Running   0          3d        172.17.12.127   bramble-3-pi3
weave-net-pjzhn                             2/2       Running   0          4d        172.17.12.134   bramble-1-pi3

ghost commented Jul 24, 2018

hi All, I think I'm OK - can anyone tell me if I need to run Weave on the workers? If so, I'm getting a port 8080 error. Many TIAs ;-)

NAME            STATUS    ROLES     AGE       VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
bramble-1-pi3   Ready     master    4h        v1.11.1   172.17.12.134   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-2-pi3   Ready     <none>    48m       v1.11.1   172.17.12.135   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
bramble-3-pi3   Ready     <none>    48m       v1.11.1   172.17.12.127   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
pi@bramble-1-pi3:~ $ kubectl get pods -n kube-system
NAME                                        READY     STATUS             RESTARTS   AGE
coredns-78fcdf6894-nkjxz                    1/1       Running            0          4h
coredns-78fcdf6894-q8489                    1/1       Running            0          4h
etcd-bramble-1-pi3                          1/1       Running            0          4h
kube-apiserver-bramble-1-pi3                1/1       Running            0          4h
kube-controller-manager-bramble-1-pi3       1/1       Running            0          4h
kube-proxy-fr9mg                            1/1       Running            0          48m
kube-proxy-qnb9q                            1/1       Running            0          48m
kube-proxy-vm7gg                            1/1       Running            0          4h
kube-scheduler-bramble-1-pi3                1/1       Running            0          4h
kubernetes-dashboard-6948bdb78-ndfzk        0/1       CrashLoopBackOff   3          2m
kubernetes-dashboard-head-6b79997c9-mc5q5   1/1       Running            0          2m
weave-net-dmzl5                             2/2       Running            0          48m
weave-net-n7gf8                             2/2       Running            0          48m
weave-net-pjzhn                             2/2       Running            0          3h

pi@bramble-3-pi3:~ $ kubectl apply -f \
>  "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
error: unable to recognize "https://cloud.weave.works/k8s/net?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxMSIsIEdpdFZlcnNpb246InYxLjExLjEiLCBHaXRDb21taXQ6ImIxYjI5OTc4MjcwZGMyMmZlY2M1OTJhYzU1ZDkwMzM1MDQ1NDMxMGEiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE4LTA3LTE3VDE4OjUzOjIwWiIsIEdvVmVyc2lvbjoiZ28xLjEwLjMiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYXJtIn0K": Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused```

OK so it looks like Weave is running across the cluster so presumably only requires installing on the master:

weave-net-dmzl5                             2/2       Running   0          3d        172.17.12.135   bramble-2-pi3
weave-net-n7gf8                             2/2       Running   0          3d        172.17.12.127   bramble-3-pi3
weave-net-pjzhn                             2/2       Running   0          4d        172.17.12.134   bramble-1-pi3
@mikegleasonjr

This comment has been minimized.

Show comment
Hide comment
@mikegleasonjr

mikegleasonjr Jul 28, 2018

@micedwards how did you get the latest 1.11.1 to even have a successful install?

mikegleasonjr commented Jul 28, 2018

@micedwards how did you get the latest 1.11.1 to even have a successful install?

@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Aug 14, 2018

OK, 1.11.2 is out. I will give it a spin tonight or tomorrow to see how it fares with this Gist.

deurk commented Aug 14, 2018

OK, 1.11.2 is out. I will give it a spin tonight or tomorrow to see how it fares with this Gist.

@deurk

This comment has been minimized.

Show comment
Hide comment
@deurk

deurk Aug 17, 2018

Took me a bit longer than expected to get to test it but I managed to make it work 👍

root@rpi-k8s-master-1:~# kubectl get nodes -o wide
NAME               STATUS    ROLES     AGE       VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
rpi-k8s-master-1   Ready     master    11h       v1.11.2   192.168.0.60   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.52-v7+      docker://18.6.0
rpi-k8s-slave-1    Ready     <none>    3h        v1.11.2   192.168.0.61   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
root@rpi-k8s-master-1:~# kubectl get pods --namespace=kube-system -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE               NOMINATED NODE
coredns-78fcdf6894-mrbcb                   1/1       Running   0          11h       172.19.0.2     rpi-k8s-master-1   <none>
coredns-78fcdf6894-vmzmw                   1/1       Running   0          11h       172.19.0.4     rpi-k8s-master-1   <none>
etcd-rpi-k8s-master-1                      1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-apiserver-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-controller-manager-rpi-k8s-master-1   1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-flannel-ds-9cllm                      1/1       Running   0          11m       192.168.0.61   rpi-k8s-slave-1    <none>
kube-flannel-ds-r8pf6                      1/1       Running   0          11m       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-4hjcn                           1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-n4pmb                           1/1       Running   0          3h        192.168.0.61   rpi-k8s-slave-1    <none>
kube-scheduler-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
root@rpi-k8s-master-1:~# docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:19:46 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:15:34 2018
  OS/Arch:          linux/arm
  Experimental:     false
root@rpi-k8s-master-1:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
root@rpi-k8s-master-1:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:14:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
Flannel 0.9.1

deurk commented Aug 17, 2018

Took me a bit longer than expected to get to test it but I managed to make it work 👍

root@rpi-k8s-master-1:~# kubectl get nodes -o wide
NAME               STATUS    ROLES     AGE       VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
rpi-k8s-master-1   Ready     master    11h       v1.11.2   192.168.0.60   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.52-v7+      docker://18.6.0
rpi-k8s-slave-1    Ready     <none>    3h        v1.11.2   192.168.0.61   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.50-v7+      docker://18.6.0
root@rpi-k8s-master-1:~# kubectl get pods --namespace=kube-system -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE               NOMINATED NODE
coredns-78fcdf6894-mrbcb                   1/1       Running   0          11h       172.19.0.2     rpi-k8s-master-1   <none>
coredns-78fcdf6894-vmzmw                   1/1       Running   0          11h       172.19.0.4     rpi-k8s-master-1   <none>
etcd-rpi-k8s-master-1                      1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-apiserver-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-controller-manager-rpi-k8s-master-1   1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-flannel-ds-9cllm                      1/1       Running   0          11m       192.168.0.61   rpi-k8s-slave-1    <none>
kube-flannel-ds-r8pf6                      1/1       Running   0          11m       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-4hjcn                           1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
kube-proxy-n4pmb                           1/1       Running   0          3h        192.168.0.61   rpi-k8s-slave-1    <none>
kube-scheduler-rpi-k8s-master-1            1/1       Running   0          11h       192.168.0.60   rpi-k8s-master-1   <none>
root@rpi-k8s-master-1:~# docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:19:46 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:15:34 2018
  OS/Arch:          linux/arm
  Experimental:     false
root@rpi-k8s-master-1:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
root@rpi-k8s-master-1:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:14:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}
Flannel 0.9.1
@alledingenmaken

This comment has been minimized.

Show comment
Hide comment
@alledingenmaken

alledingenmaken Aug 20, 2018

Thank you very much for the write-up, very straight-forward and working like a charm. Just make sure to update the Raspbian images to the latest kernel version and to follow the instructions. On stretch light you don't have to create the dhcpcd.conf file yourself for static IP's, just adjust the existing file. And also sometimes it helps to wait for a bit, getting all the kube-system containers in ready and running state can take a while..now getting ready for OpenFaaS and maybe running some in-house Ghost-blogs.

alledingenmaken commented Aug 20, 2018

Thank you very much for the write-up, very straight-forward and working like a charm. Just make sure to update the Raspbian images to the latest kernel version and to follow the instructions. On stretch light you don't have to create the dhcpcd.conf file yourself for static IP's, just adjust the existing file. And also sometimes it helps to wait for a bit, getting all the kube-system containers in ready and running state can take a while..now getting ready for OpenFaaS and maybe running some in-house Ghost-blogs.

@iceteahh

This comment has been minimized.

Show comment
Hide comment
@iceteahh

iceteahh Aug 23, 2018

@deurk How could you get it work? I init k8s 1.11.2 but I get this error: "failed to pull image [k8s.gcr.io/kube-proxy-arm:v1.11.2]: exit status 1". Can you help me fix it?
I used pi 3 model B

iceteahh commented Aug 23, 2018

@deurk How could you get it work? I init k8s 1.11.2 but I get this error: "failed to pull image [k8s.gcr.io/kube-proxy-arm:v1.11.2]: exit status 1". Can you help me fix it?
I used pi 3 model B

@mrlesmithjr

This comment has been minimized.

Show comment
Hide comment
@mrlesmithjr

mrlesmithjr Aug 24, 2018

Just wanted to share here as I had been watching this thread off and on for a while as @aaronkjones directed me here. The latest deployment using Ansible for all of this now works again. I am using Weave BTW.

https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster

mrlesmithjr commented Aug 24, 2018

Just wanted to share here as I had been watching this thread off and on for a while as @aaronkjones directed me here. The latest deployment using Ansible for all of this now works again. I am using Weave BTW.

https://github.com/mrlesmithjr/ansible-rpi-k8s-cluster

@Dgaylard

This comment has been minimized.

Show comment
Hide comment
@Dgaylard

Dgaylard Aug 24, 2018

Has anyone been able to get their Rpi3 k8s cluster integrated with Gitlab-CE? I'm trying to integrate my cluster right now and it's failing to install Helm-Tiller through Gitlab CE - just failing to connect in general really.

Dgaylard commented Aug 24, 2018

Has anyone been able to get their Rpi3 k8s cluster integrated with Gitlab-CE? I'm trying to integrate my cluster right now and it's failing to install Helm-Tiller through Gitlab CE - just failing to connect in general really.

@arussellsaw

This comment has been minimized.

Show comment
Hide comment
@arussellsaw

arussellsaw Aug 28, 2018

figured i'd contribute some deviations from the instructions that helped me, i managed to get this running (28/08/2018) by specifying version 1.8.3 on kubelet, kubectl, kubeadm install, and using the flannel network.

arussellsaw commented Aug 28, 2018

figured i'd contribute some deviations from the instructions that helped me, i managed to get this running (28/08/2018) by specifying version 1.8.3 on kubelet, kubectl, kubeadm install, and using the flannel network.

@Insulince

This comment has been minimized.

Show comment
Hide comment
@Insulince

Insulince Aug 29, 2018

Thank you for your example install commands for v1.9.7 @chito4! I was that tired sould you referenced that needed that info to get this working.

Got it running on Raspberry Pi 3B+, 16GB Micro SD card, Raspbian Stretch Lite OS, Kubernetes v1.9.7 (kubeadm, kubectl), Weave CNI, Docker 18.06.1-ce

I haven't attempted any nodes except the master node so far, but will update if I run into problems.

Insulince commented Aug 29, 2018

Thank you for your example install commands for v1.9.7 @chito4! I was that tired sould you referenced that needed that info to get this working.

Got it running on Raspberry Pi 3B+, 16GB Micro SD card, Raspbian Stretch Lite OS, Kubernetes v1.9.7 (kubeadm, kubectl), Weave CNI, Docker 18.06.1-ce

I haven't attempted any nodes except the master node so far, but will update if I run into problems.

@joliver

This comment has been minimized.

Show comment
Hide comment
@joliver

joliver Sep 7, 2018

I was able to get a 7-node Raspberry Pi cluster running using:

  • Raspbery Pi 3 B+
  • Raspbian Stretch Lite (2018-06-27)
  • Docker v18.06.1-ce (the latest apt-get installable as of 2018-09-07)
  • Kubernetes v1.11.2
  • Weave networking

Here is my exact system state:

$ cat /proc/device-tree/model
Raspberry Pi 3 Model B Plus Rev 1.3
$ uname -a
Linux pi-master 4.14.62-v7+ #1134 SMP Tue Aug 14 17:10:10 BST 2018 armv7l GNU/Linux
$ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=830d7945-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory

Note: Here's a really easy way to append the cgroup stuff to the /boot/cmdline.txt file:
sudo sed -i 's/ rootwait$/ rootwait cgroup_enable=cpuset cgroup_enable=memory/g' /boot/cmdline.txt

When I initialized the cluster, I used the following command:

$ sudo kubeadm init --token-ttl=0 --kubernetes-version v1.11.2 --apiserver-advertise-address=192.168.1.48
$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:30:52 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:26:37 2018
  OS/Arch:          linux/arm
  Experimental:     false
$ kubeadm version # (formatted for readability)
kubeadm version: &version.Info{
  Major:"1",
  Minor:"11",
  GitVersion:"v1.11.2",
  GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239",
  GitTreeState:"clean",
  BuildDate:"2018-08-07T23:14:39Z",
  GoVersion:"go1.10.3",
  Compiler:"gc",
  Platform:"linux/arm",
}
$ kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
pi-master    Ready     master    13m       v1.11.2
pi-node-01   Ready     <none>    5m        v1.11.2
pi-node-02   Ready     <none>    5m        v1.11.2
pi-node-03   Ready     <none>    5m        v1.11.2
pi-node-04   Ready     <none>    5m        v1.11.2
pi-node-05   Ready     <none>    5m        v1.11.2
pi-node-06   Ready     <none>    5m        v1.11.2
$ kubectl get pods --namespace=kube-system
NAME                                READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-zl96s            1/1       Running   0          12m
coredns-78fcdf6894-zxc95            1/1       Running   0          12m
etcd-pi-master                      1/1       Running   0          11m
kube-apiserver-pi-master            1/1       Running   1          11m
kube-controller-manager-pi-master   1/1       Running   0          11m
kube-proxy-2zblw                    1/1       Running   0          6m
kube-proxy-5wb5l                    1/1       Running   0          6m
kube-proxy-6cngc                    1/1       Running   0          6m
kube-proxy-6sk8t                    1/1       Running   0          6m
kube-proxy-dczbt                    1/1       Running   0          6m
kube-proxy-rtvtm                    1/1       Running   0          12m
kube-proxy-zvwph                    1/1       Running   0          6m
kube-scheduler-pi-master            1/1       Running   0          11m
weave-net-49hfp                     2/2       Running   1          6m
weave-net-cz7pm                     2/2       Running   0          6m
weave-net-j75wt                     2/2       Running   0          6m
weave-net-jb7vp                     2/2       Running   0          6m
weave-net-kmzd8                     2/2       Running   0          6m
weave-net-kzspl                     2/2       Running   0          10m
weave-net-wr692                     2/2       Running   1          6m
$ dpkg -l | egrep "kube|docker"
ii  docker-ce                       18.06.1~ce~3-0~raspbian      armhf        Docker: the open-source application container engine
ii  kubeadm                         1.11.2-00                    armhf        Kubernetes Cluster Bootstrapping Tool
ii  kubectl                         1.11.2-00                    armhf        Kubernetes Command Line Tool
ii  kubelet                         1.11.2-00                    armhf        Kubernetes Node Agent
ii  kubernetes-cni                  0.6.0-00                     armhf        Kubernetes CNI

joliver commented Sep 7, 2018

I was able to get a 7-node Raspberry Pi cluster running using:

  • Raspbery Pi 3 B+
  • Raspbian Stretch Lite (2018-06-27)
  • Docker v18.06.1-ce (the latest apt-get installable as of 2018-09-07)
  • Kubernetes v1.11.2
  • Weave networking

Here is my exact system state:

$ cat /proc/device-tree/model
Raspberry Pi 3 Model B Plus Rev 1.3
$ uname -a
Linux pi-master 4.14.62-v7+ #1134 SMP Tue Aug 14 17:10:10 BST 2018 armv7l GNU/Linux
$ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=830d7945-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory

Note: Here's a really easy way to append the cgroup stuff to the /boot/cmdline.txt file:
sudo sed -i 's/ rootwait$/ rootwait cgroup_enable=cpuset cgroup_enable=memory/g' /boot/cmdline.txt

When I initialized the cluster, I used the following command:

$ sudo kubeadm init --token-ttl=0 --kubernetes-version v1.11.2 --apiserver-advertise-address=192.168.1.48
$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:30:52 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:26:37 2018
  OS/Arch:          linux/arm
  Experimental:     false
$ kubeadm version # (formatted for readability)
kubeadm version: &version.Info{
  Major:"1",
  Minor:"11",
  GitVersion:"v1.11.2",
  GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239",
  GitTreeState:"clean",
  BuildDate:"2018-08-07T23:14:39Z",
  GoVersion:"go1.10.3",
  Compiler:"gc",
  Platform:"linux/arm",
}
$ kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
pi-master    Ready     master    13m       v1.11.2
pi-node-01   Ready     <none>    5m        v1.11.2
pi-node-02   Ready     <none>    5m        v1.11.2
pi-node-03   Ready     <none>    5m        v1.11.2
pi-node-04   Ready     <none>    5m        v1.11.2
pi-node-05   Ready     <none>    5m        v1.11.2
pi-node-06   Ready     <none>    5m        v1.11.2
$ kubectl get pods --namespace=kube-system
NAME                                READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-zl96s            1/1       Running   0          12m
coredns-78fcdf6894-zxc95            1/1       Running   0          12m
etcd-pi-master                      1/1       Running   0          11m
kube-apiserver-pi-master            1/1       Running   1          11m
kube-controller-manager-pi-master   1/1       Running   0          11m
kube-proxy-2zblw                    1/1       Running   0          6m
kube-proxy-5wb5l                    1/1       Running   0          6m
kube-proxy-6cngc                    1/1       Running   0          6m
kube-proxy-6sk8t                    1/1       Running   0          6m
kube-proxy-dczbt                    1/1       Running   0          6m
kube-proxy-rtvtm                    1/1       Running   0          12m
kube-proxy-zvwph                    1/1       Running   0          6m
kube-scheduler-pi-master            1/1       Running   0          11m
weave-net-49hfp                     2/2       Running   1          6m
weave-net-cz7pm                     2/2       Running   0          6m
weave-net-j75wt                     2/2       Running   0          6m
weave-net-jb7vp                     2/2       Running   0          6m
weave-net-kmzd8                     2/2       Running   0          6m
weave-net-kzspl                     2/2       Running   0          10m
weave-net-wr692                     2/2       Running   1          6m
$ dpkg -l | egrep "kube|docker"
ii  docker-ce                       18.06.1~ce~3-0~raspbian      armhf        Docker: the open-source application container engine
ii  kubeadm                         1.11.2-00                    armhf        Kubernetes Cluster Bootstrapping Tool
ii  kubectl                         1.11.2-00                    armhf        Kubernetes Command Line Tool
ii  kubelet                         1.11.2-00                    armhf        Kubernetes Node Agent
ii  kubernetes-cni                  0.6.0-00                     armhf        Kubernetes CNI
@mgazza

This comment has been minimized.

Show comment
Hide comment
@mgazza

mgazza Sep 21, 2018

Still not joy using the scripts.

cat /proc/device-tree/model
Raspberry Pi 2 Model B Rev 1.1
uname -a
Linux node-1 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
>   echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
>   sudo apt-get update -q && \
>   sudo apt-get install -qy kubeadm
OK
deb http://apt.kubernetes.io/ kubernetes-xenial main
Hit:1 http://archive.raspberrypi.org/debian stretch InRelease
Get:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]
Hit:4 https://download.docker.com/linux/raspbian stretch InRelease
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf Packages [18.3 kB]
Fetched 42.3 kB in 3s (13.3 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
kubeadm is already the newest version (1.12.0-rc.1-00).
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.100 --ignore-preflight-errors=ALL
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.12.0-rc.1" Control plane version: "1.11.3"
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [node-1 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [node-1 localhost] and IPs [192.168.1.100 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

mgazza commented Sep 21, 2018

Still not joy using the scripts.

cat /proc/device-tree/model
Raspberry Pi 2 Model B Rev 1.1
uname -a
Linux node-1 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
>   echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
>   sudo apt-get update -q && \
>   sudo apt-get install -qy kubeadm
OK
deb http://apt.kubernetes.io/ kubernetes-xenial main
Hit:1 http://archive.raspberrypi.org/debian stretch InRelease
Get:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease [15.0 kB]
Hit:4 https://download.docker.com/linux/raspbian stretch InRelease
Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf Packages [18.3 kB]
Fetched 42.3 kB in 3s (13.3 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
kubeadm is already the newest version (1.12.0-rc.1-00).
sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.1.100 --ignore-preflight-errors=ALL
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.12.0-rc.1" Control plane version: "1.11.3"
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [node-1 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [node-1 localhost] and IPs [192.168.1.100 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster
@LeeXun

This comment has been minimized.

Show comment
Hide comment

LeeXun commented Sep 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment