Skip to content

Instantly share code, notes, and snippets.

@StephenSorriaux
Created October 25, 2018 18:47
Show Gist options
  • Star 33 You must be signed in to star a gist
  • Fork 16 You must be signed in to fork a gist
  • Save StephenSorriaux/fa07afa57c931c84d1886b08c704acfe to your computer and use it in GitHub Desktop.
Save StephenSorriaux/fa07afa57c931c84d1886b08c704acfe to your computer and use it in GitHub Desktop.
Install Kubernetes on bare-metal ArchLinux host

Installing Kubernetes on ArchLinux

Packages

pacman -S curl docker ebtables ethtool wget unzip

Also cfssl is needed but available on AUR, using pacaur

pacaur -S cfssl

Configuring Docker

Add --iptables=false and --ip-masq=false parameters to the dockerd daemon in the docker systemd service (/usr/lib/systemd/system/docker.service).

Allow bridged IPV4 traffic to iptables' chains using:

sysctl net.bridge.bridge-nf-call-iptables=1

If Docker was previsously used, clean the iptables rules using:

iptables -F
iptables -t nat -F

Start or restart Docker. systemctl enable docker && systemctl restart docker

Installing CNI

export CNI_VERSION="v0.6.0"
mkdir -p /opt/cni/bin
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz

Installing CRI

export CRICTL_VERSION="v1.11.1"
mkdir -p /opt/bin
curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C /opt/bin -xz

Installing kubeadm, kubelet, kubectl

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"

mkdir -p /opt/bin
cd /opt/bin
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
chmod +x {kubeadm,kubelet,kubectl}

curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl enable kubelet && systemctl start kubelet

Creating the cluster

kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

At this point you should be able to use

$ kubectl get no
NAME                 STATUS   ROLES    AGE   VERSION
stephen-arch-linux   Ready    master   31m   v1.12.1

Adding Flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

Once Flannel is up, your cluster is up and running.

Making the master schedulable

kubectl taint nodes --all node-role.kubernetes.io/master-

Accessing your k8s services from outside the cluster

Install an Ingress controller, for instance the NGINX Ingress Controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

Dont forget the k8s service:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml

Using the Ingress object you will be able to access your services.

Using local volume as PersistentVolumeClaim

Create a storageClass (this object is not namespaced):

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Make it the default one:

kubectl annotate storageclass local-storage storageclass.kubernetes.io/is-default-class=true

For each PersistentVolumeClaim, you will need to manually create a PersistentVolume:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /etc/kubernetes/local
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: beta.kubernetes.io/os
          operator: In
          values:
          - linux

Be sure that the spec.local.path exists on the host.

@thapakazi
Copy link

Meanwhile on this universe, following via your gist I stumbled upon these:

Jun 06 08:51:48 xps kubelet[3426]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified
by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 06 08:51:48 xps kubelet[3426]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified
by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 06 08:51:48 xps kubelet[3426]: F0606 08:51:48.956071    3426 server.go:248] unable to load client CA file /etc/kubernetes/pki/ca.crt
: open /etc/kubernetes/pki/ca.crt: no such file or director

I am googling out solution to this.

@thapakazi
Copy link

LOoks like I need to manually run kubeadm init first.

kubeadm init --pod-network-cidr=10.244.0.0/16 --dry-run >> init.log

Meanwhile, I stumbled upon yet another problem.

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

as I was missing the kernel module: found it documented here

modprobe br_netfilter

@StephenSorriaux
Copy link
Author

Hi,

Yes you need to kubeadm init to generate certs & everything else, it is in the Creating the cluster part.

I was not aware that some distrib would not have the br_netfilter module running by default. Which one are you using and what version?

@thapakazi
Copy link

I am also arch user :)

@Choongkyu
Copy link

Choongkyu commented Feb 22, 2020

hello. thanks for documenting this. I had tried to follow this after getting a number of errors while following the guide here https://wiki.archlinux.org/index.php/Kubernetes but am still struggling with seeing my one node as showing as NotReady, even after applying flannel.yml with kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml. One of the discrepancies between the guide and what I needed to do is to run kubectl as root. I'm not sure if requirements changed since your having written this. Any help would be much appreciated.

@vi2co
Copy link

vi2co commented Feb 24, 2020

Hi @zebralight, I had the the issue with Kube cluster "NotReady" status. In my case it was wrong CNI installation folder (/usr/bin instead of /opt/cni/bin). But you can troubleshoot this issue using journalctl -u kubelet -r command ( -r means latest logs are on top of the screen).

One of the discrepancies between the guide and what I needed to do is to run kubectl as root.

Just run:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config`

The full list of commands which for Manjaro Awesome 18.1.3 (5.3.11-1-MANJARO kernel ):

############### SUDO SHELL ###############################
pacman -S ebtables ethtool socat
modprobe br_netfilter
echo "br_netfilter" > /etc/modules-load.d/br-netfilter.conf
sysctl net.bridge.bridge-nf-call-iptables=1
#### Replace Docker service ExecStart /usr/lib/systemd/system/docker.service with the following:
ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd --iptables=false --ip-masq=false -H fd://
systemctl daemon-reload

export CNI_VERSION="v0.8.5"
mkdir -p /opt/cni/bin
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz

export CRICTL_VERSION="v1.17.0"
mkdir -p /opt/cri
curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C /opt/cri -xz

cd /usr/bin
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
chmod +x {kubeadm,kubelet,kubectl}

curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" > /etc/systemd/system/kubelet.service
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl enable kubelet

sudo kubeadm init --pod-network-cidr=10.244.0.0/24 --apiserver-advertise-address=0.0.0.0

echo 'KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd  --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1"' > /var/lib/kubelet/kubeadm-flags.env
################# Normal user shell ###################################
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml`

@krisnova
Copy link

krisnova commented May 1, 2020

Sup nerds.

OFC the download directory changed in 1.18 of Kubernetes.

The new tree is for debian packages is here and we can steal the kubelet.service unit file and 10-kubeadm.conf as follows:

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"

mkdir -p /opt/bin
cd /opt/bin
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
chmod +x {kubeadm,kubelet,kubectl}

curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl enable kubelet && systemctl start kubelet

The commit with the changes: kubernetes/website@5cf0d19

Also I think the Arch docs suggests /usr/lib/systemd/system for installed packages - which is where I am putting mine. But whatevs. https://wiki.archlinux.org/index.php/systemd#Writing_unit_files

@StefanLobbenmeier
Copy link

StefanLobbenmeier commented Jun 7, 2020

@kris-nova small type in the snippet - ${RELEASE_VERSION} should be ${RELEASE} right?

Also
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" still gives me 404 with the current Release of v1.18.3: https://raw.githubusercontent.com/kubernetes/release/v1.18.3/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service

Found it: https://github.com/kubernetes/website/blob/master/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md

RELEASE_VERSION="v0.2.7"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment