Initial notes on install self-managed k8s

k8s_install.md

Enable Modules

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

lsmod | egrep 'overlay|br_netfilter'

containerd config

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.tom

Disable Swap

Note: might also need to remove from /etc/fstab

sudo swapoff -a

Install Kuberenetes

Note: run all all nodes

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

kver="1.24.2-00" ## or whatever (use the full semver with -)
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update

for k in kubelet kubeadm kubectl; do sudo apt-get install -y ${k}=$kver; done 
sudo apt-mark hold kubelet kubeadm kubectl 

Init the ControlNode

sudo kubeadm init --pod-network-cidr 10.24.0.0/16 --kubernetes-version ${kver%%-*}

A message like the following gives you the commands for joining worker nodes

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.30.174:6443 --token token_here
--discovery-token-ca-cert-hash sha256:shah_here

Or just run join command as when needed

kubeadm token create --print-join-command

Install networking Calico

The first time I did this, I didn't configure it after installation

kubectl create -f https://docs.projectcalico.org/archive/v3.21/manifests/tigera-operator.yaml
curl https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml -O
kubectl create -f custom-resources.yaml

calicoctl binary

Install the calicoctl

curl -L https://github.com/projectcalico/calico/releases/download/v3.23.2/calicoctl-linux-amd64 -o calicoctl

Optionals

kubectl apply -f https://raw.githubusercontent.com/linuxacademy/content-cka-resources/master/metrics-server-components.yaml
kubectl get --raw /apis/metrics.k8s.io

Notes

auto-complete:

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl
cat <<EOF >> ~/.bashrc
source /etc/bash_completion.d/kubectl
alias k=kubectl
complete -o default -F __start_kubectl k
EOF

Additional Plugins

Krew kubectl plugin installer

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

Known issue with iptables-nft

• workaround:

update-alternatives --set iptables /usr/sbin/iptables-legacy

Cleanup / Reset a cluster or nodes

sudo service kubelet stop
sudo dpkg --purge kubeadm kubelet
sudo rm -rf /etc/kubernetes /var/lib/etcd

flush/delete all iptablechains

iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X