cfrogjump/k3s-cluster.md

## k3s-cluster.md

      
    Raw
  

              k3s-cluster.md
            
          
    Deploy K3s and Rancher on Raspberry Pi 4 cluster

Today I tried to setup a small Kubernetes cluster on top of 3 Raspberry Pi 4 (4GB Memory).
Here is the steps to install the cluster.

Preparation

I have 3 Raspberry Pi 4 stacked with PoE headers and connected to a PoE switch at home. The are connected to Internet through a home router. All Pis are equipped with a 64GB Samsung SDXC card flushed with Ubuntu 20.04 image.
Deploy K3s

Basically I followed Install and configure a Kubernetes cluster to deploy the cluster.

On 1st Pi (Control node), run following commands:

export K3S_KUBECONFIG_MODE="644"
export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik"
curl -sfL https://get.k3s.io | sh -

# Get token of the k3s svc
sudo cat /var/lib/rancher/k3s/server/node-token

On 2nd and 3rd Pi (Worker node), run following command

export K3S_URL="https://piattop.zhenfang.home:6443"
export K3S_KUBECONFIG_MODE="644"
export K3S_TOKEN="<TOKEN>"
curl -sfL https://get.k3s.io | sh -
Deploy loadbalancer and ingress controller

Here we deploy the loadbalancer and ingress controller onto the cluster.

Use Helm(v3) to install Loadbalancer

helm repo add metallb https://metallb.github.io/metallb
helm repo update

kubectl create secret generic -n metallb-system metallb-memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

noglob helm install metallb metallb/metallb --namespace kube-system \
  --set configInline.address-pools[0].name=default \
  --set configInline.address-pools[0].protocol=layer2 \
  --set configInline.address-pools[0].addresses[0]=10.0.4.200-10.0.4.250

Use Helm(v3) to install ingress controller

kubectl create namespace ingress-controller
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update
helm install nginx-ingress nginx-stable/nginx-ingress \
  --namespace ingress-controller \
  --set controller.image.repository=quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm64 \
  --set controller.image.tag=0.25.1 \
  --set controller.image.runAsUser=33 \
  --set defaultBackend.enabled=false
Install Cert-manager

Cert manager will help you to grant TLS keys from Letsencrypt. I created a separate namespace to install it. I refer to Rancher's cert-manager install doc
# Install the CustomResourceDefinition resources separately
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml

# **Important:**
# If you are running Kubernetes v1.15 or below, you
# will need to add the `--validate=false` flag to your
# kubectl apply command, or else you will receive a
# validation error relating to the
# x-kubernetes-preserve-unknown-fields field in
# cert-manager’s CustomResourceDefinition resources.
# This is a benign error and occurs due to the way kubectl
# performs resource validation.

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v0.12.0
Install Rancher

In order to manage the cluster properly. I installed Rancher 2 onto the cluster. The reference doc is Install Rancher on Kubernetes Cluster
kubectl create namespace cattle-system
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=<HOST_NAME> \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=<EMAIL_ADDR>
N.B. There is some "race-condition" during rancher deployment. The certificate granting might ends up failure or pending. Here you need to do following checks:

Is the dst-nat and masqurading properly set on home router
Is the certificate created successfully.

kubectl -n cattle-system get certificate
kubectl -n cattle-system get issuer
kubectl -n cattle-system get certificaterequest
kubectl -n cattle-system describe certificaterequest rancher

If the certificaterequest got stuck. Try to delete the ingress and re-create it.

helm -n cattle-system get values rancher > rancher-values.yaml
kubectl -n cattle-system delete ingress rancher
helm -n cattle-system upgrade -i rancher \
  rancher-latest/rancher -f rancher-values.yaml
After Rancher is installed, we could check if Rancher is accessible from WAN and LAN.

The cluster is shown in below pic.

Prepare cluster-issuer for further deployment

In order to make use of cert-managet to other services, create cluster-issuer.
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: <EMAIL>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: <EMAIL>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF
NFS provisioner

I have an Synology diskstation home with NFS server enabled. In order to get pvc support on K3s cluster, i use following command to deploy the NFS client provisioner.
helm -n nfs-provisioner install diskstation \
  stable/nfs-client-provisioner \
  --set nfs.server=<DISKSTATION_IP> \
  --set nfs.path=<NFS_MOUNTING_PATH> \
  --set image.repository=kopkop/nfs-client-provisioner-arm64
N.B. Remember to install NFS client on every worker node first if it is not pre-installed.
sudo apt update && sudo apt -y install nfs-common