Basic local k3s setup with Multipass, k3sup, metallb and nginx

k3s-dev-env.md

Using Multipass VMs and K3s for Nucleus Development Env

Assumes multipass is installed and configured, Multipass can be installed on Linux, Mac and Windows. If using Linux, It's highly recommended to configure Multipass to use KVM/Libvirt.

Multipass

Create a could-init file with user settings:

users:
  - default
  - name: ubuntu
    sudo:  ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      -   <ssh-rsa-key-pasted-here>

Create a control plane VM and two workload VMs for your k3s cluster:

multipass launch -c2 -m4G -d50G -n k3s-ctrl  --cloud-init cloud-init
multipass launch -c2 -m4G -d50G -n k3s-1  --cloud-init cloud-init
multipass launch -c2 -m4G -d50G -n k3s-2 --cloud-init cloud-init

Perform a multipass list to obtain status and IP info:

Name                    State             IPv4             Image
k3s-1                   Running           192.168.122.157  Ubuntu 20.04 LTS
k3s-2                   Running           192.168.122.65   Ubuntu 20.04 LTS
k3s-ctrl                Running           192.168.122.54   Ubuntu 20.04 LTS

Kubernetes install using k3sup

Install the control plane, NOTE we want to disble the built in default Traefik ingressand Klipper Load Balancer services:

k3sup install -ip 192.168.122.54 --cluster --user ubuntu --k3s-channel stable --local-path ~/k3s-kube-config --merge --k3s-extra-args '--no-deploy traefik --write-kubeconfig-mode 644'

After the control-plane services have been deployed succesfully you can use k3sup to deploy k3s to the worker nodes and join them to the cluster:

k3sup join --ip 192.168.122.157 --server-ip 192.168.122.54 --user ubuntu
k3sup join --ip 192.168.122.65 --server-ip 192.168.122.54 --user ubuntu

MetalLB (https://metallb.org/installation/#installation-by-manifest)

Create a metallb namespace and install the controller and daemonset:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

Now you'll need to configure MetalLB for your IP ranges. In our case we're using the simple L2 model; you'll need to give MetalLB a range of IPs that are valid for the multipass vm network to use. In our example, our Multipass VMs are in the range 192.168.122.x, so we'll grab a few off the top of that range.

metal-lb-config.yml:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.122.245-192.168.122.250
EOF

NGINX

We use the kubernetes upstream community version of Nginx:

helm upgrade --install ingress-nginx ingress-nginx  --repo https://kubernetes.github.io/ingress-nginx --namespace kube-system

Test that everything is working:

kubectl get svc -A
NAMESPACE     NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
default       kubernetes                           ClusterIP      10.43.0.1       <none>            443/TCP                      38m
kube-system   ingress-nginx-controller             LoadBalancer   10.43.64.197    192.168.122.245   80:31194/TCP,443:32646/TCP   26m
kube-system   ingress-nginx-controller-admission   ClusterIP      10.43.13.159    <none>            443/TCP                      26m
kube-system   kube-dns                             ClusterIP      10.43.0.10      <none>            53/UDP,53/TCP,9153/TCP       38m
kube-system   metrics-server                       ClusterIP      10.43.194.131   <none>            443/TCP                      38m

NOTE the ingress-nginx-controller EXTERNAL-IP entry... (in this case 192.168.122.245), that is your external Load Balancer IP to access your services of type LB. This should be the first address in the range you specified in your MetalLB config.