k3s kubevip services

k3s-kubevip-services.txt

alias nctl='sudo /usr/local/bin/nerdctl -n k8s.io -a /run/k3s/containerd/containerd.sock'
alias kube-vip='nctl run --network host --rm ghcr.io/kube-vip/kube-vip:v0.8.1'

... or without nerdctl
TIMESTAMP=$(date +%s)
export CONTAINERD_ADDRESS='/run/k3s/containerd/containerd.sock'
export CONTAINERD_NAMESPACE='k8s.io'
alias kube_vip='sudo -E ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:v0.8.1 vip /kube-vip'
...

kube-vip manifest daemonset --interface $INTERFACE --address=$VIP_ADDRESS \
--controlplane --services --enableEndpointSlices --servicesElection --taint --inCluster \
--bgp --localAS $LOC_AS --bgpRouterID $BGP_ID --bgppeers 10.16.100.101:$LOC_AS::false | \
tee kubevip-$TIMESTAMP.yaml

... or use a kubeconfig (!!) pointing to k3s' default
kube-vip manifest daemonset --interface $INTERFACE --address=$VIP_ADDRESS \
--controlplane --services --enableEndpointSlices --servicesElection --taint \
--bgp --localAS $LOC_AS --bgpRouterID $BGP_ID --bgppeers 10.16.100.101:$LOC_AS::false | \
yq '.spec.template.spec.volumes[0].hostPath.path = "/etc/rancher/k3s/k3s.yaml"' | tee kubevip-$TIMESTAMP.yaml

### add bgp peers to routing equipment, peer-id above is an example 10.16.100.101

# for k3s/rke2 disable servicelb (k3s) and cloud-controller (k3s/rke2) --
...
disable-cloud-controller: true
### disable: servicelb  ... (optional, these nodes have multiple addresses)
### kept this servicelb, label nodes as below:
### kubectl label node/$NODE svccontroller.k3s.cattle.io/enablelb=false
### Edit: see comments, the bundled servicelb interferes with the kube-vip services controller
### basically kube-vip services > servicelb for my purposes

...

kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
kubectl create configmap -n kube-system kubevip --from-literal cidr-global=10.16.123.224/29
kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml

### upgrades are manifest-regen updating the image version, perform diff-eval & kubectl apply -f 

added --taint because the ds should only be on cp nodes, k3s.yaml needed for services discovery, not available on worker-only nodes.

... still some weirdness with this, added the disable: servicelb config option to test.

Edit: Experiment successful. >12h with the servicelb disabled, and the kube-vip services are still fine, this proves servicelb does interfere with kube-vip operation. This is called out in the kube-vip docs.

Added ctr commands in case nerdctl is not available or necessary. Set environment accordingly.