Skip to content

Instantly share code, notes, and snippets.

@lisa

lisa/README.md

Last active June 13, 2022 04:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lisa/c24a0da7060c156f54201e7dc216cc2d to your computer and use it in GitHub Desktop.
Save lisa/c24a0da7060c156f54201e7dc216cc2d to your computer and use it in GitHub Desktop.
Download ZIP
Adding a new control-plane node to an existing Kubernetes cluster
Raw
README.md

Adding a new control-plane node to an existing Kubernetes cluster

This document is meant to collect information from various sources to serve as a quick and dirty reminder to myself before ending up on lisa.dev

$ k get node -o wide
NAME         STATUS   ROLES                  AGE      VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
kube2        Ready    <none>                 2y338d   v1.22.8   192.168.86.11   <none>        Ubuntu 18.04.6 LTS               5.1.0-rockchip64   containerd://1.5.5
kube3        Ready    <none>                 2y338d   v1.22.8   192.168.86.12   <none>        Ubuntu 18.04.6 LTS               5.1.0-rockchip64   containerd://1.5.5
kube4        Ready    control-plane          8m45s    v1.24.1   192.168.86.13   <none>        Debian GNU/Linux 11 (bullseye)   5.15.32-v8+        containerd://1.4.13
kubemaster   Ready    control-plane,master   2y338d   v1.23.7   192.168.86.10   <none>        Ubuntu 18.04.6 LTS               5.1.0-rockchip64   containerd://1.5.5

Pre-Requisites

  • A single-control-plane ("master") node cluster
  • Root/privileged access
  • DNS access

This guide is not targeted at any cloud provider, your mileage may vary.

Set Up DNS

(Note: This section is based on upstream's ha-considerations)

In your DNS provider, add a new A (and/or AAAA) record for your cluster's highly available endpoint. Ex

kube-api.example.com A 192.168.86.10
kube-api.example.com A 192.168.86.13

Set up haproxy

(Note: This section is based on upstream's ha-considerations)

This will configure haproxy to proxy port 8443/tcp to 6443/tcp.

Create /etc/haproxy/haproxy.cfg:

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:8443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server kubemaster 192.168.86.10:6443 check
        server kube4 192.168.86.13:6443 check

On each control-plane node, create /etc/kubernetes/manifests/haproxy.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.1.4
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        port: 6443
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

Prepare Certs

Ensure your kube-system/configmaps/kubeadm-config ConfigMap has a controlPlaneEndpoint for your DNS setup: ex controlPlaneEndpoint: kube-api.example.com:8443. While you're in there, add a apiServer.certSANs list with the same controlPlaneEndpoint name (sans port) IP addresses of your control-plane nodes in a list.

Use kubeadm to renew your certs:

rm -f /etc/kubernetes/pki/apiserver.{crt,key}
kubeadm certs renew all
kubeadm init phase upload-certs --upload-certs

Take note of the key from the previous kubeadm init phase upload-certs --upload-certs output.

Restart kube-apiserver, kube-controller-manager, kube-scheduler (and haproxy for good measure) pods:

kubectl -n kube-system delete pod/kube-apiserver-kubemaster pod/kube-controller-manager-kubemaster pod/kube-scheduler-kubemaster pod/haproxy-kubemaster

Create a join token for the new node

kubeadm token create --certificate-key kube4 --print-join-command --description "join kube4"

Join the new node

kubeadm join 192.168.86.10:6443 --token r36anl.wdeqvrvdpervn6cq --discovery-token-ca-cert-hash sha256:c5f9959f650b9d61245bec9ae31eab990db3e353af1b687b28c95ae97bd519a1 --control-plane --certificate-key <output from kubeadm init phase upload-certs --upload-certs> --control-plane --apiserver-advertise-address 192.168.86.13 --node-name kube4

In my case I must provide --apiserver-advertise-address because Debian is dumb and is multihoming this interface. The command is based on the output from the previous kubeadm token create.

Update your KUBECONFIG

Edit your ~/.kube/config to change the server to the new HA address

Update other cluster components

The remainder here is TODO, but don't forget that some cluster components will want to take advantage of the HA API Server now, including kube-proxy.

kube-proxy update

Edit kube-system/configmaps/kube-proxy:

apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
# ...
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://kube-api.example.com:8443
      name: default
# ...

TODO

  • Edit metallb speaker DaemonSet to add node-role.kubernetes.io/control-plane:NoSchedule toleration
  • Add a Service for haproxy (since it otherwise seems to use the locally installed haproxy sigh)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment