Skip to content

Instantly share code, notes, and snippets.

@ssro

ssro/K3S setup in HA mode.md

Last active October 31, 2023 05:21
Show Gist options
  • Save ssro/3dcd712a99ad89840820a9f113e5da11 to your computer and use it in GitHub Desktop.
Save ssro/3dcd712a99ad89840820a9f113e5da11 to your computer and use it in GitHub Desktop.
Download ZIP
K3S HA Setup
Raw
K3S setup in HA mode.md

Create a highly available k3s cluster with external datastore

Prereqs

For this particular setup we will need 3 machines to act as kubernetes (k3s) masters and one machine to act as a datastore

The datastore will be a mysql (mariadb) server although you can choose different options

The operating system chosen for this task is Debian 11 (theoretically the setup can be performed on any linux machine)

Once the machines are ready and network communication is allowed between them we can proceed to...

Setup MySQL (MariaDB) server

sudo apt update && sudo apt upgrade -y && sudo apt install mariadb-server -y

After the db server is installed, let's configure it properly

sudo mysql_secure_installation
Enter current password for root (enter for none): [ENTER]
Switch to unix_socket authentication [Y/n] y
Change the root password? [Y/n] y
New password: YOUR$uper$ecr3tPa$s
Re-enter new password: YOUR$uper$ecr3tPa$s
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] y
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y

Now let's allow network connections to the db server

sudo vi /etc/mysql/mariadb.conf.d/50-server.cnf

Comment bind-address = 127.0.0.1 and add bind-address = YOUR_SERVER_IP (or change the ip to point to your server's ip)

Next step is to create a user and a database and allow this user to connect from the k3s servers to our db server. The network that we use in this example is 10.11.3.0/24

mysql -uroot -p
MariaDB [(none)]> CREATE DATABASE k3s;

MariaDB [(none)]> CREATE USER 'k3s'@'10.11.3.%' IDENTIFIED BY 'SomePassworD';

MariaDB [(none)]> GRANT ALL PRIVILEGES ON k3s.* TO 'k3s'@'10.11.3.%';

Finally restart the db server:

sudo systemctl restart mariadb.service

Check if it's binding to the IP address you specified

netstat -an | grep 3306

tcp        0      0 YOUR_SERVER_IP:3306        0.0.0.0:*               LISTEN

MySQL (MariaDB) setup is now complete

Setup k3s servers (masters)

Create a random string to act as a setup token, for example:

tr -cd '[:alnum:]' < /dev/urandom | fold -w32 | head -n1

Execute the following command on the first server

curl -sLS https://get.k3s.io | INSTALL_K3S_EXEC="server --disable traefik --disable servicelb --write-kubeconfig-mode 644 --kubelet-arg eviction-hard=memory.available<300Mi --kubelet-arg=image-gc-high-threshold=85 --kubelet-arg=image-gc-low-threshold=80 --kube-apiserver-arg=enable-aggregator-routing=true" INSTALL_K3S_VERSION="v1.28.2+k3s1" K3S_DATASTORE_ENDPOINT="mysql://k3s:SomePassworD@tcp(YOUR_SERVER_IP:3306)/k3s" K3S_TOKEN="RANDOM_STRING_TOKEN" sh -

For the rest of the servers, just disable metrics-server. It seems that it's not working properly if it's enabled on all machines (maybe a bug?)

curl -sLS https://get.k3s.io | INSTALL_K3S_EXEC="server --disable metrics-server --disable traefik --disable servicelb --write-kubeconfig-mode 644 --kubelet-arg eviction-hard=memory.available<300Mi --kubelet-arg=image-gc-high-threshold=85 --kubelet-arg=image-gc-low-threshold=80 --kube-apiserver-arg=enable-aggregator-routing=true" INSTALL_K3S_VERSION="v1.28.2+k3s1" K3S_DATASTORE_ENDPOINT="mysql://k3s:SomePassworD@tcp(YOUR_SERVER_IP:3306)/k3s" K3S_TOKEN="RANDOM_STRING_TOKEN" sh -

Don't worry, the metrics-server will be scheduled on other nodes in case it gets evicted/deleted

NOTE: in the above setup we disabled servicelb and traefik, so there will be no ingress controller. As an alternative nginx-ingress and metallb can be installed.

NOTE: Some params can be tweaked according to your needs.

After a succesfull run, you should see your masters active:

kubectl get nodes

NAME             STATUS   ROLES                  AGE   VERSION
ip-10-11-3-150   Ready    control-plane,master   97s   v1.21.4+k3s1
ip-10-11-3-24    Ready    control-plane,master   23s   v1.21.4+k3s1
ip-10-11-3-215   Ready    control-plane,master   12s   v1.21.4+k3s1

Setup k3s servers (agents)

Agents can be installed using the following command:

curl -sLS https://get.k3s.io | K3S_URL="https://IP_OF_ONE_MASTER:6443" INSTALL_K3S_EXEC="agent" INSTALL_K3S_VERSION="v1.28.2+k3s1" K3S_DATASTORE_ENDPOINT="mysql://k3s:SomePassworD@tcp(YOUR_SERVER_IP:3306)/k3s" K3S_TOKEN="RANDOM_STRING_TOKEN" sh -

Failure simulation

Let's uninstall k3s from one of the servers:

/usr/local/bin/k3s-uninstall.sh

Monitor the cluster status:

kubectl get nodes

NAME             STATUS     ROLES                  AGE     VERSION
ip-10-11-3-24    NotReady   control-plane,master   5m56s   v1.21.4+k3s1
ip-10-11-3-150   Ready      control-plane,master   7m10s   v1.21.4+k3s1
ip-10-11-3-215   Ready      control-plane,master   5m45s   v1.21.4+k3s1

We see that the affected node is shown as NotReady

Running the k3s setup above, it will render the node Ready. Yes, is that simple!

kubectl get nodes

NAME             STATUS   ROLES                  AGE   VERSION
ip-10-11-3-150   Ready    control-plane,master   10m   v1.21.4+k3s1
ip-10-11-3-215   Ready    control-plane,master   9m    v1.21.4+k3s1
ip-10-11-3-24    Ready    control-plane,master   9m    v1.21.4+k3s1

Adding nodes

For any new node that you'd like to add (as master), just re-run the k3s command (keep in mind to disable the metrics-server). For agents, just run the setup with the correct parameters (agent)

Final thoughts

An external datastore as opposed to embedded etcd has one major benefit: nodes can be removed or added without any issues. With embedded etcd this is difficult, if not impossible.

Keep your external datastore backed up at all times!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment