Setup K3s HA with embed db (etcd)

vm-contabo-K3s-multi-nodes.md

K3s on VM Contabo

Before we create K3s node on contabo vm, we need to setup static routes so all our vm can communicate each other

Creating static routes

Static routes reference

Here i used ubuntu latest 22.04 distro we need to configure netplan and add static routes to it. The following examples assume that the server with the IPv4 address 62.172.167.31 and the server with the IPv4 address 62.172.167.89 want to communicate with each other.

Because of some particularities in Netplan, there are two routes that have to be added for each server ip. Please note the prefixes that that got changed to /25:

#/etc/netplan/01-netcfg.yaml
...
   ens18:
     routes:
       ...
       - to: 62.172.167.0/25
         via: 62.172.160.1
       - to: 62.172.167.128/25
         via: 62.172.160.1

Save it then type this command at once ip route flush table main; ip route flush cache; netplan apply

Notes: do not forget to set it on each server or vm.

after we configure the network then setup the K3s server master.

K3S Installing

first log to your 1st vm so it can be our control plane master, follow the command below:

Update the ubuntu

sudo apt update

Install k3s master

curl -sfL https://get.k3s.io | sh -s - server --cluster-init --disable traefik --write-kubeconfig-mode 644

wait til finish the setup, after installation is finish then check the k8s node kubectl get nodes

get k3s node-token cat /var/lib/rancher/k3s/server/node-token for join server or get k3s token cat /var/lib/rancher/k3s/server/token for join agent then copy to your notes.

for the information better check tls control plane master with this command

curl -vv --cacert /var/lib/rancher/k3s/server/tls/server-ca.crt https://<ip-node-master>:6443/api/v1/pods

* About to connect() to <ip-node-master> port 6443 (#0)
*   Trying <ip-node-master>...
* Connected to <ip-node-master> (<ip-node-master>) port 6443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /var/lib/rancher/k3s/server/tls/server-ca.crt
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=k3s,O=k3s
* 	start date: Jan 06 02:21:16 2022 GMT
* 	expire date: Jan 06 02:21:16 2023 GMT
* 	common name: k3s
* 	issuer: CN=k3s-server-ca@1641435676
> GET /api/v1/pods HTTP/1.1
> User-Agent: curl/7.29.0
> Host: <ip-node-master>:6443
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Audit-Id: 1d30dc47-bdf6-434d-ab8d-3cb74d411c57
< Cache-Control: no-cache, private
< Content-Type: application/json
< Date: Thu, 06 Jan 2022 11:31:58 GMT
< Content-Length: 165
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host <ip-node-master> left intact

Install K3s join node

log to 2nd server or vm, then update the distro.

setup K3s join node server:

curl -sfL https://get.k3s.io | sh -s - server --server https://<ip-node-master>:6443 --write-kubeconfig-mode 644 --token <node-token-from-master>

setup K3s join node agent:

curl -sfL https://get.k3s.io | sh -s - agent --server https://<ip-node-master>:6443 --token <token-from-master>

wait til finish the setup, after installation is finish then check on vm control master plane

kubectl get nodes

Accessing the Cluster from Outside with kubectl

Copy /etc/rancher/k3s/k3s.yaml on your machine located outside the cluster as ~/.kube/config. Then replace “localhost” with the IP or name of your K3s server. kubectl can now manage your K3s cluster

SSH-COPY_ID