K3s Installation Notes

K3sInstallationNotes.md

K3s Installation Notes (for running Gitlab Runner)

Install k3s master and expose api:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 --bind-address 0.0.0.0" sh -

The --write-kubeconfig-mode 644 gives /etc/rancher/k3s/k3s.yaml group and world read permissions so that we can run kubectl without sudo. The INSTALL_K3S_EXEC argument can be modified at /etc/systemd/system/multi-user.target.wants/k3s.service.

Configuration Options that we can put into INSTALL_K3S_EXEC: https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/

Check that the cluster is accessible from the outside (taken from here):

curl --insecure https://SERVER_IP:6443/

You should see something like this:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Note that you may need to create a firewall rule or disable the firewall (sudo ufw disable) to allow external access.

Create a namespace for gitlab runner:

sudo kubectl create namespace gitlab

Register gitlab-runner (on the master machine) and modify its configuration to match (mainly host, cert_file, key_file and ca_file):

[[runners]]
  name = "<RUNNER_NAME>"
  url = "<GITLAB_URL>"
  token = "<SOME_TOKEN_FROM_THE_UI>"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = "https://<HOST_NAME>:6443"
    cert_file = "/var/lib/rancher/k3s/server/tls/client-admin.crt"
    key_file = "/var/lib/rancher/k3s/server/tls/client-admin.key"
    ca_file = "/var/lib/rancher/k3s/server/tls/server-ca.crt"
    bearer_token_overwrite_allowed = false
    # default image
    image = "docker:19.03.12"
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    privileged = true
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    # default services
    [[runners.kubernetes.services]]
      name = "docker:19.03.12-dind"
    # choose which node to run on (hard constraint)
    #[runners.kubernetes.node_selector]
    #  disktype = "ssd"
    # choose which node to run on (soft constraint)
    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.node_affinity]
        [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
        weight = 100
        [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
          [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
           key = "disktype"
           operator = "In"
           values = ["ssd"]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]
    [runners.kubernetes.node_tolerations]
      "node.kubernetes.io/disk-pressure" = "NoSchedule"

• The node_selector configuration enforces that the runner only runs on certain nodes with special labels (disktype=ssd) in our case.
• The affinity configuration makes the runner prefer nodes labelled with disktype=ssd.
• The tolerations configuration allows the runner to scheule pods on nodes that report low disk space.

To label a node (reference):

# add label disktype=ssd to a node
kubectl label nodes <your-node-name> disktype=ssd

To add another machine to this cluster:

# get the server token on the master
sudo cat /var/lib/rancher/k3s/server/node-token
# run this on the new machine
curl -sfL https://get.k3s.io | K3S_URL=https://<MASTER_NAME_OR_IP>:6443 K3S_TOKEN=<TOKEN_FROM_ABOVE> INSTALL_K3S_EXEC="agent" sh -

The INSTALL_K3S_EXEC argument is optional and can be modified at /etc/systemd/system/multi-user.target.wants/k3s-agent.service.

Configuration Options that we can put into INSTALL_K3S_EXEC: https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/

Some useful links:

• https://docs.gitlab.com/runner/executors/#kubernetes-executor
• https://docs.gitlab.com/runner/install/kubernetes.html#configuring-gitlab-runner-using-the-helm-chart
• Kubectl cheatsheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Kubernetes dashboard:

https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/ Obtain token:

sudo k3s kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token

Helm access:

https://rancher.com/docs/k3s/latest/en/cluster-access/

helm --kubeconfig /etc/rancher/k3s/k3s.yaml ls --all-namespaces

Customize kubelet argument (to configure Eviction Thresholds, etc.):

e.g. Add --kubelet-arg eviction-hard=memory.available<100Mi to INSTALL_K3S_EXEC during installation or in the service configurations (/etc/systemd/system/multi-user.target.wants/k3s.service or /etc/systemd/system/multi-user.target.wants/k3s-agent.service)

Other notes:

k3s.yml syntax is a direct translation from the command line arguments https://rancher.com/docs/k3s/latest/en/installation/install-options/#registration-options-for-the-k3s-server

Server (master node) and agent (node) service file location:

https://kauri.io/collections/Build%20your%20very%20own%20self-hosting%20platform%20with%20Raspberry%20Pi%20and%20Kubernetes/(38)-install-and-configure-a-kubernetes-cluster-w/

• Server: /etc/systemd/system/k3s.service
• Agent: /etc/systemd/system/k3s-agent.service

Add the cluster to gitlab (this is not necessary for setting up the runner): https://docs.gitlab.com/ee/user/project/clusters/add_remove_clusters.html#add-existing-cluster

References

k3s + Gitlab install notes