superseb

RKE2 commands

• Updated on May 29 to accommodate etcd container not having /bin/sh available anymore.

Install

curl -sL https://get.rke2.io | sh
systemctl daemon-reload
systemctl start rke2-server

patrick0057

working with etcd keys, notes

restore snapshot to work with

docker run --rm -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd etcdctl snapshot restore snapshot.db

run snapshot to work with

docker run --name etcd -d -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd

dump keys to debug

Oats87

kubectl get configmap -n kube-system cattle-controllers -o json | jq -r '.metadata.annotations["control-plane.alpha.kubernetes.io/leader"]' | jq -r ".holderIdentity"

dkeightley

http_code: %{http_code}\n
http_connect: %{http_connect}\n
time_total: %{time_total}\n
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
size_download: %{size_download}\n

Oats87

#!/bin/bash
export KUBECONFIG=$(pwd)/kube_config_cluster.yml
IFS=$'\n'; for i in $(kubectl get secrets --all-namespaces | grep "service-account-token"); do ns=$(echo $i | awk '{print $1}'); sec=$(echo $i | awk '{print $2}'); kubectl patch secret -n $ns $sec -p '{"metadata":{"finalizers": []}}' --type=merge && kubectl delete secret $sec -n $ns --wait=false; done
IFS=$'\n'; for i in $(kubectl get pods --all-namespaces | grep -v "NAME"); do  ns=$(echo $i | awk '{print $1}'); pod=$(echo $i | awk '{print $2}'); kubectl delete pod $pod -n $ns --wait=false; done

patrick0057

etcd performance testing and optimization

If your etcd logs start showing messages like the following, your storage might be too slow for etcd or the server might be doing too much for etcd to operate properly.

2019-08-11 23:27:04.344948 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:293" took too long (1.530802357s) to execute

If you storage is really slow you will even see it throwing alerts in your monitoring system. What can you do the verify the performance of your storage? If the storage is is not performing correctly, how can you fix it? After researching this I found an IBM article that went over this extensively. Their findings on how to test were very helpful. The biggest factor is your storage latency. If it is not well below 10ms in the 99th percentile, you will see warnings in the etcd logs. We can test this with a tool called fio which I will outline below.

Testing etcd per

superseb

Troubleshooting Kubernetes commands

Commands belonging to the Rancher webinar Troubleshooting Kubernetes

etcd

Check etcd members

docker exec etcd etcdctl member list

janeczku

Steps to change the URL of Rancher installation and switch from a self-signed certificate to a certificate signed by recognized CA.

1. Change the Rancher server-url setting to the new URL:
   • Navigate to https://<old_rancher_hostname>/g/settings/advanced
   • Edit server-url to https://<new_rancher_hostname>
2. Clear the private CA certificate for the old certificate
   • Navigate to https://<old_rancher_hostname>/g/settings/advanced
   • Next to cacerts click context menu -> View in API
   • Click Edit
   • Clear the content of the value field

Oats87

{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ]
}

Oats87

# Docker 17.03.2 on RHEL/CentOS 7

Rancher has discovered a few issues when running Upstream Docker 17.03.2 on RHEL/CentOS 7. This document is being written to document recommendations for Docker configuration in order to ensure reliability while operating Kubernetes and Rancher with RHEL/CentOS 7.

### Overlay2 Storage Driver

Currently, Upstream Docker 17.03.2 performs kernel version validations to enable overlay2. As official overlay2 support from the upstream Linux kernel was not enabled until version 4.0 or higher than the kernel. Red Hat backported overlay2 support to 3.10.0-514 of their kernel.

Overlay2 support can be enabled by setting the following contents in the `/etc/docker/daemon.json` file and restarting Docker. Please note that you must do this on a fresh docker installation, or remove all running containers before performing this action.
```{