patrick0057

etcd performance testing and optimization

If your etcd logs start showing messages like the following, your storage might be too slow for etcd or the server might be doing too much for etcd to operate properly.

2019-08-11 23:27:04.344948 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:293" took too long (1.530802357s) to execute

If you storage is really slow you will even see it throwing alerts in your monitoring system. What can you do the verify the performance of your storage? If the storage is is not performing correctly, how can you fix it? After researching this I found an IBM article that went over this extensively. Their findings on how to test were very helpful. The biggest factor is your storage latency. If it is not well below 10ms in the 99th percentile, you will see warnings in the etcd logs. We can test this with a tool called fio which I will outline below.

Testing etcd per

patrick0057

Update self signed certificate on single install of Rancher 2.x

1. Download Rancher single tool on the server that is running your Rancher container:

   curl -LO https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh

2. Run script so that it upgrades your installation (you can upgrade to the same version) and pass flags to indicate that you want to regenerate your self signed certificate. The most reliable way is to just specify all of your options on the command line but the script does have an easy to use automated system as well as shown in option b.

   a. Specify all flags on command line, including any rancher options you had and docker options. Option -s is required for generating new 10 year self signed SSL certificates.

patrick0057

Deploy new cluster agent YAML

If you've recently made changes to your Rancher installation like updating the server URL or changing the Rancher installation SSL, then you will likely need to redeploy your cluster agent YAML files from the Rancher.

1. Create a local admin user for use with this tutorial. Without this, the script cannot login to get the new deployment file. You cannot use user accounts that are tied to third party authentication such as LDAP, Active Directory or GitHub to name a few.
2. Login to a single controlplane node of the cluster you need to redeploy your agent YAML to.
3. Download the script:

   curl -LO https://github.com/patrick0057/cluster-agent-tool/raw/master/cluster-agent-tool.sh

wget https://github.com/patrick0057/cluster-agent-tool/raw/master/cluster-agent-tool.sh

patrick0057

kube-apiserver restart loop

If the kube-apiserver is in a restart loop, it is possible that one of the etcd servers it is trying to connect to is no longer reachable. It should be able to just move on to the next etcd server but in some rare cases it does not. In those situations you need to remove the bad etcd servers from its startup options as shown below.

1. Get runlike command for kube-apiserverwith the following command:

   docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver

   Example output:

patrick0057

go dumps for Rancher 2.x support

If you are experiencing random crashing of your Rancher 2.x pods or docker container, sometimes Rancher support will ask you to take a go routine dump or a memory dump. Below are the commmands you need to run inside of your Rancher container or pod.

#exec into Rancher pod or container first
mkdir dumps
curl localhost:6060/debug/pprof/goroutine -o dumps/goroutine
curl localhost:6060/debug/pprof/heap -o dumps/heap
curl localhost:6060/debug/pprof/threadcreate -o dumps/threadcreate
curl localhost:6060/debug/pprof/block -o dumps/block
curl localhost:6060/debug/pprof/mutex -o dumps/mutex

patrick0057

Saving a copy of this script here for later use. I found this at the following stackoverflow post.

• https://stackoverflow.com/questions/21752067/counting-open-files-per-process

#!/bin/bash
cd /proc
for pid in [0-9]*
do
    echo "PID = $pid with $(ls /proc/$pid/fd/ | wc -l) file descriptors"
done | sort -rn -k5 | head | while read -r _ _ pid _ fdcount _

patrick0057

Change Rancher 2.x server-url

Single server installation

During this tutorial it is recommended to use the rancher-single-tool for Rancher single server installations. It isn't required but it makes the process much easier. As a result this guide will be based on using that tool.

1. Download the rancher-single-tool to the node that is running your rancher server container.

      curl -LO https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh
      wget https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh

patrick0057

Overview

When using RKE 0.2.0 and newer, if you have restored a cluster with the incorrect rkestate file you will end up a state where your infrastructure pods will not start. This includes all pods in kube-system, cattle-system and ingress-nginx. As a result of these core pods not starting, all of your workload pods will be unable to function correctly. If you find yourself in this situation you can use the directions below to fix the cluster.

Recovery

1. Delete all service-account-token secrets in kube-system, cattle-system and ingress-nginx namespaces.

{
kubectl get secret -n cattle-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n cattle-system delete secret " $1) }'
kubectl get secret -n kube-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n kube-system delete secret " $1) }'

patrick0057

lssh function

Changelog

• 06.03.2020 - added zsh function that works for WSL
• 04.21.2020 - updated mac os x code to work with zsh and improved instructions.
• 01.02.2020 - added Windows Subsystem for Linux and broke out each OS into its own section for easy copy and paste
• 12.10.2019 - added support for Rancher 1.6 tar.gz files (requires gtar on mac)
• 12.06.2019 - made command lazier by not requiring user to paste the IP.

Description

Quick bash function to make my life easier when sshing into Rancher nodes. Make sure to update your default web browser download directory by modifying line 2 of the script. For mac: brew install findutils

patrick0057

Major disaster preparation and recovery

In a perfect world our clusters would never experience a complete and total failure where data from all nodes is unrecoverable. Unfortunately this scenario is very possible and has happened before. In this article I will outline how to best prepare your environment for recovery in situations like this.

Situation: Employee A accidentally deletes all of the VM's for a production cluster after testing his latest script. How do you recover?

Option A: Keep VM snapshots of all of the nodes so that you can just restore them if they are deleted.

Option B: Manually bootstrap a new controlplane and etcd node to match one of the original nodes that were deleted.

In this article, I'm going to focus on Option B. In order to bootstrap a controlplane,etcd node, you will need an etcd snapshot, Kubernetes certificates and the runlike commands from the core Kubernetes components. If you prepare ahead of time for something like this, you can save a lot of time when it comes