patrick0057

#!/bin/bash
#Blank values must be filled in unless specified otherwise by preceding comment.
#Name of the rancher environment you want to deploy your node to
export RANCHER_ENVIRONMENT='Default'
#Full path of your Rancher CLI executable
export RANCHERCLI_PATH='/usr/local/bin/rancher'
export MACHINE_DRIVER='azure'
#Name of your Node
export NAME=''
#AZURE_ENVIRONMENT can be: AzureChinaCloud, AzureGermanCloud, AzurePublicCloud or AzureUSGovernmentCloud

patrick0057

Change Rancher 2.x server hostname

Credit for the information in this document goes to Superseb. I am just publishing it in an easy to follow gist for later use. Before starting on this guide ensure you have offline backups of etcd for your local Rancher cluster and all of your downstream clusters. Steps outlined in this document are unsupported, use at your own risk. I recommend performing steps in a test environment first.

1. Navigate to Global> Settings> then find server-url in the list, click the triple dot and then "Edit'. Change the server-url to your desired value.
2. Navigate to https://$server-url/v3/clusterregistrationtoken?clusterId=$CLUSTERID and grab the value from Data> insecureCommand>

   • Example value:

     curl --insecure -sfL https://$server-url/v3/import/2bdrqnkjzc7rbjsvg6j6dv9hgttmjgl84dw8tz775qkczq8qkkhh6t.yaml | kubectl apply -f -`
     

patrick0057

Quick gist for curling etcd metrics. There are better ways to get the metrics but I'm creating this gist anyway in case I need to reference this again later.

export etcd_endpoint=$(docker exec etcd netstat -lpna | grep \:2379 | grep tcp | grep LISTEN | tr -s " " | cut -d" " -f4)

{ for var in $(docker inspect --format '{{ .Config.Env }}' etcd | sed 's/[][]//g'); do
if [[ "$var" == *"ETCDCTL_CERT"* ]] || [[ "$var" == *"ETCDCTL_KEY"* ]]; then
export ${var}
fi
done }

patrick0057

cmd/cloud-controller-manager/app/controllermanager.go:          return c.ClientBuilder.ClientOrDie(serviceAccountName)
cmd/cloud-controller-manager/app/options/options.go:    c.VersionedClient = rootClientBuilder.ClientOrDie("shared-informers")
cmd/kube-controller-manager/app/apps.go:                ctx.ClientBuilder.ClientOrDie("daemon-set-controller"),
cmd/kube-controller-manager/app/apps.go:                ctx.ClientBuilder.ClientOrDie("statefulset-controller"),
cmd/kube-controller-manager/app/apps.go:                ctx.ClientBuilder.ClientOrDie("replicaset-controller"),
cmd/kube-controller-manager/app/apps.go:                ctx.ClientBuilder.ClientOrDie("deployment-controller"),
cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie("horizontal-pod-autoscaler")
cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie("horizontal-pod-autoscaler")
cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie

patrick0057

Rancher 1.6 IPSec

Troubleshooting

If you are experiencing issues with containers communicating to each other in your Rancher 1.6 environment, your ipsec might be having some issues. In this article I will go over common troubleshooting steps and procedures to correct the problem.

• exec into one of your ipsec-router containers and run the following ipsec test

for i in `curl -s rancher-metadata/latest/self/service/containers/| cut -f1 -d=` ; do ping -c2 `curl -s curl rancher-metadata/latest/self/service/containers/$i/primary_ip` ; done

patrick0057

Below I'm archiving a slightly modified quote from Alena Prokharchyk in this Gist as it contains useful information that I would like to reference later.

Memory resources in Kubernetes are mesured in bytes, and can be expressed as an integer with one of these suffixes: E, P, T, G, M, K - decimal suffixes, or Ei, Pi, Ti, Gi, Mi, Ki - binary > suffixes (more commonly used for memory), or omit the suffix altogether (https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory). Lowercase "m" > is not a recommended suffix for memory.

Cluster.requested.memory is comprised of corresponding node.requested.memory. node.requested.memory is the sum of requested.memory of all the pods scheduled on this node.

While most of the nodes have memory with binary suffixes - memory": "2630Mi", two of them have a suffix "m":

node1.acme.com = 3731881984500m >

patrick0057

working with etcd keys, notes

restore snapshot to work with

docker run --rm -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd etcdctl snapshot restore snapshot.db

run snapshot to work with

docker run --name etcd -d -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd

dump keys to debug

patrick0057

Major disaster preparation and recovery

In a perfect world our clusters would never experience a complete and total failure where data from all nodes is unrecoverable. Unfortunately this scenario is very possible and has happened before. In this article I will outline how to best prepare your environment for recovery in situations like this.

Situation: Employee A accidentally deletes all of the VM's for a production cluster after testing his latest script. How do you recover?

Option A: Keep VM snapshots of all of the nodes so that you can just restore them if they are deleted.

Option B: Manually bootstrap a new controlplane and etcd node to match one of the original nodes that were deleted.

In this article, I'm going to focus on Option B. In order to bootstrap a controlplane,etcd node, you will need an etcd snapshot, Kubernetes certificates and the runlike commands from the core Kubernetes components. If you prepare ahead of time for something like this, you can save a lot of time when it comes

patrick0057

lssh function

Changelog

• 06.03.2020 - added zsh function that works for WSL
• 04.21.2020 - updated mac os x code to work with zsh and improved instructions.
• 01.02.2020 - added Windows Subsystem for Linux and broke out each OS into its own section for easy copy and paste
• 12.10.2019 - added support for Rancher 1.6 tar.gz files (requires gtar on mac)
• 12.06.2019 - made command lazier by not requiring user to paste the IP.

Description

Quick bash function to make my life easier when sshing into Rancher nodes. Make sure to update your default web browser download directory by modifying line 2 of the script. For mac: brew install findutils

patrick0057

Overview

When using RKE 0.2.0 and newer, if you have restored a cluster with the incorrect rkestate file you will end up a state where your infrastructure pods will not start. This includes all pods in kube-system, cattle-system and ingress-nginx. As a result of these core pods not starting, all of your workload pods will be unable to function correctly. If you find yourself in this situation you can use the directions below to fix the cluster.

Recovery

1. Delete all service-account-token secrets in kube-system, cattle-system and ingress-nginx namespaces.

{
kubectl get secret -n cattle-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n cattle-system delete secret " $1) }'
kubectl get secret -n kube-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n kube-system delete secret " $1) }'