cert rotate

gistfile1.txt

In Rancher 2.0 and 2.1, the auto generated certificates for Rancher provisioned clusters have 1 year of expiry. It means if you created a Rancher provisioned cluster about 1 year ago, you need to rotate the certificates, otherwise the cluster will go into a bad state when the certificates expire. It is better to rotate the certificates before they expire. The rotation is a one time operation as the newly generated certs will be valid for the next 10 years.


Rancher v2.2.4 provides UI support for certificate rotation. If upgrading your 2.0.x or 2.1.x clusters to 2.2.x is not an option, you can upgrade them to 2.0.15 and 2.1.10 respectively. These versions have a certificate rotation support via API (more instructions are here https://rancher.com/docs/rancher/v2.x/en/cluster-admin/certificate-rotation/#certificate-rotation-in-rancher-v2-1-x-and-v2-0-x)


Steps to rotate certs on a working cluster which certificates haven't expired yet
================================================================================

Here is the list of steps needed to safely rotate the certificates on a cluster which certificates are still valid:

1) As a preliminary step, update your cluster so it goes through Rancher Kubernetes Engine provisioning process that refreshes the cluster state and configs. For that, you can either upgrade your cluster to the newer k8s version, or simply change one of the existing parameters on a cluster to trigger the cluster reconcile. The latter can be done via `Edit -> Edit as YAML -> change addon_job_timeout to 50 -> click Save`
2) If you are on 2.2.4, rotate certificates from the UI by clicking verticle ellipses option -> "Rotate Certificates" -> "Rotate all service certificates" -> Save. If you are on 2.0.15 or 2.1.10, do the same from API (https://rancher.com/docs/rancher/v2.x/en/cluster-admin/certificate-rotation/#certificate-rotation-in-rancher-v2-1-x-and-v2-0-x)

As the result, the certificates will be rotated, and the cluster would be fine using them for the next 10 years.


Steps to rotate certs on a cluster which certificates are already expired
=======================================================================

If your cluster is already in a bad sitation where the certificates have expired for them, the safest option would be to:

1) Update your setup to 2.2.4
2) Login to your etcd and control plane nodes and check if `/etc/kubernetes/.tmp` has a `kube-apiserver-requestheader-ca.pem` present. If not, perform the following manual copy:

```
cp kube-ca.pem kube-apiserver-requestheader-ca.pem
cp kube-ca-key.pem kube-apiserver-requestheader-ca-key.pem
cp kube-apiserver.pem kube-apiserver-proxy-client.pem
cp kube-apiserver-key.pem kube-apiserver-proxy-client-key.pem
```

3) Call Rotate ertificates from the UI. If you don't see any activity on the cluster while the rotation is going on, and the log is still saying "Expired cert", perform the steps described in this issue https://github.com/rancher/rancher/issues/20822
4) After the rotation is finished, go to the Nodes view for your cluster, and check the state of Worker only nodes. If the state is not Active, do the following:

* Copy the following certificates from a Kubernetes control plane node to each worker node, under the same location:

/etc/kubernetes/ssl/kube-node.pem /etc/kubernetes/ssl/kube-proxy.pem

* Restart kubelet and proxy containers

If that doesn't bring your cluster back to an active state, contact Rancher support.


UPDATE: if you are running Rancher in HA mode, and used RKE - Rancher Kontainer Engine - of version less than v0.2.0 to provision the cluster where Rancher server gets installed using Helm, the certificates on that management cluster have to be rotated using RKE CLI. Here are the steps:


```
Steps to rotate certs on a working cluster which certificates haven't expired yet
============================================

* Upgrade RKE CLI to the latest version of v0.2.x
* Run rke up to refresh your cluster.
* Rotate certificate using the following command:

./rke cert rotate --config cluster.yml


Steps to rotate certs on a cluster which certificates are already expired
==========================================================
* Upgrade RKE CLI to the latest version of v0.2.x
* Rotate certificate using the following command:

./rke cert rotate --config cluster.yml

```