cert rotate

gistfile1.txt

---
title: "Kubernetes certificate expiry and rotation in Rancher Kubernetes clusters"
author: alex-seymour
date: 2019-06-14
description: "This guide details how to rotate certificates for Rancher launched, and Rancher Kubernetes Engine CLI provisioned, Kubernetes clusters, both before expiry when certificates are still valid, and also in the event that the certificates have already expired."
type: "blog"
tags: [Kubernetes, RKE, Rancher, Certificates]
categories: [blog]
image: "/img/featured-images/featured-images_security.png"
URL: /blog/2019/kubernetes-certificate-expiry-and-rotation-in-rancher-kubernetes-clusters
---

## Introduction

Kubernetes clusters use multiple certificates, to provide both encryption of traffic to the Kubernetes components as well as authentication of these requests. These certificates are auto-generated in [Rancher launched](https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/), and [Rancher Kubernetes Engine (RKE) CLI](https://rancher.com/docs/rke/latest/en/) provisioned, Kubernetes clusters.

In Rancher v2.0.x and v2.1.x, the auto-generated certificates for [Rancher launched Kubernetes clusters](https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/) have a validity period of 1 year, meaning these certificates will expire 1 year after the cluster is provisioned. The same applies to Kubernetes clusters provisioned via the [Rancher Kubernetes Engine (RKE) CLI](https://rancher.com/docs/rke/latest/en/) v0.1.x.

This means that if you created a Rancher launched, or RKE CLI provisioned, Kubernetes cluster about 1 year ago, you need to rotate the certificates. If no action is taken, when the certificates expire, the cluster will go into an error state and the Kubernetes API for the cluster will become unavailable. It is better to rotate the certificates before they expire to avoid an interruption in cluster availability. The rotation is a one time operation, as the newly generated certificates will be valid for the next 10 years.

The instructions below detail how to rotate the certificates in Rancher launched and RKE CLI provisioned clusters, both before expiry when certificates are still valid, and also in the event that the certificates have already expired.

**Rotating Kubernetes certificates may result in your cluster being temporarily unavailable as components are restarted. For production environments, it’s recommended to perform this action during a maintenance window.**

## Rotating Kubernetes certificates for Rancher launched Kubernetes clusters

Rancher v2.2.4 provides UI support for certificate rotation. If upgrading your Rancher v2.0.x or v2.1.x instances to v2.2.x is not an option, you can upgrade them to v2.0.15 and v2.1.10 respectively. These versions contain certificate rotation support via the API, and detailed steps for this can be found in the [documentation here](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/certificate-rotation/#certificate-rotation-in-rancher-v2-1-x-and-v2-0-x).


### Steps to rotate certificates on a working cluster for which the certificates are valid and have not expired yet

To rotate the certificates on a Rancher launched cluster for which certificates are still valid, follow these steps:

1. As a preliminary step, update your cluster so it goes through Rancher Kubernetes Engine (RKE) provisioning process that refreshes the cluster state and configurations. For that, you can either upgrade your cluster to a newer Kubernetes version, or simply change one of the existing parameters on a cluster to trigger the cluster reconciliation process via RKE.
  -  To upgrade the Kubernetes version, browse to the cluster in the Rancher UI, click the vertical ellipses, and click `Edit`, select the newer `Kubernetes Version` under `Cluster Options` and click `Save`.
  - To trigger reconciliation by changing a parameter with minimal impact, browse to the cluster in the Rancher UI, click the vertical ellipses and click `Edit`, then click `Edit as YAML` update `change addon_job_timeout` to `50`, and click `Save`.
2. Rotate the certificates:
  - Rancher v2.2.4: If you are running Rancher v2.2.4, rotate certificates from the UI, to do so browse to the cluster in the Rancher UI, click the vertical ellipses, click `Rotate Certificates`, select `Rotate all service certificates` and click `Save`.
  - Rancher v2.0.15 or v2.1.10: If you are running Rancher v2.0.15 or v2.1.10, perform the certificate rotation from the API, per the [documentation here](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/certificate-rotation/#certificate-rotation-in-rancher-v2-1-x-and-v2-0-x).

Following these steps, the certificates will be rotated, with a validity of 10 years.

### Steps to rotate certificates on a cluster for which the certificates are already expired

If your Rancher launched Kubernetes cluster is already in an error state, in which the certificates have expired, follow these steps to rotate the certificate:

1. [Upgrade your Rancher](https://rancher.com/docs/rancher/v2.x/en/upgrades/upgrades/) instance to v2.2.4
2. Open a shell session to the etcd and control plane nodes for the cluster and check if the directory `/etc/kubernetes/.tmp` contains the file `kube-apiserver-requestheader-ca.pem`. If this file is absent, perform the following manual copy:

  ```
  cp /etc/kubernetes/.tmp/kube-ca.pem /etc/kubernetes/.tmp/kube-apiserver-requestheader-ca.pem
  cp /etc/kubernetes/.tmp/kube-ca-key.pem /etc/kubernetes/.tmp/kube-apiserver-requestheader-ca-key.pem
  cp /etc/kubernetes/.tmp/kube-apiserver.pem /etc/kubernetes/.tmp/kube-apiserver-proxy-client.pem
  cp /etc/kubernetes/.tmp/kube-apiserver-key.pem /etc/kubernetes/.tmp/kube-apiserver-proxy-client-key.pem
```

3. Rotate certificates from the UI, to do so browse to the cluster in the Rancher UI, click the vertical ellipses, click `Rotate Certificates`, select `Rotate all service certificates` and click `Save`.
4. If the UI shows no activity on the cluster while the rotation is going on, and the log still reports `Expired cert`, perform the steps described in [Rancher Issue #20822](https://github.com/rancher/rancher/issues/20822).
5. After the rotation is finished, browse to the `Nodes` view for the cluster within the Rancher UI, and check the state of `Worker` only nodes. If the state is not `Active`, do the following:
  - Copy the following certificates from a Kubernetes control plane node to each worker node, under the same location:
  ```
  /etc/kubernetes/ssl/kube-node.pem
  /etc/kubernetes/ssl/kube-proxy.pem
  ```
  - Restart the `kubelet` and `kube-proxy` containers on each worker:
  ```
  docker restart kubelet
  docker restart kube-proxy
  ```


## Rotating Kubernetes certificates for clusters provisioned via the Rancher Kubernetes Engine (RKE) CLI

If you are running Rancher in High Availability (HA) mode, and used RKE of version less than v0.2.0 to provision the cluster where Rancher server gets installed using Helm, the certificates on that management cluster have to be rotated using the RKE CLI. Here are the steps:

### Steps to rotate certificates on a working cluster for which the certificates are valid and have not expired yet

To rotate the certificates on an RKE v0.1.x provisioned cluster for which certificates are still valid, follow these steps:

1. Upgrade the RKE CLI to the latest version of v0.2.x, the RKE releases and downloads can be found [here on GitHub](https://github.com/rancher/rke/releases/).
2. Run `rke up --config cluster.yml` to refresh your cluster. **N.B. Please ensure that both your cluster.yml configuration file and the kube_config_cluster.yml file are present in the working directory when invoking RKE**
3. Rotate certificate using the following command: `rke cert rotate --config cluster.yml`

### Steps to rotate certificates on a cluster for which the certificates are already expired

If your RKE v0.1.x provisioned cluster is already in an error state, in which the certificates have expired, follow these steps to rotate the certificate:

1. Upgrade the RKE CLI to the latest version of v0.2.x, the RKE releases and downloads can be found [here on GitHub](https://github.com/rancher/rke/releases/).
2. Rotate certificate using the following command: `rke cert rotate --config cluster.yml`  **N.B. Please ensure that both your cluster.yml configuration file and the kube_config_cluster.yml file are present in the working directory when invoking RKE**