Update certs on old kubeadm clusters

kubeadm-cert-renew-howto.md

Howto renew certs on older kubeadm clusters

There's a bug with old versions of kubeadm, in that it can update certificates, but doesn't correctly "wire them in".

Are you trying to connect to the cluster and getting:

$ kubectl <foo>
Unable to connect to the server: x509: certificate has expired or is not yet valid

The apiserver will continue to use the old cert in cluster made with older versions of kubeadm.

That's the bug that is referenced in the warning block here.

We need to get the apiserver to pick up renewed certs. Here's how to do it...

Regen certs

You can check certificate expirations with this command:

kubeadm alpha certs check-expiration

You'll see the expiry for the various control-plane certs.

Renew them with:

kubeadm alpha certs renew

Kewl! But! That won't actually update the running certs...

Update the kubelet conf

Make a backup of the kubelet.conf, just in case everything goes bust:

cp /etc/kubernetes/kubelet.conf kubelet.conf.bak

Regenerate the kubelet.conf to include the updated certificates:

kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf

Restart the kubelet:

systemctl restart kubelet

Recreate the apiserver container

Restart the apiserver container once kubelet is back up and running:

docker ps | grep apiserver
docker rm -f <apiserver-container>

Kubelet will fire the apiserver back up with the new certs!

Check that your cluster is back up and running:

kubectl get nodes

You may need to refresh the admin config:

cp /etc/kubernetes/admin.conf ~/.kube/config

Do the same for the Controller and Scheduler

The other control plane services are also still running with out-of-date certs.

Restart the scheduler and controller-manager containers as well

docker ps | grep 'scheduler\|controller'
docker rm -f <container-id> <container-id>

Fin.