Check certificate state on Rancher v2.2 clusters

check-certificate-state-rancherv22.md

Check certificate state on Rancher v2.2 clusters

Steps to verify certificate state for custom clusters in v2.2

Check certificates locally on nodes

Run script below to output certificate checksums for certificates present on the node.

#!/bin/sh
if [ -d /opt/rke/etc/kubernetes ]; then
  K8S_DIR="/opt/rke/etc/kubernetes"
else
  K8S_DIR="/etc/kubernetes"
fi

for cert in `find "${K8S_DIR}/ssl" -type f -name *.pem | grep -v "\-key\.pem$" | grep -v kube-admin | sort`; do
        md5sum $cert
done

if [ -d "${K8S_DIR}/.tmp" ]; then
  for cert in `find "${K8S_DIR}/.tmp" -type f -name *.pem | grep -v "\-key\.pem$" | grep -v kube-admin | sort`; do md5sum $cert; done
fi

if command -v openssl >/dev/null 2>&1; then
    if [ -f "${K8S_DIR}/ssl/kube-ca.pem" ]; then
        for cert in `find "${K8S_DIR}/ssl" -type f -name *.pem | grep -v "\-key\.pem$" | grep -v kube-admin | sort`; do
                if  [ "$(basename $cert)" = "kube-apiserver-proxy-client.pem" ] || [ "$(basename $cert)" = "kube-apiserver-requestheader-ca.pem" ] || [ "$(basename $cert)" = "kube-ca.pem" ]; then
                        continue
                fi
                SSL_CERT_DIR=/dummy
                SSL_CERT_FILE=/dummy
                openssl verify -CAfile "${K8S_DIR}/ssl/kube-ca.pem" $cert
                openssl x509 -in $cert -noout -dates
        done
    fi
fi

find "${K8S_DIR}/ssl" -type f -name *.pem -exec ls -la {} \; | sort

Check certificates stored in the custom cluster itself

Can be run in the embedded kubectl for the custom cluster

Checksums

for key in $(kubectl -n kube-system get configmap full-cluster-state -o json | jq -r '.data["full-cluster-state"]' | jq -r . | jq -r '.currentState.certificatesBundle | to_entries[] | .key'); do echo $(kubectl -n kube-system get configmap full-cluster-state -o json | jq -r '.data["full-cluster-state"]' | jq -r . | jq -r --arg key $key '.currentState.certificatesBundle[$key].certificatePEM' | head -n -1 | md5sum) $key; done

Compare certificatesBundle

Compare currentState vs desiredState

for key in $(kubectl -n kube-system get configmap full-cluster-state -o json | jq -r '.data["full-cluster-state"]' | jq -r . | jq -r '.currentState.certificatesBundle | to_entries[] | .key'); do CURRENTBASE64=$(kubectl -n kube-system get configmap full-cluster-state -o json | jq -r '.data["full-cluster-state"]' | jq -r . | jq -r --arg key $key '.currentState.certificatesBundle[$key].certificatePEM | @base64'); DESIREDBASE64=$(kubectl -n kube-system get configmap full-cluster-state -o json | jq -r '.data["full-cluster-state"]' | jq -r . |  jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM | @base64'); if [ $CURRENTBASE64 != $DESIREDBASE64 ]; then echo "$key: WRONG"; else echo "$key: CORRECT"; fi; done

Check certificates stored for cluster in Rancher

This can be run in the local cluster in case of HA Rancher or kubeconfig generated by RKE.

Needed: cluster id for the cluster (c-xxxxx)

metadata Certs

for cert in $(docker exec $CONTID kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .data.cluster  | base64 -d | jq -r '.metadata.Certs' | jq -r 'keys[]' | grep -v kube-admin | sort); do echo $(docker exec $CONTID kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .data.cluster  | base64 -d | jq -r '.metadata.Certs' | jq -r '.["'"$cert"'"].CertPEM' | head -n -1 | md5sum) $cert; done

Show currentState vs desiredState

CONTID=$(docker ps | grep -E "rancher/rancher:|rancher/rancher |rancher/rancher@|rancher_rancher" | awk '{ print $1 }')
CLUSTERID=c-xxxxx
for key in $(docker exec $CONTID kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .data.cluster  | base64 -d | jq -r '.metadata.fullState' | jq -r '.currentState.certificatesBundle' | jq -r 'keys[]' | grep -v kube-admin | sort); do echo $(docker exec $CONTID kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .data.cluster  | base64 -d | jq -r '.metadata.fullState' | jq -r --arg key $key '.currentState.certificatesBundle[$key].certificatePEM' | head -n -1 | md5sum) $key currentState; echo $(docker exec $CONTID kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .
data.cluster  | base64 -d | jq -r '.metadata.fullState' | jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM' | head -n -1 | md5sum) $key desiredState; done

cluster.rkestate (RKE only)

currentState

Linux

for key in $(cat cluster.rkestate | jq -r '.currentState.certificatesBundle | keys[]'); do echo $(cat cluster.rkestate | jq -r --arg key $key '.currentState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5sum) $key; done

MacOS

for key in $(cat cluster.rkestate | jq -r '.currentState.certificatesBundle | keys[]'); do echo $(cat cluster.rkestate | jq -r --arg key $key '.currentState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5) $key; done

desiredState

Linux

for key in $(cat cluster.rkestate | jq -r '.desiredState.certificatesBundle | keys[]'); do echo $(cat cluster.rkestate | jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5sum) $key; done

MacOS

for key in $(cat cluster.rkestate | jq -r '.desiredState.certificatesBundle | keys[]'); do echo $(cat cluster.rkestate | jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5) $key; done