fixing bad rke state secrets

extract-cluster-data-from-secret.sh

#!/bin/bash

if [ $# -eq 0 ]; then
	echo "Need the secret name"
	exit 1
fi

secretName="$1"

kubectl get secrets -n cattle-system $secretName -o json | jq -r .data.cluster | base64 --decode | jq . >cluster-state-$secretName.json

steps.md

1. ./extract-cluster-data-from-secret.sh <secret-name> - Decode the data.cluster field from the specified secret that stores the cluster's rke state. Write it to a file cluster-state-<secret-name>.json. Script:
2. vim cluster-state-<secret-name>.json - Edit cluster-state-.json. Change the value of monitor-delay and monitor-timeout from numbers to strings. We will only do this in the metadata.fullState field
3. ./update-secret.sh <secret-name> - Update the secret based on the contents of cluster-state-<secret-name>.json
4. Retrigger a cluster update so that rancher attempts to reprovision. I am not 100% if this will work if we will need to perform an explicit cert rotation

Note: this actually didnt work because the updated secret value was too long. Kept getting this error:

./update-secret.sh: line 18: /usr/bin/kubectl: Argument list too long

Could proably fix by switching from patching to doing a full update, but at this point we determined we could easily update the secret via the Rancher UI for the local cluster.

update-secret.sh

#!/bin/bash

if [ $# -eq 0 ]; then
	echo "Need the secret name"
	exit 1
fi

secretName="$1"

# Written to file just so we have a debug trail
cat cluster-state-$secretName.json | base64 -w 0 >encoded-updated-cluster-state-$secretName

newVal=`cat encoded-updated-cluster-state-$secretName`

kubectl patch -n cattle-system secret $secretName -p '{"data": {"cluster": "'"$newVal"'"}}'