Skip to content

Instantly share code, notes, and snippets.

@mclarke47
Created June 27, 2017 13:09
Show Gist options
  • Save mclarke47/77b513d61548dbbd319ca15eef8bf414 to your computer and use it in GitHub Desktop.
Save mclarke47/77b513d61548dbbd319ca15eef8bf414 to your computer and use it in GitHub Desktop.
kubernetes controller rolling reboot
#!/usr/bin/env bash
kubectl get nodes -o=json | jq -c -r '.items | map(select(.spec | has("taints"))) | map(select(.spec.taints[].value == "master")) | map(select(.spec.taints[].key == "node.alpha.kubernetes.io/role")) | .[]' | while read object; do
NODE_NAME=$(echo "$object" | jq -r .metadata.name)
INSTANCE_ID=$(echo "$object" | jq -r .spec.externalID)
echo "rebooting node=$NODE_NAME instance-id=$INSTANCE_ID"
kubectl drain ${NODE_NAME} --ignore-daemonsets --force
aws ec2 reboot-instances --instance-ids ${INSTANCE_ID}
sleep 30
echo "Node rebooted, waiting for node to be marked as 'Ready'"
END=0
while [ ${END} -eq 0 ]; do
RESULT=$(kubectl get nodes | grep ${NODE_NAME})
if [ $? -eq 0 ]; then
if echo "$RESULT" | grep ${NODE_NAME} | grep -q "(TLS|EOF|NotReady)"; then
X=1
else
echo "${NODE_NAME} is marked as Ready"
END=1
fi
else
echo "$RESULT"
fi
echo "${NODE_NAME} NotReady"
sleep 10
done
kubectl uncordon ${NODE_NAME}
done
@mclarke47
Copy link
Author

Kinda gross and needs cleaned up but seems to work for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment