Rolling restart of all nodes in an AKS cluster

aks-restart-nodes.sh

#!/bin/bash

set -e

resourceGroupDefault='<set your default here, to avoid having to specify in the common case>'
resourceGroup=${RESOURCE_GROUP:-$resourceGroupDefault}
clusterNameDefault='<set your default here>'
clusterName=${CLUSTER_NAME:-$clusterNameDefault}
regionDefault='<set your default here>'
region=${REGION:-$regionDefault}
force=false
dryrun=false
nodes=''

function print_usage() {
  echo "Usage: $0 [<options>]"
  echo ""
  echo "-n|--node <node>                    The name of a node to restart."
  echo "                                    By default, a rolling restart of all nodes"
  echo "                                    is performed."
  echo ""
  echo "--resource-group <group-name>       The resource group of the cluster."
  echo "                                    Can also be set by RESOURCE_GROUP"
  echo "                                    Default: $resourceGroupDefault"
  echo ""
  echo "--cluster-name <cluster-name>       The name of the cluster."
  echo "                                    Can also be set by CLUSTER_NAME"
  echo "                                    Default: $clusterNameDefault"
  echo ""
  echo "--region <azure-region>             The Azure region in which the cluster is."
  echo "                                    Can also be set by REGION"
  echo "                                    Default: $regionDefault"
  echo ""
  echo "-f|--force                          Restart node(s) without first draining."
  echo "                                    Useful if draining a node fails."
  echo ""
  echo "-d|--dry-run                        Just print what to do; don't actually do it"
  echo ""
  echo "-h|--help                           Print usage and exit."
}

while [[ $# -gt 0 ]]
do
  key="$1"

  case $key in
    -n|--node)
    node="$2"
    shift
    shift
    ;;
    --resource-group)
    resourceGroup="$2"
    shift
    shift
    ;;
    --cluster-name)
    clusterName="$2"
    shift
    shift
    ;;
    --region)
    region="$2"
    shift
    shift
    ;;
    -f|--force)
    force=true
    shift
    ;;
    --dry-run)
    dryrun=true
    shift
    ;;
    -h|--help)
    print_usage
    exit 0
    ;;
    *)
    print_usage
    exit 1
    ;;
  esac
done

group="MC_${resourceGroup}_${clusterName}_$region"

function wait_for_status() {
  node=$1
  reason=$2
  i=0
  while [[ $i -lt 30 ]]; do
    status=$(kubectl get node $node -o "jsonpath={.status.conditions[?(.reason==\"$reason\")].type}")
    if [[ "$status" == "Ready" ]]; then
      echo "$reason after $((i*2)) seconds"
      break;
    else
      sleep 2s
      i=$(($i+1))
    fi
  done
  if [[ $i == 30 ]]; then
    echo "Error: Did not reach $reason state within 1 minute"
    exit 1
  fi
}

if [ -z "$node" ]; then
  nodes=$(kubectl get nodes -o jsonpath={.items[*].metadata.name})
else
  nodes="$node"
fi

for node in $nodes; do
  if $force; then
    echo "WARNING: --force specified, restarting node $node without draining first"
    if $dryrun; then
      echo "kubectl cordon $node"
    else
      kubectl cordon "$node"
    fi
  else
    echo "Draining $node..."
    if $dryrun; then
      echo "kubectl drain $node --ignore-daemonsets --delete-local-data"
    else
      kubectl drain "$node" --ignore-daemonsets --delete-local-data
    fi
  fi

  echo "Initiating VM restart for $node..."
  if $dryrun; then
    echo "az vm restart --resource-group $group --name $node"
  else
    az vm restart --resource-group "$group" --name "$node"
  fi

  if ! $dryrun; then
    echo "Waiting for $node to start back up..."
    wait_for_status $node KubeletNotReady
    wait_for_status $node KubeletReady
  fi

  echo "Re-enabling $node for scheduling"

  if $dryrun; then
    echo "kubectl uncordon $node"
  else
    kubectl uncordon "$node"
  fi
done

This script seems to assume something things will always succeed (like kubectl drain), as there is no set -e and/or no checking of return codes from the external executables.

(BTW, thanks for this. It's a great kick start!)

@bradwilson Yeah, this is a pretty early sketch of what we eventually ended up with - if I remember, I can check back on Monday when I'm back from vacation and see if I can find the current version of the script and update this gist. Ping me if you want it and haven't heard anything mid-week!

@bradwilson didn't ping you, but I am :-). May I kindly ask for an update, @tomasaschan?

@AmanG2205 I've updated the gist with the last edition of the script checked into that project's git repo. However, it's been almost a year since I did any work on that project - and therefore, since I ran the script last - so I leave no guarantees that it still works without modification.

Also, since the script does a rolling restart, note that you need to have one extra node's worth of capacity to be able to pull it off. Room for improvement for the script would be to do something like this:

1. detect whether auto-scaling is enabled, and turn it off if it is
2. scale up by one node
3. do the rolling restart as currently implemented
4. turn on auto-scaling again if it was on before (it will scale down again appropriately on its own) or scale down by one node if autoscaling was not enabled

I tried this and it seems KubeletNotReady either never happens or goes by too quickly in recent AKS. Commenting that wait_for_status line out made this gold. Thanks!

Do not do this as it's not supported by MS if something goes wrong. MS now has a start/stop feature. Utilize az aks start/stop.

https://docs.microsoft.com/en-us/azure/aks/start-stop-cluster

@gvanriper Note that this script does a rolling restart, i.e. with zero downtime. I assume az aks stop followed by az aks start would result in downtime while the cluster is stopped?

Yes, this is a hack, and it's by no means supported in any way by Microsoft. It's also 18 months old, haven't been used (at least by me) since, and was created to solve an instability in AKS at the time, that I sincerely hope is no longer a problem (my nodes were forgetting to share networking capabilities with my containers, so the containers became impossible to reach but the cluster did not detect them as unhealthy).

@tomasaschan, you're correct that it would result in downtime. I just wanted anyone following this that hadn't noticed to know that AKS start/stop is now an option and supported by MS.