gcp-k8s-api-upgrade-friction.md

friction when upgrading GKE Kubernetes cluster versions

1. You can see update is available in GCP console for the master. Selecting it provides an array of options, including some containing -gke.1.
2. You choose one of those options, await cluster upgrade, and notice your nodes are out of sync. Knowing you don't want nodes to stay out of sync long, you decide to go ahead and take care of it.
3. You discover there is no way to update node version from the console. This is fine, and you find the documentation for the CLI command to do so. You attempt to upgrade it to the same version you chose for your master, including -gke.1. You get a 400 for bad version number.
4. You look up supported versions for master and nodes. -gke.1 is not on the list for nodes.
5. This is fine, you attempt to install the non -gke.1 version to get as close as you can to the master. gcloud tells you that you can only install versions to the node that explicitly match master.
6. You say what in the fuck.
7. You update your master to vanilla version instead of the -gke.1 version (which you assumed was some explicitly curated version for GKE that should be preferable)
8. You run the command gcloud container clusters upgrade squad --cluster-version=v1.7.8 --zone us-central1-a and gcloud can't figure out that v1.7.8 is the same thing as 1.7.8.
9. You finally issue the command for the version minus the "v".
10. You have now managed to update both your master and your nodes API versions and have them in sync.

From gcloud container clusters upgrade --help:

--cluster-version=CLUSTER_VERSION
The Kubernetes release version to which to upgrade the cluster's nodes.

If provided, the --cluster-version must be no greater than the cluster
master's minor version (x.X.x), and must be a latest patch version
(x.x.X).

Should have read the CLI docs instead of web docs, but I had no idea it was optional and I'm guessing it thinks 1.7.6-gke.1 is a greater version than 1.7.6, even though thats not true, and the error message does not say that:

gcloud container clusters upgrade squad --cluster-version=1.7.6 --zone us-central1-a
All nodes (3 nodes) of cluster [squad] will be upgraded from version
[1.6.10] to version [1.7.6]. This operation is long-running and will
block other operations on the cluster (including delete) until it has
run to completion.

Do you want to continue (Y/n)?  Y

ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400, message=auto-upgrade is enabled, nodes can only be upgraded to current master version..