Upgrading etcd cluster from APIv2 data storage to APIv3 data storage for Kubernetes

etcd_upgrade.md

Checklist for offline etcd v2 API -> v3 API

Adapted from https://coreos.com/etcd/docs/latest/op-guide/v2-migration.html#migrate-data

Note that this strategy results in kube-apiserver downtime, but this should not affect the state of the cluster, it will only prevent changes during the migration

• Ensure there is no v3 data in etcd as this will be overwritten during the migration, verify that the output of the following command is 0:

ETCDCTL_API=3 etcdctl get "" --from-key --keys-only --limit 1 | wc -l

• Stop Chef on the kube-apiservers: sudo service chef-client stop
• Stop apiserver on the kube-apiservers to prevent writing new data during the migration: $ systemctl stop kube-apiserver.service
• Wait for ~1 minute for all state on all etcd nodes to converge
• Verify that node state has convered using by checking that the raft index of all members is the same (or off by 1) beta:

$ ETCDCTL_API=3 etcdctl --endpoints "http://172.20.240.5:2379,http://172.20.240.6:2379,http://172.20.240.7:2379" -w table endpoint status

prod:

$ ETCDCTL_API=3 etcdctl --endpoints "http://172.31.0.96:2379,http://172.31.0.135:2379,http://172.31.0.219:2379" -w table endpoint status

• Stop etcd on every member in the cluster: $ sudo systemctl stop etcd-etcd<#>.service
• Migrate all etcd v2 keys into v3 by executing the following command on every member in the cluster: $ ETCDCTL_API=3 etcdctl migrate --data-dir /var/lib/etcd
• Confirm that keys have been migrated the following command should return a non-zero value:

$ ETCDCTL_API=3 etcdctl get "" --from-key --keys-only --limit 1 | wc -l

• Add the --storage-backend=etcd3 flag to the kube-apiserver via Chef (this defaults to etcd3 in Kubernetes >=v1.6) -- https://github.com/TeachersPayTeachers/chef/pull/3537
• Update the etcd backup script to use ECTDCTL_API=3 -- https://github.com/TeachersPayTeachers/chef/pull/3538
• Start the kube-apiservers back up