Skip to content

Instantly share code, notes, and snippets.

@scollier
Created July 17, 2018 17:07
Show Gist options
  • Save scollier/6dcede55ba66540bb0744a866b7f1cd4 to your computer and use it in GitHub Desktop.
Save scollier/6dcede55ba66540bb0744a866b7f1cd4 to your computer and use it in GitHub Desktop.

Recovering ETCD

Creating a RW single node cluster: On the etcd node that you did not remove from the cluster, stop all etcd services:

# systemctl stop etcd.service

Add the following line to the etcd.conf file

# echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf

Restart the services

# systemctl restart etcd.service

Remove the line that was added above

# sed -i '/^ETCD_FORCE_NEW_CLUSTER=/d' /etc/etcd/etcd.conf

Once the two lost nodes are recovered

Add them back with etcdctl

# etcdctl -C https://192.168.1.2:2379 --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt member add master2.example.com https://192.168.1.2:2380 2>/dev/null | grep '^ETCD_INITIAL_CLUSTER='

Add the name back to the etcd.conf file:

# sed -i '/^ETCD_NAME=/d' /etc/etcd/etcd.conf
# echo "ETCD_NAME=master2" >> /etc/etcd/etcd.conf

The returned value from the above command should be added to the etcd.conf file for that node:

# sed -i '/^ETCD_INITIAL_CLUSTER=/d' /etc/etcd/etcd.conf
# echo "ETCD_INITIAL_CLUSTER=$OUTPUT" >> /etc/etcd/etcd.conf

Lastly, tell the nodes that they are joining an existing cluster

# sed -i '/^ETCD_INITIAL_CLUSTER_STATE=/d' /etc/etcd/etcd.conf
echo "ETCD_INITIAL_CLUSTER_STATE=existing" >> /etc/etcd/etcd.conf
systemctl restart etcd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment