Recovering ETCD
Creating a RW single node cluster: On the etcd node that you did not remove from the cluster, stop all etcd services:
# systemctl stop etcd.service
Add the following line to the etcd.conf file
# echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
Restart the services
# systemctl restart etcd.service
Remove the line that was added above
# sed -i '/^ETCD_FORCE_NEW_CLUSTER=/d' /etc/etcd/etcd.conf
Once the two lost nodes are recovered
Add them back with etcdctl
# etcdctl -C https://192.168.1.2:2379 --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt member add master2.example.com https://192.168.1.2:2380 2>/dev/null | grep '^ETCD_INITIAL_CLUSTER='
Add the name back to the etcd.conf file:
# sed -i '/^ETCD_NAME=/d' /etc/etcd/etcd.conf
# echo "ETCD_NAME=master2" >> /etc/etcd/etcd.conf
The returned value from the above command should be added to the etcd.conf file for that node:
# sed -i '/^ETCD_INITIAL_CLUSTER=/d' /etc/etcd/etcd.conf
# echo "ETCD_INITIAL_CLUSTER=$OUTPUT" >> /etc/etcd/etcd.conf
Lastly, tell the nodes that they are joining an existing cluster
# sed -i '/^ETCD_INITIAL_CLUSTER_STATE=/d' /etc/etcd/etcd.conf
echo "ETCD_INITIAL_CLUSTER_STATE=existing" >> /etc/etcd/etcd.conf
systemctl restart etcd