Distributes updated x509 tls certs to etcd2 kube nodes and resolves outdated cert problems

dist-upated-tls-certs-etcd-nodes.sh

#!/usr/bin/env bash
#
# Summary:
# Distributes updated x509 tls certs and resolves outdate cert problems which effectively kill your kube cluster
#   ref: https://github.com/kubernetes-incubator/kube-aws/issues/1132
#   ref: https://github.com/kubernetes-incubator/kube-aws/issues/1057
#
# NOTES: Ensure this is the correct process for your etcd2 kube cluster before using.
#       Test on a dev/test cluster first.
#       Use at own risk.
#
# OBSERVATIONS: 
# Output of etcdctl cluster-health shows x509 cert errors:
#   etcdctl cluster-health
#   cluster may be unhealthy: failed to list members
#   Error:  client: etcd cluster is unavailable or misconfigured
#   error #0: x509: certificate has expired or is not yet valid
#
# TESTED/VALIDATED ON BELOW VERSION:
# KUBE-AWS-VERSION: v0.9.3-rc.5
# AMI-ID: ami-b31d43d3
#
# STEPS:
#
# 1. Follow your kube provisioning process to update your certificates
#       for kube-aws this appears to be a moving target for future releases but older releases were essentially
#         kube-aws render credentials
#         kube-aws update
#
# 2. Modify this script to include all of your etcd2 nodes IP addresses
#
# 3. Either cd into the updated $kube_cluster/credentials directory or modify path if desired
#
# 4. Execute this script with no args

# scp updated certs to etcd2 nodes
for i in ca.pem  ca.pem.enc  etcd-client-key.pem  etcd-client-key.pem.enc  etcd-client.pem  etcd-client.pem.enc  etcd-key.pem  etcd-key.pem.enc  etcd.pem  etcd.pem.enc ; do 
    scp $i core@10.1.12.4:/var/tmp 
    scp $i core@10.1.13.4:/var/tmp
    scp $i core@10.1.12.5:/var/tmp
    # add additional etcd2 nodes here
done

# move to correct locations and set requisite perms, etc.
ssh core@10.1.12.4 "sudo mv /var/tmp/*.pem /etc/etcd2/ssl; sudo mv /var/tmp/*.enc /etc/etcd2/ssl; sudo chown root:root /etc/etcd2/ssl/*; sudo systemctl restart etcd2; sudo systemctl status etcd2"
ssh core@10.1.13.4 "sudo mv /var/tmp/*.pem /etc/etcd2/ssl; sudo mv /var/tmp/*.enc /etc/etcd2/ssl; sudo chown root:root /etc/etcd2/ssl/*; sudo systemctl restart etcd2; sudo systemctl status etcd2"
ssh core@10.1.12.5 "sudo mv /var/tmp/*.pem /etc/etcd2/ssl; sudo mv /var/tmp/*.enc /etc/etcd2/ssl; sudo chown root:root /etc/etcd2/ssl/*; sudo systemctl restart etcd2; sudo systemctl status etcd2"
# add additional etcd2 nodes here

have you tried this script with etcd3? kube-aws now ships with etcd 3.x

your api-server,pem will also expire. you need to be rotating that as well. What is the method you tried

for those coming after, my /etc/etcd-environment had a number of arguments that specified the /etc/ssl/certs directory as the place needed for those values