Redeploy OpenShift 3.11 certs for environment provisioned using https://github.com/jaredhocutt/openshift-provision

redeploy-certs.md

SSH into your master:

sudo ./op.py --env-file vars/aws.env --vars-file vars/aws.yml ssh

Become the root user:

sudo -i

Renew the certificate with your additional wildcard domain (fill in the environment variables):

export AWS_ACCESS_KEY_ID=ABCDEFG
export AWS_SECRET_ACCESS_KEY=abcdedf
export CERT_EMAIL=email@example.com
export OPENSHIFT_PUBLIC_HOSTNAME=ocp.example.io
export EXTRA_WILDCARD="*.otherapps.ocp.example.io"

docker run --rm --name certbot \
   -v "/etc/letsencrypt:/etc/letsencrypt:z" \
   -v "/var/lib/letsencrypt:/var/lib/letsencrypt:z" \
   -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
   -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
   certbot/dns-route53 certonly \
   --non-interactive \
   --agree-tos \
   --email "${CERT_EMAIL}" \
   --dns-route53 \
   --dns-route53-propagation-seconds 45 \
   --server https://acme-v02.api.letsencrypt.org/directory \
   --domain "${OPENSHIFT_PUBLIC_HOSTNAME}" \
   --domain "*.apps.${OPENSHIFT_PUBLIC_HOSTNAME}" \
   --domain "${EXTRA_WILDCARD}" \
   --expand

Double check that your symlinks point to the new certs:

ls -lah /etc/letsencrypt/live/${OPENSHIFT_PUBLIC_HOSTNAME}

total 4.0K
drwxr-xr-x+ 2 root root  93 Aug  9 16:10 .
drwxr-x---+ 3 root root  53 Apr 11 19:22 ..
lrwxrwxrwx. 1 root root  49 Aug  9 16:10 cert.pem -> ../../archive/ocp311.jared.redhatgov.io/cert3.pem
lrwxrwxrwx. 1 root root  50 Aug  9 16:10 chain.pem -> ../../archive/ocp311.jared.redhatgov.io/chain3.pem
lrwxrwxrwx. 1 root root  54 Aug  9 16:10 fullchain.pem -> ../../archive/ocp311.jared.redhatgov.io/fullchain3.pem
lrwxrwxrwx. 1 root root  52 Aug  9 16:10 privkey.pem -> ../../archive/ocp311.jared.redhatgov.io/privkey3.pem
-rw-r-xr--+ 1 root root 692 Apr 11 19:22 README

Give ec2-user permissions to read the certs:

setfacl --modify u:ec2-user:rx --recursive --logical /etc/letsencrypt

Exit the root user and go back to being the ec2-user:

exit

Run the redeploy-certificates.yml playbook (this takes about 12 minutes):

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/redeploy-certificates.yml -v

Hi Jared,
Suggested addition to these instructions, see https://access.redhat.com/solutions/4006641 for more info.

Add troubleshooting for "Openshift redeploy certificates playbooks fail when expiration date is less than 365 days" failure.

The Openshift redeploy certificates playbook fails when expiration date of the current certificates is equal or less than 365 days, preventing the certificates dates to be extended. There are two options to resolve this which are provided in the link above.

Here's what the error looks like for reference:

"TASK [openshift_certificate_expiry : Fail when certs are near or already expired] *************************************************************************
Wednesday 06 November 2019 16:41:03 +0000 (0:00:01.017) 0:01:06.312 ****
fatal: [ip-172-31-66-172.ec2.internal]: FAILED! => {
"changed": false
}

MSG:

Cluster certificates found to be expired or within 365 days of expiring. You may view the report at /home/ec2-user/cert-expiry-report.20191106T164057.html or /home/ec2-user/cert-expiry-report.20191106T164057.json."