Skip to content

Instantly share code, notes, and snippets.

@abutcher
Last active April 23, 2020 16:31
  • Star 7 You must be signed in to star a gist
  • Fork 9 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save abutcher/2e13e963a6c241cc5e90 to your computer and use it in GitHub Desktop.

OpenShift Named Certificates

This is a short guide explaining how to deploy and manage custom SNI or "named" certificates via openshift-ansible. These custom certificates will be served for public facing console and API.

1. Setting up

NOTE: I'm running ansible from the first master so I'm creating the certificates in /root/ on my first master. We will use openshift-ansible to deploy certificates to masters and certificates will be ultimately stored within /etc/origin/master/named_certificates/ on masters. I could create these named certificates on my local system if I intended to use my local system as the ansible control host.

Create CA Certificate

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -subj '/C=US/ST=North Carolina/L=Raleigh/O=Red Hat, Inc./OU=OpenShift/CN=public-example.com/emailAddress=none@public-example.com/' -out rootCA.pem

Create master.public-example.com.crt

openssl genrsa -out master.public-example.com.key 2048
openssl req -new -key master.public-example.com.key -subj '/C=US/ST=North Carolina/L=Raleigh/O=Red Hat, Inc./OU=OpenShift/CN=master.public-example.com/emailAddress=none@public-example.com/' -out master.public-example.com.csr
openssl x509 -req -in master.public-example.com.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out master.public-example.com.crt -days 500

Create wildcard-flibberty-jibbet.com.crt

openssl genrsa -out wildcard-flibberty-jibbet.com.key 2048
openssl req -new -key wildcard-flibberty-jibbet.com.key -subj '/C=US/ST=North Carolina/L=Raleigh/O=Red Hat, Inc./OU=OpenShift/CN=*.flibberty-jibbet.com/emailAddress=none@flibberty-jibbet.comm/' -out wildcard-flibberty-jibbet.com.csr
openssl x509 -req -in wildcard-flibberty-jibbet.com.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out wildcard-flibberty-jibbet.com.crt -days 500

Modify /etc/hosts

192.168.122.224 master.public-example.com master.flibberty-jibbet.com

2. Install using master.public-example.com certificate

Create an HA OpenShift cluster and specify the openshift_master_named_certificates inventory variable. Modify paths to match the certificate paths on the system where you will be running ansible and where the certificates were created.

openshift_master_named_certificates=[{"certfile": "/root/master.public-example.com.crt", "keyfile": "/root/master.public-example.com.key"}]

Verify Configuration for master.public-example.com

Ensure certificates exist in /etc/origin/master/named_certificates/

$ ls /etc/origin/master/named_certificates/
master.public-example.com.crt  master.public-example.com.key

Verify master configuration

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
  namedCertificates:
  - certFile: /etc/origin/master/named_certificates/master.public-example.com.crt
    keyFile: /etc/origin/master/named_certificates/master.public-example.com.key
    names:
    - "master.public-example.com"

Ensure API serves the default certificate

$ curl -vk https://192.168.122.224:8443
* About to connect() to 192.168.122.224 port 8443 (#0)
*   Trying 192.168.122.224...
* Connected to 192.168.122.224 (192.168.122.224) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*       subject: CN=172.30.0.1
*       start date: Nov 17 00:53:36 2015 GMT
*       expire date: Nov 16 00:53:37 2017 GMT
*       common name: 172.30.0.1
*       issuer: CN=openshift-signer@1447721613
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.122.224:8443
> Accept: */*

Ensure API serves certificate for master.public-example.com

$ curl -vk https://master.public-example.com:8443
* About to connect() to master.public-example.com port 8443 (#0)
*   Trying 192.168.122.224...
* Connected to master.public-example.com (192.168.122.224) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*       subject: E=none@public-example.com,CN=master.public-example.com,OU=OpenShift,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
*       start date: Nov 17 00:22:26 2015 GMT
*       expire date: Mar 31 00:22:26 2017 GMT
*       common name: master.public-example.com
*       issuer: E=none@public-example.com,CN=public-example.com,OU=OpenShift,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: master.public-example.com:8443
> Accept: */*

3. Re-run ansible with wildcard-flibberty-jibbet.com certificate

When we re-run ansible with different openshift_master_named_certificates the original certificates should remain in place and they should also remain configured.

Modify host inventory, commenting the original openshift_master_named_certificates variable and add a new variable for wildcard-flibberty-jibbet.com.{crt,key}.

#openshift_master_named_certificates=[{"certfile": "/root/master.public-example.com.crt", "keyfile": "/root/master.public-example.com.key"}]
openshift_master_named_certificates=[{"certfile": "/root/wildcard-flibberty-jibbet.com.crt", "keyfile": "/root/wildcard-flibberty-jibbet.com.key"}]

Verify Configuration for wildcard-flibberty-jibbet.com

Ensure certificates exist in /etc/origin/master/named_certificates/

Both sets of certificates will exist on the filesystem.

$ ls /etc/origin/master/named_certificates/
master.public-example.com.crt  master.public-example.com.key  wildcard-flibberty-jibbet.com.crt  wildcard-flibberty-jibbet.com.key

Verify master configuration

Both certificates will be configured.

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
  namedCertificates:
  - certFile: /etc/origin/master/named_certificates/master.public-example.com.crt
    keyFile: /etc/origin/master/named_certificates/master.public-example.com.key
    names:
    - "master.public-example.com"
  - certFile: /etc/origin/master/named_certificates/wildcard-flibberty-jibbet.com.crt
    keyFile: /etc/origin/master/named_certificates/wildcard-flibberty-jibbet.com.key
    names:
    - "*.flibberty-jibbet.com"

Ensure API serves the default certificate

$ curl -vk https://192.168.122.224:8443
* About to connect() to 192.168.122.224 port 8443 (#0)
*   Trying 192.168.122.224...
* Connected to 192.168.122.224 (192.168.122.224) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*       subject: CN=172.30.0.1
*       start date: Nov 17 00:53:36 2015 GMT
*       expire date: Nov 16 00:53:37 2017 GMT
*       common name: 172.30.0.1
*       issuer: CN=openshift-signer@1447721613
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.122.224:8443
> Accept: */*

Ensure API serves certificate for master.flibberty-jibbet.com

$ curl -vk https://master.flibberty-jibbet.com:8443
* About to connect() to master.flibberty-jibbet.com port 8443 (#0)
*   Trying 192.168.122.224...
* Connected to master.flibberty-jibbet.com (192.168.122.224) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*       subject: E=none@flibberty-jibbet.comm,CN=*.flibberty-jibbet.com,OU=OpenShift,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
*       start date: Nov 17 00:22:37 2015 GMT
*       expire date: Mar 31 00:22:37 2017 GMT
*       common name: *.flibberty-jibbet.com
*       issuer: E=none@public-example.com,CN=public-example.com,OU=OpenShift,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: master.flibberty-jibbet.com:8443
> Accept: */*

4. Re-run ansible with no certificates

If we have previously ran ansible with openshift_master_named_certificates and then re-run with no variable set, the original configuration and certificates should remain in place.

Comment all previous openshift_master_named_certificates inventory variables.

#openshift_master_named_certificates=[{"certfile": "/root/master.public-example.com.crt", "keyfile": "/root/master.public-example.com.key"}]
#openshift_master_named_certificates=[{"certfile": "/root/wildcard-flibberty-jibbet.com.crt", "keyfile": "/root/wildcard-flibberty-jibbet.com.key"}]

Verify Configuration with no certificates

Ensure certificates exist in /etc/origin/master/named_certificates/

Both sets of certificates will exist on the filesystem.

$ ls /etc/origin/master/named_certificates/
master.public-example.com.crt  master.public-example.com.key  wildcard-flibberty-jibbet.com.crt  wildcard-flibberty-jibbet.com.key

Verify master configuration

Both certificates will be configured.

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
  namedCertificates:
  - certFile: /etc/origin/master/named_certificates/master.public-example.com.crt
    keyFile: /etc/origin/master/named_certificates/master.public-example.com.key
    names:
    - "master.public-example.com"
  - certFile: /etc/origin/master/named_certificates/wildcard-flibberty-jibbet.com.crt
    keyFile: /etc/origin/master/named_certificates/wildcard-flibberty-jibbet.com.key
    names:
    - "*.flibberty-jibbet.com"

5. Re-run ansible and overwrite configured certificates

Modify host inventory, uncommenting the openshift_master_named_certificate inventory variable for master.public-example.com, commenting the openshift_master_named_certificates inventory variable for wildcard-flibberty-jibbet.com and add openshift_master_overwrite_named_certificates=true.

openshift_master_named_certificates=[{"certfile": "/root/master.public-example.com.crt", "keyfile": "/root/master.public-example.com.key"}]
#openshift_master_named_certificates=[{"certfile": "/root/wildcard-flibberty-jibbet.com.crt", "keyfile": "/root/wildcard-flibberty-jibbet.com.key"}]
openshift_master_overwrite_named_certificates=true

Verify Configuration with overwritten certificates

Ensure correct certificates exist in /etc/origin/master/named_certificates/

wildcard-flibbery-jibbet.{crt,key} should be absent.

$ ls /etc/origin/master/named_certificates/
master.public-example.com.crt  master.public-example.com.key

Verify master configuration

Similarly, wildcard-flibbery-jibbet.{crt,key} should be absent from master configuration.

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
  namedCertificates:
  - certFile: /etc/origin/master/named_certificates/master.public-example.com.crt
    keyFile: /etc/origin/master/named_certificates/master.public-example.com.key
    names:
    - "master.public-example.com"

6. Re-run ansible and clear certificates

Modify host inventory, setting openshift_master_overwrite_named_certificates=true. If this variable is set and no openshift_master_named_certificates are configured, all certificates and configuration will be cleared.

#openshift_master_named_certificates=[{"certfile": "/root/master.public-example.com.crt", "keyfile": "/root/master.public-example.com.key"}]                                                                       
#openshift_master_named_certificates=[{"certfile": "/root/wildcard-flibberty-jibbet.com.crt", "keyfile": "/root/wildcard-flibberty-jibbet.com.key"}]                                                               
openshift_master_overwrite_named_certificates=true

Verify Configuration with cleared certificates

Ensure no certificates exist in /etc/origin/master/named_certificates/

The named_certificates directory should be absent.

$ ls /etc/origin/master/named_certificates/
ls: cannot access /etc/origin/master/named_certificates/: No such file or directory

Verify master configuration

No certificates will be configured.

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment