Skip to content

Instantly share code, notes, and snippets.

@tbielawa

tbielawa/README.md

Last active October 10, 2016 18:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tbielawa/fde89c7d617e8df55188dc6419bfcd77 to your computer and use it in GitHub Desktop.
Save tbielawa/fde89c7d617e8df55188dc6419bfcd77 to your computer and use it in GitHub Desktop.
Download ZIP
How to QE the openshift-ansible Certificate Expiration Checker Role
Raw
README.md

This gist explains how to QE the certificate checking role.

Setup

  • Install an ocp containized env with:

Inventory file named hosts like this:

[OSEv3:children]
nodes
nfs
masters
lb
etcd
  • three masters/node(with embeded etcd), one node and one lb
  • all certificates in default(assum that all certs will not expire in 30 days.)

Actions

  • Create a new playbook or copy the 4th example from the role's README (in this gist, see the file playbook.yaml)
  • Ensure that the vars section has openshift_certificate_expiry_warning_days set to a sufficiently high number. This number must be greater than the number of days before some of your certificates expire. This condition will ensure we generate WARNINGS when we run the playbook
  • Ensure that the vars section has openshift_certificate_expiry_save_json_results set to yes. This will ensure the results are saved in a JSON file for later review.
  • Ensure the inventory file you created when you provisioned the cluster is representative of the current cluster
  • Set hosts to either all or any subset of host groups you wish to check
  • Run the playbook:
$ ansible-playbook -v -i ./hosts ./test-cert-expiry-playbook.yaml

If the -v option was given to ansible-playbook then there will be results produced on the screen as well (in this gist, see ansible-playbook.log for an example)

Verify that all hosts specified in hosts in your playbook were checked.

Expected Results

The following certificates will be checked if they are present on each examined host:

  • Master/Node Service Certificates
  • Router/Registry Service Certificates from etcd secrets
  • Master/Node/Router/Registry/Admin kubeconfigs
  • Etcd certificates

If the playbook was ran with -v then output similar to ansible-playbook.log will be displayed on the screen. If -v was omitted then you will only see task headers printed. Check results will be hidden.

The task results will not be failures.

The check results will be saved in a JSON file located at the default path for the role, /tmp/cert-expiry-report.json.

Verify that certificates are WARNING by parsing the summary results from the JSON file. In this gist see cert-expiry-report.json for reference.

The README in the role explains how to do this after the example results in the JSON section. I will include those notes below to save time:

The summary from the json data can be easily checked for warnings/expirations using a variety of command-line tools.

For exampe, using grep we can look for the word summary and print out the 2 lines after the match (-A2):

$ grep -A2 summary /tmp/cert-expiry-report.json
    "summary": {
        "warning": 16,
        "expired": 0

If available, the jq tool can also be used to pick out specific values. Example 1 and 2 below show how to select just one value, either warning or expired. Example 3 shows how to select both values at once:

$ jq '.summary.warning' /tmp/cert-expiry-report.json
16
$ jq '.summary.expired' /tmp/cert-expiry-report.json
0
$ jq '.summary.warning,.summary.expired' /tmp/cert-expiry-report.json
16
0
  • On a new cluster the certificates will not be expired. The expired result will be 0.
  • With the variable openshift_certificate_expiry_warning_days set high enough the warning result will be some positive number (relative to the exact configuration of your OCP cluster)
Raw
ansible-playbook.log
Using /etc/ansible/ansible.cfg as config file
PLAY [Check cert expirys] ******************************************************
TASK [openshift_certificate_expiry : Check cert expirys on host] ***************
ok: [192.168.124.11] => {
"changed": false,
"check_results": {
"etcd": [],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:n01.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:57",
"health": "warning",
"path": "/etc/origin/node/system:node:n01.example.com.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:10:54.472640",
"show_all": "False",
"warn_before_date": "2020-11-18 14:10:54.472640",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:192.168.124.11, DNS:n01.example.com, DNS:192.168.124.11, IP Address:192.168.124.11",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [],
"router": []
},
"rc": 0,
"summary": {
"etcd_certificates": 0,
"expired": 0,
"kubeconfig_certificates": 1,
"ok": 1,
"registry_certs": 0,
"router_certs": 0,
"system_certificates": 2,
"total": 3,
"warning": 2
}
}
MSG:
Checked 3 total certificates. Expired/Warning/OK: 0/2/1. Warning window: 1500 days
ok: [192.168.124.110] => {
"changed": false,
"check_results": {
"etcd": [],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:n02.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:58",
"health": "warning",
"path": "/etc/origin/node/system:node:n02.example.com.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:11:00.085800",
"show_all": "False",
"warn_before_date": "2020-11-18 14:11:00.085800",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:192.168.124.110, DNS:n02.example.com, DNS:192.168.124.110, IP Address:192.168.124.110",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [],
"router": []
},
"rc": 0,
"summary": {
"etcd_certificates": 0,
"expired": 0,
"kubeconfig_certificates": 1,
"ok": 1,
"registry_certs": 0,
"router_certs": 0,
"system_certificates": 2,
"total": 3,
"warning": 2
}
}
MSG:
Checked 3 total certificates. Expired/Warning/OK: 0/2/1. Warning window: 1500 days
ok: [192.168.124.148] => {
"changed": false,
"check_results": {
"etcd": [
{
"cert_cn": "CN:etcd-signer@1474563722",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:25",
"health": "warning",
"path": "/etc/etcd/ca.crt"
},
{
"cert_cn": "CN:m01.example.com, IP Address:192.168.124.148",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:39",
"health": "warning",
"path": "/etc/etcd/server.crt"
},
{
"cert_cn": "CN:m01.example.com, IP Address:192.168.124.148",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:40",
"health": "warning",
"path": "/etc/etcd/peer.crt"
}
],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:m01.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:57",
"health": "warning",
"path": "/etc/origin/node/system:node:m01.example.com.kubeconfig"
},
{
"cert_cn": "O:system:cluster-admins, CN:system:admin",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:40",
"health": "warning",
"path": "/etc/origin/master/admin.kubeconfig"
},
{
"cert_cn": "O:system:masters, CN:system:openshift-master",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:39",
"health": "warning",
"path": "/etc/origin/master/openshift-master.kubeconfig"
},
{
"cert_cn": "O:system:routers, CN:system:openshift-router",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:41",
"health": "warning",
"path": "/etc/origin/master/openshift-router.kubeconfig"
},
{
"cert_cn": "O:system:registries, CN:system:openshift-registry",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:41",
"health": "warning",
"path": "/etc/origin/master/openshift-registry.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:11:00.382347",
"show_all": "False",
"warn_before_date": "2020-11-18 14:11:00.382347",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:172.30.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:m01.example.com, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.30.0.1, DNS:192.168.124.148, IP Address:172.30.0.1, IP Address:192.168.124.148",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:39",
"health": "warning",
"path": "/etc/origin/master/master.server.crt"
},
{
"cert_cn": "CN:172.30.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:m01.example.com, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.30.0.1, DNS:192.168.124.148, IP Address:172.30.0.1, IP Address:192.168.124.148",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [
{
"cert_cn": "CN:172.30.101.81, DNS:docker-registry-default.router.default.svc.cluster.local, DNS:docker-registry.default.svc.cluster.local, DNS:172.30.101.81, IP Address:172.30.101.81",
"days_remaining": 725,
"expiry": "2018-10-05 18:54:29",
"health": "warning",
"path": "/api/v1/namespaces/default/secrets/registry-certificates"
}
],
"router": [
{
"cert_cn": "CN:router.default.svc, DNS:router.default.svc, DNS:router.default.svc.cluster.local",
"days_remaining": 712,
"expiry": "2018-09-22 17:48:23",
"health": "warning",
"path": "/api/v1/namespaces/default/secrets/router-certs"
}
]
},
"rc": 0,
"summary": {
"etcd_certificates": 3,
"expired": 0,
"kubeconfig_certificates": 5,
"ok": 2,
"registry_certs": 1,
"router_certs": 1,
"system_certificates": 4,
"total": 14,
"warning": 12
}
}
MSG:
Checked 14 total certificates. Expired/Warning/OK: 0/12/2. Warning window: 1500 days
TASK [openshift_certificate_expiry : Generate expiration report HTML] **********
skipping: [192.168.124.148] => {
"changed": false,
"skip_reason": "Conditional check failed",
"skipped": true
}
TASK [openshift_certificate_expiry : Generate expiration results JSON] *********
changed: [192.168.124.148 -> localhost] => {
"changed": true,
"checksum": "7d8c377c0d61e0e749cea8dcbe8d76a3e9b41288",
"dest": "/tmp/cert-expiry-report.json",
"gid": 1000,
"group": "tbielawa",
"md5sum": "dcc1af0b147b0191e8836fdbadffb224",
"mode": "0664",
"owner": "tbielawa",
"secontext": "unconfined_u:object_r:user_tmp_t:s0",
"size": 6970,
"src": "/home/tbielawa/.ansible/tmp/ansible-tmp-1476123060.91-21395945722414/source",
"state": "file",
"uid": 1000
}
PLAY RECAP *********************************************************************
192.168.124.11 : ok=1 changed=0 unreachable=0 failed=0
192.168.124.110 : ok=1 changed=0 unreachable=0 failed=0
192.168.124.148 : ok=2 changed=1 unreachable=0 failed=0
Playbook run took 0 days, 0 hours, 0 minutes, 8 seconds
Raw
cert-expiry-report.json
{
"data": {
"192.168.124.148": {
"etcd": [
{
"cert_cn": "CN:etcd-signer@1474563722",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:25",
"health": "warning",
"path": "/etc/etcd/ca.crt"
},
{
"cert_cn": "CN:m01.example.com, IP Address:192.168.124.148",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:39",
"health": "warning",
"path": "/etc/etcd/server.crt"
},
{
"cert_cn": "CN:m01.example.com, IP Address:192.168.124.148",
"days_remaining": 347,
"expiry": "2017-09-22 17:02:40",
"health": "warning",
"path": "/etc/etcd/peer.crt"
}
],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:m01.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:57",
"health": "warning",
"path": "/etc/origin/node/system:node:m01.example.com.kubeconfig"
},
{
"cert_cn": "O:system:cluster-admins, CN:system:admin",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:40",
"health": "warning",
"path": "/etc/origin/master/admin.kubeconfig"
},
{
"cert_cn": "O:system:masters, CN:system:openshift-master",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:39",
"health": "warning",
"path": "/etc/origin/master/openshift-master.kubeconfig"
},
{
"cert_cn": "O:system:routers, CN:system:openshift-router",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:41",
"health": "warning",
"path": "/etc/origin/master/openshift-router.kubeconfig"
},
{
"cert_cn": "O:system:registries, CN:system:openshift-registry",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:41",
"health": "warning",
"path": "/etc/origin/master/openshift-registry.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:11:00.382347",
"show_all": "False",
"warn_before_date": "2020-11-18 14:11:00.382347",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:172.30.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:m01.example.com, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.30.0.1, DNS:192.168.124.148, IP Address:172.30.0.1, IP Address:192.168.124.148",
"days_remaining": 712,
"expiry": "2018-09-22 17:04:39",
"health": "warning",
"path": "/etc/origin/master/master.server.crt"
},
{
"cert_cn": "CN:172.30.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:m01.example.com, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:172.30.0.1, DNS:192.168.124.148, IP Address:172.30.0.1, IP Address:192.168.124.148",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [
{
"cert_cn": "CN:172.30.101.81, DNS:docker-registry-default.router.default.svc.cluster.local, DNS:docker-registry.default.svc.cluster.local, DNS:172.30.101.81, IP Address:172.30.101.81",
"days_remaining": 725,
"expiry": "2018-10-05 18:54:29",
"health": "warning",
"path": "/api/v1/namespaces/default/secrets/registry-certificates"
}
],
"router": [
{
"cert_cn": "CN:router.default.svc, DNS:router.default.svc, DNS:router.default.svc.cluster.local",
"days_remaining": 712,
"expiry": "2018-09-22 17:48:23",
"health": "warning",
"path": "/api/v1/namespaces/default/secrets/router-certs"
}
]
},
"192.168.124.11": {
"etcd": [],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:n01.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:57",
"health": "warning",
"path": "/etc/origin/node/system:node:n01.example.com.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:10:54.472640",
"show_all": "False",
"warn_before_date": "2020-11-18 14:10:54.472640",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:192.168.124.11, DNS:n01.example.com, DNS:192.168.124.11, IP Address:192.168.124.11",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [],
"router": []
},
"192.168.124.110": {
"etcd": [],
"kubeconfigs": [
{
"cert_cn": "O:system:nodes, CN:system:node:n02.example.com",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:58",
"health": "warning",
"path": "/etc/origin/node/system:node:n02.example.com.kubeconfig"
}
],
"meta": {
"checked_at_time": "2016-10-10 14:11:00.085800",
"show_all": "False",
"warn_before_date": "2020-11-18 14:11:00.085800",
"warning_days": 1500
},
"ocp_certs": [
{
"cert_cn": "CN:192.168.124.110, DNS:n02.example.com, DNS:192.168.124.110, IP Address:192.168.124.110",
"days_remaining": 712,
"expiry": "2018-09-22 17:08:59",
"health": "warning",
"path": "/etc/origin/node/server.crt"
}
],
"registry": [],
"router": []
}
},
"summary": {
"warning": 16,
"expired": 0
}
}
Raw
playbook.yaml
---
- name: Check cert expirys
hosts: all
become: yes
gather_facts: no
vars:
openshift_certificate_expiry_warning_days: 1500
openshift_certificate_expiry_save_json_results: yes
roles:
- role: openshift_certificate_expiry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment