Skip to content

Instantly share code, notes, and snippets.

@magick93

magick93/Approve node certificates when bootstrapping

Created March 22, 2019 00:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save magick93/814aaa5f825c5021c2d031d8b7944c28 to your computer and use it in GitHub Desktop.
Save magick93/814aaa5f825c5021c2d031d8b7944c28 to your computer and use it in GitHub Desktop.
Download ZIP
Unable to install okd 3.11 due to error with Approve node certificates when bootstrapping
Raw
Approve node certificates when bootstrapping
Using module file /home/user/git/sourcetopia/openshift-install/openshift/ansible/roles/lib_openshift/library/oc_csr_approve.py
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'echo ~deploy && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '/home/deploy\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729 `" && echo ansible-tmp-1553214624.6-3330728421729="` echo /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729 `" ) && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, 'ansible-tmp-1553214624.6-3330728421729=/home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> PUT /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmpJvNOzG TO /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/oc_csr_approve.py
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d '[k8s-ctl01.dev.mydomain.com]'
<k8s-ctl01.dev.mydomain.com> (0, 'sftp> put /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmpJvNOzG /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/oc_csr_approve.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 15 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/deploy size 0\r\ndebug3: Looking up /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmpJvNOzG\r\ndebug3: Sent message fd 15 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/oc_csr_approve.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:3446\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 3446 bytes at 65536\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'chmod u+x /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/ /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/oc_csr_approve.py && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d -tt k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=jhyurlmuibnpcshzckniaezebrggzhsn] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-jhyurlmuibnpcshzckniaezebrggzhsn; /usr/bin/python /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/oc_csr_approve.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<k8s-ctl01.dev.mydomain.com> (1, '\r\n\r\n{"server_csrs": null, "exception": " File \\"/tmp/ansible_HSGjw2/ansible_module_oc_csr_approve.py\\", line 24, in <module>\\n from json.decoder import JSONDecodeError\\n", "raw_failures": [], "oc_get_nodes": {"items": [{"status": {"capacity": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "4", "memory": "16266800Ki"}, "addresses": [{"type": "InternalIP", "address": "10.4.111.101"}, {"type": "Hostname", "address": "k8s-ctl01.dev.mydomain.com"}], "nodeInfo": {"kernelVersion": "3.10.0-957.el7.x86_64", "kubeletVersion": "v1.11.0+d4cacc0", "containerRuntimeVersion": "docker://1.13.1", "machineID": "bfee8bc7a3cd4673a31ac520eb963eb2", "kubeProxyVersion": "v1.11.0+d4cacc0", "bootID": "c4c9e366-57c7-4334-933b-dab93702b583", "osImage": "CentOS Linux 7 (Core)", "architecture": "amd64", "systemUUID": "1C582A42-E6DF-7E8C-DBB8-DB6315262B0C", "operatingSystem": "linux"}, "allocatable": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "4", "memory": "16164400Ki"}, "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"sizeBytes": 1172234437, "names": ["docker.io/openshift/origin-node@sha256:fd552ffd0bdbbcf6c5c646ec7c086b2fafde0aa560711038b1e35c524d867a10", "docker.io/openshift/origin-node:v3.11.0"]}, {"sizeBytes": 1166879223, "names": ["docker.io/openshift/origin-node@sha256:d198706b58f39b87af543cf73e37cc50632a214bac107d70b6d99034982d214a"]}, {"sizeBytes": 828734067, "names": ["docker.io/openshift/origin-control-plane@sha256:508dad04d4c6c7c6ea1e50ac16a5aeffb3854d391b392d74a68c1bf767c608e0", "docker.io/openshift/origin-control-plane:v3.11", "docker.io/openshift/origin-control-plane:v3.11.0"]}, {"sizeBytes": 826232564, "names": ["docker.io/openshift/origin-control-plane@sha256:bf28e6729781f2d68c9ea121cee5c649a084ae9a4dadefded9321e2e381a2549"]}, {"sizeBytes": 383673671, "names": ["docker.io/openshift/origin-deployer@sha256:88eedcec3c877bea09653f037174f50356071e265d93889964a954a73b03aa8e", "docker.io/openshift/origin-deployer:v3.11", "docker.io/openshift/origin-deployer:v3.11.0"]}, {"sizeBytes": 261526128, "names": ["docker.io/openshift/origin-pod@sha256:91f0f130c991c2a6a9f0e9da43f7e56296bacdc46be0cd768125ed3e6f7ef09d", "docker.io/openshift/origin-pod:v3.11.0"]}, {"sizeBytes": 258456390, "names": ["docker.io/openshift/origin-pod@sha256:67b51982ad05b446e4045b7e2507768c2d25b2eb61ccadd7b6a718b1262c25b8"]}, {"sizeBytes": 37269372, "names": ["quay.io/coreos/etcd@sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948", "quay.io/coreos/etcd:v3.2.22"]}], "conditions": [{"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientDisk", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient disk space available", "type": "OutOfDisk"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientMemory", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient memory available", "type": "MemoryPressure"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasNoDiskPressure", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has no disk pressure", "type": "DiskPressure"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientPID", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient PID available", "type": "PIDPressure"}, {"status": "True", "lastTransitionTime": "2019-03-22T00:27:39Z", "reason": "KubeletReady", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet is posting ready status", "type": "Ready"}]}, "kind": "Node", "spec": {}, "apiVersion": "v1", "metadata": {"name": "k8s-ctl01.dev.mydomain.com", "labels": {"beta.kubernetes.io/os": "linux", "node-role.kubernetes.io/master": "true", "kubernetes.io/hostname": "k8s-ctl01.dev.mydomain.com", "node-role.kubernetes.io/compute": "true", "node-role.kubernetes.io/infra": "true", "beta.kubernetes.io/arch": "amd64"}, "namespace": "", "resourceVersion": "21959", "creationTimestamp": "2019-03-21T20:38:37Z", "annotations": {"volumes.kubernetes.io/controller-managed-attach-detach": "true", "node.openshift.io/md5sum": "a4305b3db4427b8d4bd21c1a11115c5d"}, "selfLink": "/api/v1/nodes/k8s-ctl01.dev.mydomain.com", "uid": "4dfe8828-4c19-11e9-add9-005056aa572c"}}], "kind": "List", "apiVersion": "v1", "metadata": {"selfLink": "", "resourceVersion": ""}}, "changed": false, "client_csrs": {}, "failed": true, "state": "unknown", "all_subjects_found": ["subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\\n", "subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\\n"], "client_approve_results": [], "rc": 0, "invocation": {"module_args": {"oc_bin": "oc", "oc_conf": "/etc/origin/master/admin.kubeconfig", "node_list": ["k8s-worker01.dev.mydomain.com", "k8s-worker02.dev.mydomain.com", "k8s-worker03.dev.mydomain.com", "k8s-ctl01.dev.mydomain.com"]}}, "unwanted_csrs": [{"status": {"conditions": [{"message": "This CSR was approved by kubectl certificate approve.", "type": "Approved", "reason": "KubectlApprove", "lastUpdateTime": "2019-03-22T00:06:42Z"}], "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVQ3NzVlZMeTY4clhBcVhwbnczekEvQTg1TXRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFbmJJU2FSajZHMVpQR2RMUWlDL1lpekpxOXFWUwpBWEorSnJ6N3d1QTUveDNEZ1d5emJSQWN3bkZGbWFRUTZQZk1VWWsyY3A3MnZtS1NORVNvOUlUYWRLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVQTzJvQWlTaEc0bGlLZTRGSDFQTGltb0MwOHN3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUF3YmNITEVScHZTQkVZQUFUMlduVWlGOFppNE5rUGs5ZWRReisvcGJkeGNOeEIKUmZmSzFpRFNSYmRDNjh4ZjBHUjJxdklDWDdwMVhRUURueXJ3K2JvR2JUMUdxUVB4Z1RBTXdHVUtmcTFoMENWbQo5a0g5VE9sNTFWczRYZkhxZFZzdFA1Z3RUT0FpUFNzaWdVSmM3L3FKSHdNeHQ3aytNVmlJY1V1aUd1SjljRWFsClJ0TjZLeHBBTGx4WVA4aVZETTY4VTJISTgwalRrVFRjbW1yb2VzRnlPb1BtSlN5ZjRCNWExWWllcTB6SGNKcEwKYlRsSmpWTDl5QTRZQXdtQ0JzSmxyOCtVMlVweUhYY0hscERmV1ZvT1krRDhaQ2IzaFNRdnBlVkdJNE5OTCswOApQNVd4Ui9LK0xORElGTDVYR1ZDSG5rbm5IL041alZvNVEwUGY5bkZ5Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlJ6Q0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU2RzaEpwR1BvYlZrOFowdENJTDlpTE1tcjJwVklCY240bXZQdkM0RG4vSGNPQmJMTnQKRUJ6Q2NVV1pwQkRvOTh4UmlUWnludmErWXBJMFJLajBoTnAwb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnVGVnSnluemkwZVl1ZHE4cU9LR3hOaXc1YlZheGkyK2sKbHV3NGVNUzVwOFFDSVFEUWRsaTY0LzJ6cldQajlQNjVYWFlSMUR0ME1pMzNERnRnQURQTHdPNnFtQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-bsf6g", "namespace": "", "resourceVersion": "19872", "generateName": "csr-", "creationTimestamp": "2019-03-22T00:06:21Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-bsf6g", "uid": "52f26e8c-4c36-11e9-add9-005056aa572c"}}, {"status": {"conditions": [{"message": "This CSR was approved by kubectl certificate approve.", "type": "Approved", "reason": "KubectlApprove", "lastUpdateTime": "2019-03-22T00:06:36Z"}], "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVTFVQcFY4eitRRHZseDhpa3AydjBpRTA2citRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFanE5MUxpTFlpVzBlVFQrNU4vYVNwRUJIWFFBWgp1ZE1RQTVlbTI1Z3JvWWt0WGlycnJKN0NrcmdveGtDd09hMTBtNkMrMm5uNjU4T3ZwSjg0UHZXYkhLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVwRGtRRjNXS3ArdmlhSmdlVWlBbHJPRXNueWt3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUNDL0JJL0hKYXBhWG1kTUpsd2MvejhlMkRiazBVSWp3UlhjM0lET0FxckRoeGsKR2VOd0tNcUJJc2Fic1pWcXlzMWtqSVBaS0VFWTdwVHFhZU54blFRT2FucWZYMmNuZTV1aGEvbUo0U2NmRUorQQpQOGJaSU1GRzVIS1FPNlBPOSt5SXhKR3I4emFpM1hsd2s0YUNGZ2xCWEQ3SzRsZ0Noc0xXZjdZQWtKcmRKRGEvCkRhNmpzbFQvQithZlpKbmJZeUJSZkNqeVdkUE5lbXdsU1V2ZS9SZ3Z6eXZsMllmTFlpMWJ3bmw2RXNrTW84aWoKUVdBR0tpd3FkUnNCdGNOSStJY1JHYzFDZWY4ZFFhdGVGWFl6c2VINHdONWpwRWZoYllMOVRCbEJ1QnQwWVpZOQpJTlZxUVRGVEZQVzBJWmZQcGU4ZXhUOFR3TUJlRjBoS05uUk03blZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlNEQ0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU09yM1V1SXRpSmJSNU5QN2szOXBLa1FFZGRBQm01MHhBRGw2YmJtQ3VoaVMxZUt1dXMKbnNLU3VDakdRTEE1clhTYm9MN2FlZnJudzYra256Zys5WnNjb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQVBFWlV4L0M0UHFncWhpKytBU3NCbWNCN3Z0SGNTcjMKUWJhL2dSS2hIUWwvQWlFQTJQbzVDK1lzQmFMcWlCMU1NcGQ4THFIS3d3ck0yL0NKczN0RHQyWDRwMkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-v85dj", "namespace": "", "resourceVersion": "19857", "generateName": "csr-", "creationTimestamp": "2019-03-22T00:04:51Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-v85dj", "uid": "1d5cb3b4-4c36-11e9-add9-005056aa572c"}}], "server_approve_results": [], "msg": "Could not find csr for nodes: k8s-worker03.dev.mydomain.com, k8s-worker02.dev.mydomain.com, k8s-worker01.dev.mydomain.com"}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to k8s-ctl01.dev.mydomain.com closed.\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'rm -f -r /home/deploy/.ansible/tmp/ansible-tmp-1553214624.6-3330728421729/ > /dev/null 2>&1 && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: Approve node certificates when bootstrapping (1 retries left).Result was: {
"all_subjects_found": [
"subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\n",
"subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\n"
],
"attempts": 30,
"changed": false,
"client_approve_results": [],
"client_csrs": {},
"invocation": {
"module_args": {
"node_list": [
"k8s-worker01.dev.mydomain.com",
"k8s-worker02.dev.mydomain.com",
"k8s-worker03.dev.mydomain.com",
"k8s-ctl01.dev.mydomain.com"
],
"oc_bin": "oc",
"oc_conf": "/etc/origin/master/admin.kubeconfig"
}
},
"msg": "Could not find csr for nodes: k8s-worker03.dev.mydomain.com, k8s-worker02.dev.mydomain.com, k8s-worker01.dev.mydomain.com",
"oc_get_nodes": {
"apiVersion": "v1",
"items": [
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"node.openshift.io/md5sum": "a4305b3db4427b8d4bd21c1a11115c5d",
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2019-03-21T20:38:37Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/hostname": "k8s-ctl01.dev.mydomain.com",
"node-role.kubernetes.io/compute": "true",
"node-role.kubernetes.io/infra": "true",
"node-role.kubernetes.io/master": "true"
},
"name": "k8s-ctl01.dev.mydomain.com",
"namespace": "",
"resourceVersion": "21959",
"selfLink": "/api/v1/nodes/k8s-ctl01.dev.mydomain.com",
"uid": "4dfe8828-4c19-11e9-add9-005056aa572c"
},
"spec": {},
"status": {
"addresses": [
{
"address": "10.4.111.101",
"type": "InternalIP"
},
{
"address": "k8s-ctl01.dev.mydomain.com",
"type": "Hostname"
}
],
"allocatable": {
"cpu": "4",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "16164400Ki",
"pods": "250"
},
"capacity": {
"cpu": "4",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "16266800Ki",
"pods": "250"
},
"conditions": [
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient disk space available",
"reason": "KubeletHasSufficientDisk",
"status": "False",
"type": "OutOfDisk"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient memory available",
"reason": "KubeletHasSufficientMemory",
"status": "False",
"type": "MemoryPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has no disk pressure",
"reason": "KubeletHasNoDiskPressure",
"status": "False",
"type": "DiskPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient PID available",
"reason": "KubeletHasSufficientPID",
"status": "False",
"type": "PIDPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-22T00:27:39Z",
"message": "kubelet is posting ready status",
"reason": "KubeletReady",
"status": "True",
"type": "Ready"
}
],
"daemonEndpoints": {
"kubeletEndpoint": {
"Port": 10250
}
},
"images": [
{
"names": [
"docker.io/openshift/origin-node@sha256:fd552ffd0bdbbcf6c5c646ec7c086b2fafde0aa560711038b1e35c524d867a10",
"docker.io/openshift/origin-node:v3.11.0"
],
"sizeBytes": 1172234437
},
{
"names": [
"docker.io/openshift/origin-node@sha256:d198706b58f39b87af543cf73e37cc50632a214bac107d70b6d99034982d214a"
],
"sizeBytes": 1166879223
},
{
"names": [
"docker.io/openshift/origin-control-plane@sha256:508dad04d4c6c7c6ea1e50ac16a5aeffb3854d391b392d74a68c1bf767c608e0",
"docker.io/openshift/origin-control-plane:v3.11",
"docker.io/openshift/origin-control-plane:v3.11.0"
],
"sizeBytes": 828734067
},
{
"names": [
"docker.io/openshift/origin-control-plane@sha256:bf28e6729781f2d68c9ea121cee5c649a084ae9a4dadefded9321e2e381a2549"
],
"sizeBytes": 826232564
},
{
"names": [
"docker.io/openshift/origin-deployer@sha256:88eedcec3c877bea09653f037174f50356071e265d93889964a954a73b03aa8e",
"docker.io/openshift/origin-deployer:v3.11",
"docker.io/openshift/origin-deployer:v3.11.0"
],
"sizeBytes": 383673671
},
{
"names": [
"docker.io/openshift/origin-pod@sha256:91f0f130c991c2a6a9f0e9da43f7e56296bacdc46be0cd768125ed3e6f7ef09d",
"docker.io/openshift/origin-pod:v3.11.0"
],
"sizeBytes": 261526128
},
{
"names": [
"docker.io/openshift/origin-pod@sha256:67b51982ad05b446e4045b7e2507768c2d25b2eb61ccadd7b6a718b1262c25b8"
],
"sizeBytes": 258456390
},
{
"names": [
"quay.io/coreos/etcd@sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948",
"quay.io/coreos/etcd:v3.2.22"
],
"sizeBytes": 37269372
}
],
"nodeInfo": {
"architecture": "amd64",
"bootID": "c4c9e366-57c7-4334-933b-dab93702b583",
"containerRuntimeVersion": "docker://1.13.1",
"kernelVersion": "3.10.0-957.el7.x86_64",
"kubeProxyVersion": "v1.11.0+d4cacc0",
"kubeletVersion": "v1.11.0+d4cacc0",
"machineID": "bfee8bc7a3cd4673a31ac520eb963eb2",
"operatingSystem": "linux",
"osImage": "CentOS Linux 7 (Core)",
"systemUUID": "1C582A42-E6DF-7E8C-DBB8-DB6315262B0C"
}
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
},
"raw_failures": [],
"rc": 0,
"retries": 31,
"server_approve_results": [],
"server_csrs": null,
"state": "unknown",
"unwanted_csrs": [
{
"apiVersion": "certificates.k8s.io/v1beta1",
"kind": "CertificateSigningRequest",
"metadata": {
"creationTimestamp": "2019-03-22T00:06:21Z",
"generateName": "csr-",
"name": "csr-bsf6g",
"namespace": "",
"resourceVersion": "19872",
"selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-bsf6g",
"uid": "52f26e8c-4c36-11e9-add9-005056aa572c"
},
"spec": {
"groups": [
"system:masters",
"system:cluster-admins",
"system:authenticated"
],
"request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlJ6Q0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU2RzaEpwR1BvYlZrOFowdENJTDlpTE1tcjJwVklCY240bXZQdkM0RG4vSGNPQmJMTnQKRUJ6Q2NVV1pwQkRvOTh4UmlUWnludmErWXBJMFJLajBoTnAwb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnVGVnSnluemkwZVl1ZHE4cU9LR3hOaXc1YlZheGkyK2sKbHV3NGVNUzVwOFFDSVFEUWRsaTY0LzJ6cldQajlQNjVYWFlSMUR0ME1pMzNERnRnQURQTHdPNnFtQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=",
"usages": [
"digital signature",
"key encipherment",
"server auth"
],
"username": "system:admin"
},
"status": {
"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVQ3NzVlZMeTY4clhBcVhwbnczekEvQTg1TXRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFbmJJU2FSajZHMVpQR2RMUWlDL1lpekpxOXFWUwpBWEorSnJ6N3d1QTUveDNEZ1d5emJSQWN3bkZGbWFRUTZQZk1VWWsyY3A3MnZtS1NORVNvOUlUYWRLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVQTzJvQWlTaEc0bGlLZTRGSDFQTGltb0MwOHN3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUF3YmNITEVScHZTQkVZQUFUMlduVWlGOFppNE5rUGs5ZWRReisvcGJkeGNOeEIKUmZmSzFpRFNSYmRDNjh4ZjBHUjJxdklDWDdwMVhRUURueXJ3K2JvR2JUMUdxUVB4Z1RBTXdHVUtmcTFoMENWbQo5a0g5VE9sNTFWczRYZkhxZFZzdFA1Z3RUT0FpUFNzaWdVSmM3L3FKSHdNeHQ3aytNVmlJY1V1aUd1SjljRWFsClJ0TjZLeHBBTGx4WVA4aVZETTY4VTJISTgwalRrVFRjbW1yb2VzRnlPb1BtSlN5ZjRCNWExWWllcTB6SGNKcEwKYlRsSmpWTDl5QTRZQXdtQ0JzSmxyOCtVMlVweUhYY0hscERmV1ZvT1krRDhaQ2IzaFNRdnBlVkdJNE5OTCswOApQNVd4Ui9LK0xORElGTDVYR1ZDSG5rbm5IL041alZvNVEwUGY5bkZ5Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"conditions": [
{
"lastUpdateTime": "2019-03-22T00:06:42Z",
"message": "This CSR was approved by kubectl certificate approve.",
"reason": "KubectlApprove",
"type": "Approved"
}
]
}
},
{
"apiVersion": "certificates.k8s.io/v1beta1",
"kind": "CertificateSigningRequest",
"metadata": {
"creationTimestamp": "2019-03-22T00:04:51Z",
"generateName": "csr-",
"name": "csr-v85dj",
"namespace": "",
"resourceVersion": "19857",
"selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-v85dj",
"uid": "1d5cb3b4-4c36-11e9-add9-005056aa572c"
},
"spec": {
"groups": [
"system:masters",
"system:cluster-admins",
"system:authenticated"
],
"request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlNEQ0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU09yM1V1SXRpSmJSNU5QN2szOXBLa1FFZGRBQm01MHhBRGw2YmJtQ3VoaVMxZUt1dXMKbnNLU3VDakdRTEE1clhTYm9MN2FlZnJudzYra256Zys5WnNjb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQVBFWlV4L0M0UHFncWhpKytBU3NCbWNCN3Z0SGNTcjMKUWJhL2dSS2hIUWwvQWlFQTJQbzVDK1lzQmFMcWlCMU1NcGQ4THFIS3d3ck0yL0NKczN0RHQyWDRwMkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=",
"usages": [
"digital signature",
"key encipherment",
"server auth"
],
"username": "system:admin"
},
"status": {
"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVTFVQcFY4eitRRHZseDhpa3AydjBpRTA2citRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFanE5MUxpTFlpVzBlVFQrNU4vYVNwRUJIWFFBWgp1ZE1RQTVlbTI1Z3JvWWt0WGlycnJKN0NrcmdveGtDd09hMTBtNkMrMm5uNjU4T3ZwSjg0UHZXYkhLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVwRGtRRjNXS3ArdmlhSmdlVWlBbHJPRXNueWt3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUNDL0JJL0hKYXBhWG1kTUpsd2MvejhlMkRiazBVSWp3UlhjM0lET0FxckRoeGsKR2VOd0tNcUJJc2Fic1pWcXlzMWtqSVBaS0VFWTdwVHFhZU54blFRT2FucWZYMmNuZTV1aGEvbUo0U2NmRUorQQpQOGJaSU1GRzVIS1FPNlBPOSt5SXhKR3I4emFpM1hsd2s0YUNGZ2xCWEQ3SzRsZ0Noc0xXZjdZQWtKcmRKRGEvCkRhNmpzbFQvQithZlpKbmJZeUJSZkNqeVdkUE5lbXdsU1V2ZS9SZ3Z6eXZsMllmTFlpMWJ3bmw2RXNrTW84aWoKUVdBR0tpd3FkUnNCdGNOSStJY1JHYzFDZWY4ZFFhdGVGWFl6c2VINHdONWpwRWZoYllMOVRCbEJ1QnQwWVpZOQpJTlZxUVRGVEZQVzBJWmZQcGU4ZXhUOFR3TUJlRjBoS05uUk03blZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"conditions": [
{
"lastUpdateTime": "2019-03-22T00:06:36Z",
"message": "This CSR was approved by kubectl certificate approve.",
"reason": "KubectlApprove",
"type": "Approved"
}
]
}
}
]
}
Using module file /home/user/git/sourcetopia/openshift-install/openshift/ansible/roles/lib_openshift/library/oc_csr_approve.py
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'echo ~deploy && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '/home/deploy\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712 `" && echo ansible-tmp-1553214630.19-148976102485712="` echo /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712 `" ) && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, 'ansible-tmp-1553214630.19-148976102485712=/home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> PUT /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmp76X_RJ TO /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/oc_csr_approve.py
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d '[k8s-ctl01.dev.mydomain.com]'
<k8s-ctl01.dev.mydomain.com> (0, 'sftp> put /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmp76X_RJ /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/oc_csr_approve.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 15 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/deploy size 0\r\ndebug3: Looking up /home/user/.ansible/tmp/ansible-local-10931AT0_NQ/tmp76X_RJ\r\ndebug3: Sent message fd 15 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/oc_csr_approve.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:3446\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 3446 bytes at 65536\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'chmod u+x /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/ /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/oc_csr_approve.py && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d -tt k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=tuddxsandiilnpnlwrnfjjxpssvtizlb] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-tuddxsandiilnpnlwrnfjjxpssvtizlb; /usr/bin/python /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/oc_csr_approve.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<k8s-ctl01.dev.mydomain.com> (1, '\r\n\r\n{"server_csrs": null, "exception": " File \\"/tmp/ansible_NR_QE0/ansible_module_oc_csr_approve.py\\", line 24, in <module>\\n from json.decoder import JSONDecodeError\\n", "raw_failures": [], "oc_get_nodes": {"items": [{"status": {"capacity": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "4", "memory": "16266800Ki"}, "addresses": [{"type": "InternalIP", "address": "10.4.111.101"}, {"type": "Hostname", "address": "k8s-ctl01.dev.mydomain.com"}], "nodeInfo": {"kernelVersion": "3.10.0-957.el7.x86_64", "kubeletVersion": "v1.11.0+d4cacc0", "containerRuntimeVersion": "docker://1.13.1", "machineID": "bfee8bc7a3cd4673a31ac520eb963eb2", "kubeProxyVersion": "v1.11.0+d4cacc0", "bootID": "c4c9e366-57c7-4334-933b-dab93702b583", "osImage": "CentOS Linux 7 (Core)", "architecture": "amd64", "systemUUID": "1C582A42-E6DF-7E8C-DBB8-DB6315262B0C", "operatingSystem": "linux"}, "allocatable": {"hugepages-1Gi": "0", "hugepages-2Mi": "0", "pods": "250", "cpu": "4", "memory": "16164400Ki"}, "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"sizeBytes": 1172234437, "names": ["docker.io/openshift/origin-node@sha256:fd552ffd0bdbbcf6c5c646ec7c086b2fafde0aa560711038b1e35c524d867a10", "docker.io/openshift/origin-node:v3.11.0"]}, {"sizeBytes": 1166879223, "names": ["docker.io/openshift/origin-node@sha256:d198706b58f39b87af543cf73e37cc50632a214bac107d70b6d99034982d214a"]}, {"sizeBytes": 828734067, "names": ["docker.io/openshift/origin-control-plane@sha256:508dad04d4c6c7c6ea1e50ac16a5aeffb3854d391b392d74a68c1bf767c608e0", "docker.io/openshift/origin-control-plane:v3.11", "docker.io/openshift/origin-control-plane:v3.11.0"]}, {"sizeBytes": 826232564, "names": ["docker.io/openshift/origin-control-plane@sha256:bf28e6729781f2d68c9ea121cee5c649a084ae9a4dadefded9321e2e381a2549"]}, {"sizeBytes": 383673671, "names": ["docker.io/openshift/origin-deployer@sha256:88eedcec3c877bea09653f037174f50356071e265d93889964a954a73b03aa8e", "docker.io/openshift/origin-deployer:v3.11", "docker.io/openshift/origin-deployer:v3.11.0"]}, {"sizeBytes": 261526128, "names": ["docker.io/openshift/origin-pod@sha256:91f0f130c991c2a6a9f0e9da43f7e56296bacdc46be0cd768125ed3e6f7ef09d", "docker.io/openshift/origin-pod:v3.11.0"]}, {"sizeBytes": 258456390, "names": ["docker.io/openshift/origin-pod@sha256:67b51982ad05b446e4045b7e2507768c2d25b2eb61ccadd7b6a718b1262c25b8"]}, {"sizeBytes": 37269372, "names": ["quay.io/coreos/etcd@sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948", "quay.io/coreos/etcd:v3.2.22"]}], "conditions": [{"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientDisk", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient disk space available", "type": "OutOfDisk"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientMemory", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient memory available", "type": "MemoryPressure"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasNoDiskPressure", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has no disk pressure", "type": "DiskPressure"}, {"status": "False", "lastTransitionTime": "2019-03-21T20:38:32Z", "reason": "KubeletHasSufficientPID", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet has sufficient PID available", "type": "PIDPressure"}, {"status": "True", "lastTransitionTime": "2019-03-22T00:27:39Z", "reason": "KubeletReady", "lastHeartbeatTime": "2019-03-22T00:30:39Z", "message": "kubelet is posting ready status", "type": "Ready"}]}, "kind": "Node", "spec": {}, "apiVersion": "v1", "metadata": {"name": "k8s-ctl01.dev.mydomain.com", "labels": {"beta.kubernetes.io/os": "linux", "node-role.kubernetes.io/master": "true", "kubernetes.io/hostname": "k8s-ctl01.dev.mydomain.com", "node-role.kubernetes.io/compute": "true", "node-role.kubernetes.io/infra": "true", "beta.kubernetes.io/arch": "amd64"}, "namespace": "", "resourceVersion": "21959", "creationTimestamp": "2019-03-21T20:38:37Z", "annotations": {"volumes.kubernetes.io/controller-managed-attach-detach": "true", "node.openshift.io/md5sum": "a4305b3db4427b8d4bd21c1a11115c5d"}, "selfLink": "/api/v1/nodes/k8s-ctl01.dev.mydomain.com", "uid": "4dfe8828-4c19-11e9-add9-005056aa572c"}}], "kind": "List", "apiVersion": "v1", "metadata": {"selfLink": "", "resourceVersion": ""}}, "changed": false, "client_csrs": {}, "failed": true, "state": "unknown", "all_subjects_found": ["subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\\n", "subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\\n"], "client_approve_results": [], "rc": 0, "invocation": {"module_args": {"oc_bin": "oc", "oc_conf": "/etc/origin/master/admin.kubeconfig", "node_list": ["k8s-worker01.dev.mydomain.com", "k8s-worker02.dev.mydomain.com", "k8s-worker03.dev.mydomain.com", "k8s-ctl01.dev.mydomain.com"]}}, "unwanted_csrs": [{"status": {"conditions": [{"message": "This CSR was approved by kubectl certificate approve.", "type": "Approved", "reason": "KubectlApprove", "lastUpdateTime": "2019-03-22T00:06:42Z"}], "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVQ3NzVlZMeTY4clhBcVhwbnczekEvQTg1TXRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFbmJJU2FSajZHMVpQR2RMUWlDL1lpekpxOXFWUwpBWEorSnJ6N3d1QTUveDNEZ1d5emJSQWN3bkZGbWFRUTZQZk1VWWsyY3A3MnZtS1NORVNvOUlUYWRLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVQTzJvQWlTaEc0bGlLZTRGSDFQTGltb0MwOHN3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUF3YmNITEVScHZTQkVZQUFUMlduVWlGOFppNE5rUGs5ZWRReisvcGJkeGNOeEIKUmZmSzFpRFNSYmRDNjh4ZjBHUjJxdklDWDdwMVhRUURueXJ3K2JvR2JUMUdxUVB4Z1RBTXdHVUtmcTFoMENWbQo5a0g5VE9sNTFWczRYZkhxZFZzdFA1Z3RUT0FpUFNzaWdVSmM3L3FKSHdNeHQ3aytNVmlJY1V1aUd1SjljRWFsClJ0TjZLeHBBTGx4WVA4aVZETTY4VTJISTgwalRrVFRjbW1yb2VzRnlPb1BtSlN5ZjRCNWExWWllcTB6SGNKcEwKYlRsSmpWTDl5QTRZQXdtQ0JzSmxyOCtVMlVweUhYY0hscERmV1ZvT1krRDhaQ2IzaFNRdnBlVkdJNE5OTCswOApQNVd4Ui9LK0xORElGTDVYR1ZDSG5rbm5IL041alZvNVEwUGY5bkZ5Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlJ6Q0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU2RzaEpwR1BvYlZrOFowdENJTDlpTE1tcjJwVklCY240bXZQdkM0RG4vSGNPQmJMTnQKRUJ6Q2NVV1pwQkRvOTh4UmlUWnludmErWXBJMFJLajBoTnAwb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnVGVnSnluemkwZVl1ZHE4cU9LR3hOaXc1YlZheGkyK2sKbHV3NGVNUzVwOFFDSVFEUWRsaTY0LzJ6cldQajlQNjVYWFlSMUR0ME1pMzNERnRnQURQTHdPNnFtQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-bsf6g", "namespace": "", "resourceVersion": "19872", "generateName": "csr-", "creationTimestamp": "2019-03-22T00:06:21Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-bsf6g", "uid": "52f26e8c-4c36-11e9-add9-005056aa572c"}}, {"status": {"conditions": [{"message": "This CSR was approved by kubectl certificate approve.", "type": "Approved", "reason": "KubectlApprove", "lastUpdateTime": "2019-03-22T00:06:36Z"}], "certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVTFVQcFY4eitRRHZseDhpa3AydjBpRTA2citRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFanE5MUxpTFlpVzBlVFQrNU4vYVNwRUJIWFFBWgp1ZE1RQTVlbTI1Z3JvWWt0WGlycnJKN0NrcmdveGtDd09hMTBtNkMrMm5uNjU4T3ZwSjg0UHZXYkhLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVwRGtRRjNXS3ArdmlhSmdlVWlBbHJPRXNueWt3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUNDL0JJL0hKYXBhWG1kTUpsd2MvejhlMkRiazBVSWp3UlhjM0lET0FxckRoeGsKR2VOd0tNcUJJc2Fic1pWcXlzMWtqSVBaS0VFWTdwVHFhZU54blFRT2FucWZYMmNuZTV1aGEvbUo0U2NmRUorQQpQOGJaSU1GRzVIS1FPNlBPOSt5SXhKR3I4emFpM1hsd2s0YUNGZ2xCWEQ3SzRsZ0Noc0xXZjdZQWtKcmRKRGEvCkRhNmpzbFQvQithZlpKbmJZeUJSZkNqeVdkUE5lbXdsU1V2ZS9SZ3Z6eXZsMllmTFlpMWJ3bmw2RXNrTW84aWoKUVdBR0tpd3FkUnNCdGNOSStJY1JHYzFDZWY4ZFFhdGVGWFl6c2VINHdONWpwRWZoYllMOVRCbEJ1QnQwWVpZOQpJTlZxUVRGVEZQVzBJWmZQcGU4ZXhUOFR3TUJlRjBoS05uUk03blZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}, "kind": "CertificateSigningRequest", "spec": {"username": "system:admin", "usages": ["digital signature", "key encipherment", "server auth"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlNEQ0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU09yM1V1SXRpSmJSNU5QN2szOXBLa1FFZGRBQm01MHhBRGw2YmJtQ3VoaVMxZUt1dXMKbnNLU3VDakdRTEE1clhTYm9MN2FlZnJudzYra256Zys5WnNjb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQVBFWlV4L0M0UHFncWhpKytBU3NCbWNCN3Z0SGNTcjMKUWJhL2dSS2hIUWwvQWlFQTJQbzVDK1lzQmFMcWlCMU1NcGQ4THFIS3d3ck0yL0NKczN0RHQyWDRwMkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "groups": ["system:masters", "system:cluster-admins", "system:authenticated"]}, "apiVersion": "certificates.k8s.io/v1beta1", "metadata": {"name": "csr-v85dj", "namespace": "", "resourceVersion": "19857", "generateName": "csr-", "creationTimestamp": "2019-03-22T00:04:51Z", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-v85dj", "uid": "1d5cb3b4-4c36-11e9-add9-005056aa572c"}}], "server_approve_results": [], "msg": "Could not find csr for nodes: k8s-worker03.dev.mydomain.com, k8s-worker02.dev.mydomain.com, k8s-worker01.dev.mydomain.com"}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to k8s-ctl01.dev.mydomain.com closed.\r\n')
<k8s-ctl01.dev.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<k8s-ctl01.dev.mydomain.com> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<k8s-ctl01.dev.mydomain.com> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=deploy)
<k8s-ctl01.dev.mydomain.com> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<k8s-ctl01.dev.mydomain.com> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/user/.ansible/cp/2ea0bd867d)
<k8s-ctl01.dev.mydomain.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/2ea0bd867d k8s-ctl01.dev.mydomain.com '/bin/sh -c '"'"'rm -f -r /home/deploy/.ansible/tmp/ansible-tmp-1553214630.19-148976102485712/ > /dev/null 2>&1 && sleep 0'"'"''
<k8s-ctl01.dev.mydomain.com> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12791\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
The full traceback is:
File "/tmp/ansible_NR_QE0/ansible_module_oc_csr_approve.py", line 24, in <module>
from json.decoder import JSONDecodeError
fatal: [k8s-ctl01.dev.mydomain.com]: FAILED! => {
"all_subjects_found": [
"subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\n",
"subject=/O=system:nodes/CN=system:node:k8s-ctl01.dev.mydomain.com\n"
],
"attempts": 30,
"changed": false,
"client_approve_results": [],
"client_csrs": {},
"invocation": {
"module_args": {
"node_list": [
"k8s-worker01.dev.mydomain.com",
"k8s-worker02.dev.mydomain.com",
"k8s-worker03.dev.mydomain.com",
"k8s-ctl01.dev.mydomain.com"
],
"oc_bin": "oc",
"oc_conf": "/etc/origin/master/admin.kubeconfig"
}
},
"msg": "Could not find csr for nodes: k8s-worker03.dev.mydomain.com, k8s-worker02.dev.mydomain.com, k8s-worker01.dev.mydomain.com",
"oc_get_nodes": {
"apiVersion": "v1",
"items": [
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"node.openshift.io/md5sum": "a4305b3db4427b8d4bd21c1a11115c5d",
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2019-03-21T20:38:37Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/hostname": "k8s-ctl01.dev.mydomain.com",
"node-role.kubernetes.io/compute": "true",
"node-role.kubernetes.io/infra": "true",
"node-role.kubernetes.io/master": "true"
},
"name": "k8s-ctl01.dev.mydomain.com",
"namespace": "",
"resourceVersion": "21959",
"selfLink": "/api/v1/nodes/k8s-ctl01.dev.mydomain.com",
"uid": "4dfe8828-4c19-11e9-add9-005056aa572c"
},
"spec": {},
"status": {
"addresses": [
{
"address": "10.4.111.101",
"type": "InternalIP"
},
{
"address": "k8s-ctl01.dev.mydomain.com",
"type": "Hostname"
}
],
"allocatable": {
"cpu": "4",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "16164400Ki",
"pods": "250"
},
"capacity": {
"cpu": "4",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "16266800Ki",
"pods": "250"
},
"conditions": [
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient disk space available",
"reason": "KubeletHasSufficientDisk",
"status": "False",
"type": "OutOfDisk"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient memory available",
"reason": "KubeletHasSufficientMemory",
"status": "False",
"type": "MemoryPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has no disk pressure",
"reason": "KubeletHasNoDiskPressure",
"status": "False",
"type": "DiskPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-21T20:38:32Z",
"message": "kubelet has sufficient PID available",
"reason": "KubeletHasSufficientPID",
"status": "False",
"type": "PIDPressure"
},
{
"lastHeartbeatTime": "2019-03-22T00:30:39Z",
"lastTransitionTime": "2019-03-22T00:27:39Z",
"message": "kubelet is posting ready status",
"reason": "KubeletReady",
"status": "True",
"type": "Ready"
}
],
"daemonEndpoints": {
"kubeletEndpoint": {
"Port": 10250
}
},
"images": [
{
"names": [
"docker.io/openshift/origin-node@sha256:fd552ffd0bdbbcf6c5c646ec7c086b2fafde0aa560711038b1e35c524d867a10",
"docker.io/openshift/origin-node:v3.11.0"
],
"sizeBytes": 1172234437
},
{
"names": [
"docker.io/openshift/origin-node@sha256:d198706b58f39b87af543cf73e37cc50632a214bac107d70b6d99034982d214a"
],
"sizeBytes": 1166879223
},
{
"names": [
"docker.io/openshift/origin-control-plane@sha256:508dad04d4c6c7c6ea1e50ac16a5aeffb3854d391b392d74a68c1bf767c608e0",
"docker.io/openshift/origin-control-plane:v3.11",
"docker.io/openshift/origin-control-plane:v3.11.0"
],
"sizeBytes": 828734067
},
{
"names": [
"docker.io/openshift/origin-control-plane@sha256:bf28e6729781f2d68c9ea121cee5c649a084ae9a4dadefded9321e2e381a2549"
],
"sizeBytes": 826232564
},
{
"names": [
"docker.io/openshift/origin-deployer@sha256:88eedcec3c877bea09653f037174f50356071e265d93889964a954a73b03aa8e",
"docker.io/openshift/origin-deployer:v3.11",
"docker.io/openshift/origin-deployer:v3.11.0"
],
"sizeBytes": 383673671
},
{
"names": [
"docker.io/openshift/origin-pod@sha256:91f0f130c991c2a6a9f0e9da43f7e56296bacdc46be0cd768125ed3e6f7ef09d",
"docker.io/openshift/origin-pod:v3.11.0"
],
"sizeBytes": 261526128
},
{
"names": [
"docker.io/openshift/origin-pod@sha256:67b51982ad05b446e4045b7e2507768c2d25b2eb61ccadd7b6a718b1262c25b8"
],
"sizeBytes": 258456390
},
{
"names": [
"quay.io/coreos/etcd@sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948",
"quay.io/coreos/etcd:v3.2.22"
],
"sizeBytes": 37269372
}
],
"nodeInfo": {
"architecture": "amd64",
"bootID": "c4c9e366-57c7-4334-933b-dab93702b583",
"containerRuntimeVersion": "docker://1.13.1",
"kernelVersion": "3.10.0-957.el7.x86_64",
"kubeProxyVersion": "v1.11.0+d4cacc0",
"kubeletVersion": "v1.11.0+d4cacc0",
"machineID": "bfee8bc7a3cd4673a31ac520eb963eb2",
"operatingSystem": "linux",
"osImage": "CentOS Linux 7 (Core)",
"systemUUID": "1C582A42-E6DF-7E8C-DBB8-DB6315262B0C"
}
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
},
"raw_failures": [],
"rc": 0,
"server_approve_results": [],
"server_csrs": null,
"state": "unknown",
"unwanted_csrs": [
{
"apiVersion": "certificates.k8s.io/v1beta1",
"kind": "CertificateSigningRequest",
"metadata": {
"creationTimestamp": "2019-03-22T00:06:21Z",
"generateName": "csr-",
"name": "csr-bsf6g",
"namespace": "",
"resourceVersion": "19872",
"selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-bsf6g",
"uid": "52f26e8c-4c36-11e9-add9-005056aa572c"
},
"spec": {
"groups": [
"system:masters",
"system:cluster-admins",
"system:authenticated"
],
"request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlJ6Q0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU2RzaEpwR1BvYlZrOFowdENJTDlpTE1tcjJwVklCY240bXZQdkM0RG4vSGNPQmJMTnQKRUJ6Q2NVV1pwQkRvOTh4UmlUWnludmErWXBJMFJLajBoTnAwb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnVGVnSnluemkwZVl1ZHE4cU9LR3hOaXc1YlZheGkyK2sKbHV3NGVNUzVwOFFDSVFEUWRsaTY0LzJ6cldQajlQNjVYWFlSMUR0ME1pMzNERnRnQURQTHdPNnFtQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=",
"usages": [
"digital signature",
"key encipherment",
"server auth"
],
"username": "system:admin"
},
"status": {
"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVQ3NzVlZMeTY4clhBcVhwbnczekEvQTg1TXRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFbmJJU2FSajZHMVpQR2RMUWlDL1lpekpxOXFWUwpBWEorSnJ6N3d1QTUveDNEZ1d5emJSQWN3bkZGbWFRUTZQZk1VWWsyY3A3MnZtS1NORVNvOUlUYWRLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVQTzJvQWlTaEc0bGlLZTRGSDFQTGltb0MwOHN3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUF3YmNITEVScHZTQkVZQUFUMlduVWlGOFppNE5rUGs5ZWRReisvcGJkeGNOeEIKUmZmSzFpRFNSYmRDNjh4ZjBHUjJxdklDWDdwMVhRUURueXJ3K2JvR2JUMUdxUVB4Z1RBTXdHVUtmcTFoMENWbQo5a0g5VE9sNTFWczRYZkhxZFZzdFA1Z3RUT0FpUFNzaWdVSmM3L3FKSHdNeHQ3aytNVmlJY1V1aUd1SjljRWFsClJ0TjZLeHBBTGx4WVA4aVZETTY4VTJISTgwalRrVFRjbW1yb2VzRnlPb1BtSlN5ZjRCNWExWWllcTB6SGNKcEwKYlRsSmpWTDl5QTRZQXdtQ0JzSmxyOCtVMlVweUhYY0hscERmV1ZvT1krRDhaQ2IzaFNRdnBlVkdJNE5OTCswOApQNVd4Ui9LK0xORElGTDVYR1ZDSG5rbm5IL041alZvNVEwUGY5bkZ5Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"conditions": [
{
"lastUpdateTime": "2019-03-22T00:06:42Z",
"message": "This CSR was approved by kubectl certificate approve.",
"reason": "KubectlApprove",
"type": "Approved"
}
]
}
},
{
"apiVersion": "certificates.k8s.io/v1beta1",
"kind": "CertificateSigningRequest",
"metadata": {
"creationTimestamp": "2019-03-22T00:04:51Z",
"generateName": "csr-",
"name": "csr-v85dj",
"namespace": "",
"resourceVersion": "19857",
"selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-v85dj",
"uid": "1d5cb3b4-4c36-11e9-add9-005056aa572c"
},
"spec": {
"groups": [
"system:masters",
"system:cluster-admins",
"system:authenticated"
],
"request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlNEQ0I3Z0lCQURCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1USW5ONQpjM1JsYlRwdWIyUmxPbXM0Y3kxamRHd3dNUzVrWlhZdWRHOXdhV0V1Ym5vd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxCmhrak9QUU1CQndOQ0FBU09yM1V1SXRpSmJSNU5QN2szOXBLa1FFZGRBQm01MHhBRGw2YmJtQ3VoaVMxZUt1dXMKbnNLU3VDakdRTEE1clhTYm9MN2FlZnJudzYra256Zys5WnNjb0Vnd1JnWUpLb1pJaHZjTkFRa09NVGt3TnpBMQpCZ05WSFJFRUxqQXNnaFpyT0hNdFkzUnNNREV1WkdWMkxuUnZjR2xoTG01NmdnQ0hCQW9FYjJXSEJLd1JBQUdICkJBcUFBQUV3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQVBFWlV4L0M0UHFncWhpKytBU3NCbWNCN3Z0SGNTcjMKUWJhL2dSS2hIUWwvQWlFQTJQbzVDK1lzQmFMcWlCMU1NcGQ4THFIS3d3ck0yL0NKczN0RHQyWDRwMkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=",
"usages": [
"digital signature",
"key encipherment",
"server auth"
],
"username": "system:admin"
},
"status": {
"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1akNDQWFLZ0F3SUJBZ0lVTFVQcFY4eitRRHZseDhpa3AydjBpRTA2citRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRWtNQ0lHQTFVRUF3d2JiM0JsYm5Ob2FXWjBMWE5wWjI1bGNrQXhOVFV6TVRJNE56UTFNQjRYRFRFNQpNRE15TWpBd01ESXdNRm9YRFRJd01ETXlNVEF3TURJd01Gb3dSREVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2ClpHVnpNU3N3S1FZRFZRUURFeUp6ZVhOMFpXMDZibTlrWlRwck9ITXRZM1JzTURFdVpHVjJMblJ2Y0dsaExtNTYKTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFanE5MUxpTFlpVzBlVFQrNU4vYVNwRUJIWFFBWgp1ZE1RQTVlbTI1Z3JvWWt0WGlycnJKN0NrcmdveGtDd09hMTBtNkMrMm5uNjU4T3ZwSjg0UHZXYkhLT0JqRENCCmlUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVVwRGtRRjNXS3ArdmlhSmdlVWlBbHJPRXNueWt3TlFZRFZSMFJCQzR3TElJVwphemh6TFdOMGJEQXhMbVJsZGk1MGIzQnBZUzV1ZW9JQWh3UUtCRzlsaHdTc0VRQUJod1FLZ0FBQk1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUNDL0JJL0hKYXBhWG1kTUpsd2MvejhlMkRiazBVSWp3UlhjM0lET0FxckRoeGsKR2VOd0tNcUJJc2Fic1pWcXlzMWtqSVBaS0VFWTdwVHFhZU54blFRT2FucWZYMmNuZTV1aGEvbUo0U2NmRUorQQpQOGJaSU1GRzVIS1FPNlBPOSt5SXhKR3I4emFpM1hsd2s0YUNGZ2xCWEQ3SzRsZ0Noc0xXZjdZQWtKcmRKRGEvCkRhNmpzbFQvQithZlpKbmJZeUJSZkNqeVdkUE5lbXdsU1V2ZS9SZ3Z6eXZsMllmTFlpMWJ3bmw2RXNrTW84aWoKUVdBR0tpd3FkUnNCdGNOSStJY1JHYzFDZWY4ZFFhdGVGWFl6c2VINHdONWpwRWZoYllMOVRCbEJ1QnQwWVpZOQpJTlZxUVRGVEZQVzBJWmZQcGU4ZXhUOFR3TUJlRjBoS05uUk03blZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"conditions": [
{
"lastUpdateTime": "2019-03-22T00:06:36Z",
"message": "This CSR was approved by kubectl certificate approve.",
"reason": "KubectlApprove",
"type": "Approved"
}
]
}
}
]
}
to retry, use: --limit @/home/user/git/sourcetopia/openshift-install/openshift/ansible/playbooks/deploy_cluster.retry
PLAY RECAP **********************************************************************************************************************************************************************
k8s-ctl01.dev.mydomain.com : ok=462 changed=85 unreachable=0 failed=1
k8s-worker01.dev.mydomain.com : ok=100 changed=11 unreachable=0 failed=0
k8s-worker02.dev.mydomain.com : ok=98 changed=11 unreachable=0 failed=0
k8s-worker03.dev.mydomain.com : ok=98 changed=11 unreachable=0 failed=0
localhost : ok=11 changed=0 unreachable=0 failed=0
INSTALLER STATUS ****************************************************************************************************************************************************************
Initialization : Complete (0:00:30)
Health Check : Complete (0:02:50)
Node Bootstrap Preparation : Complete (0:02:50)
etcd Install : Complete (0:00:29)
Master Install : Complete (0:03:52)
Master Additional Install : Complete (0:00:29)
Node Join : In Progress (0:03:22)
This phase can be restarted by running: playbooks/openshift-node/join.yml
Failure summary:
1. Hosts: k8s-ctl01.dev.mydomain.com
Play: Approve any pending CSR requests from inventory nodes
Task: Approve node certificates when bootstrapping
Message: Could not find csr for nodes: k8s-worker03.dev.mydomain.com, k8s-worker02.dev.mydomain.com, k8s-worker01.dev.mydomain.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment