Annotated history of an upgrade
$ oc patch clusterversion/version --patch '{"spec":{"upstream":"https://origin-release.svc.ci.openshift.org/graph"}}' --type=merge
$ oc adm upgrade --to-latest
# CVO sees new available version, creates a job that can pull the image and extract contents across a hostPath
Dec 27 16:05:05.617 I ns=openshift-cluster-version job=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw Created pod: version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr
Dec 27 16:05:07.539 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr pulling image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
Dec 27 16:05:09.688 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
Dec 27 16:05:09.959 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr Created container
Dec 27 16:05:09.993 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr Started container
# CVO starts applying the manifests from the job, which includes its new deployment, which triggers a new scale up
Dec 27 16:05:11.052 I ns=openshift-cluster-version deployment=cluster-version-operator Scaled up replica set cluster-version-operator-7f55954567 to 1
Dec 27 16:05:11.089 I ns=openshift-cluster-version replicaset=cluster-version-operator-7f55954567 Created pod: cluster-version-operator-7f55954567-jpf9h
Dec 27 16:05:11.121 I ns=openshift-cluster-version pod=cluster-version-operator-7f55954567-jpf9h Successfully assigned openshift-cluster-version/cluster-version-operator-7f55954567-jpf9h to ip-10-0-12-206.ec2.internal
Dec 27 16:05:11.839 I ns=openshift-cluster-version pod=cluster-version-operator-7f55954567-jpf9h pulling image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
Dec 27 16:05:18.802 I ns=openshift-cluster-version pod=cluster-version-operator-7f55954567-jpf9h Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
Dec 27 16:05:18.982 I ns=openshift-cluster-version pod=cluster-version-operator-7f55954567-jpf9h Created container
Dec 27 16:05:19.019 I ns=openshift-cluster-version pod=cluster-version-operator-7f55954567-jpf9h Started container
Dec 27 16:05:19.923 I ns=openshift-cluster-version deployment=cluster-version-operator Scaled down replica set cluster-version-operator-998dc55cf to 0
Dec 27 16:05:19.924 W ns=openshift-cluster-version pod=cluster-version-operator-998dc55cf-9bx5g node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:05:19.952 I ns=openshift-cluster-version replicaset=cluster-version-operator-998dc55cf Deleted pod: cluster-version-operator-998dc55cf-9bx5g
Dec 27 16:05:20.109 I ns=openshift-cluster-version pod=cluster-version-operator-998dc55cf-9bx5g Killing container with id cri-o://cluster-version-operator:Need to kill Pod
Dec 27 16:05:20.543 W ns=openshift-cluster-version pod=cluster-version-operator-998dc55cf-9bx5g node=ip-10-0-30-101.ec2.internal container=cluster-version-operator container stopped being ready
Dec 27 16:05:21.505 W ns=openshift-cluster-version pod=cluster-version-operator-998dc55cf-9bx5g node=ip-10-0-30-101.ec2.internal deleted
# Wait 2 minutes for the first CVO pod's lease to expire (https://github.com/kubernetes/kubernetes/pull/71490 will allow us to remove this)
# New CVO also fetches and extracts contents (QUESTIONABLE: it should have the local ones it needs, probably this is due to hash equivalence on image)
Dec 27 16:07:15.060 I ns=openshift-cluster-version job=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9 Created pod: version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht
Dec 27 16:07:16.768 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht Container image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801" already present on machine
Dec 27 16:07:17.019 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht Created container
Dec 27 16:07:17.057 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht Started container
Dec 27 16:07:18.164 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht Pod sandbox changed, it will be killed and re-created.
# First operator is updated
Dec 27 16:07:19.690 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Deleted pod: cluster-network-operator-tsp2h
Dec 27 16:07:19.691 W ns=openshift-cluster-network-operator pod=cluster-network-operator-tsp2h node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:07:19.871 I ns=openshift-cluster-network-operator pod=cluster-network-operator-tsp2h Killing container with id cri-o://cluster-network-operator:Need to kill Pod
Dec 27 16:07:20.233 W ns=openshift-cluster-network-operator pod=cluster-network-operator-tsp2h node=ip-10-0-12-206.ec2.internal container=cluster-network-operator container stopped being ready
Dec 27 16:07:21.226 W ns=openshift-cluster-network-operator pod=cluster-network-operator-tsp2h node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:07:21.314 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Created pod: cluster-network-operator-rtvtz
Dec 27 16:07:23.940 W ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht Failed create pod sandbox: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_version-4.0.0-0.alpha-2018-12-26-225801-xqmk9-7b7ht_openshift-cluster-version_82c50999-09f1-11e9-b3b2-0a7ded5c62ec_1(b5852141a611c1fe0b03c5d232b8a41266062510cb2f6af4181b7c507a9a7090): Unexpected command output nsenter: cannot open /proc/26714/ns/net: No such file or directory
with error: exit status 1
Dec 27 16:07:26.291 I ns=openshift-cluster-network-operator pod=cluster-network-operator-rtvtz pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:07:28.128 I ns=openshift-cluster-network-operator pod=cluster-network-operator-rtvtz Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:07:31.716 I ns=openshift-cluster-network-operator pod=cluster-network-operator-rtvtz Created container
Dec 27 16:07:31.717 I ns=openshift-cluster-network-operator pod=cluster-network-operator-rtvtz Started container
Dec 27 16:07:32.251 W ns=openshift-cluster-network-operator pod=cluster-network-operator-8wf6q node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:07:32.281 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Deleted pod: cluster-network-operator-8wf6q
Dec 27 16:07:32.429 I ns=openshift-cluster-network-operator pod=cluster-network-operator-8wf6q Killing container with id cri-o://cluster-network-operator:Need to kill Pod
Dec 27 16:07:32.921 W ns=openshift-cluster-network-operator pod=cluster-network-operator-8wf6q node=ip-10-0-30-101.ec2.internal container=cluster-network-operator container stopped being ready
Dec 27 16:07:38.800 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Created pod: cluster-network-operator-kb8tx
Dec 27 16:07:38.800 W ns=openshift-cluster-network-operator pod=cluster-network-operator-8wf6q node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:07:44.213 I ns=openshift-cluster-network-operator pod=cluster-network-operator-kb8tx pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:07:45.897 I ns=openshift-cluster-network-operator pod=cluster-network-operator-kb8tx Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:07:49.990 I ns=openshift-cluster-network-operator pod=cluster-network-operator-kb8tx Created container
Dec 27 16:07:49.990 I ns=openshift-cluster-network-operator pod=cluster-network-operator-kb8tx Started container
Dec 27 16:07:50.966 W ns=openshift-cluster-network-operator pod=cluster-network-operator-9rxw9 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:07:51.143 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Deleted pod: cluster-network-operator-9rxw9
Dec 27 16:07:51.147 I ns=openshift-cluster-network-operator pod=cluster-network-operator-9rxw9 Killing container with id cri-o://cluster-network-operator:Need to kill Pod
Dec 27 16:07:52.197 W ns=openshift-cluster-network-operator pod=cluster-network-operator-9rxw9 node=ip-10-0-33-43.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:07:52.197 W ns=openshift-cluster-network-operator pod=cluster-network-operator-9rxw9 node=ip-10-0-33-43.ec2.internal container=cluster-network-operator container stopped being ready
Dec 27 16:07:53.088 W ns=openshift-sdn pod=sdn-controller-zwgm5 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:07:53.261 W ns=openshift-cluster-network-operator pod=cluster-network-operator-9rxw9 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:07:53.287 I ns=openshift-sdn daemonset=sdn-controller Deleted pod: sdn-controller-zwgm5
Dec 27 16:07:53.308 I ns=openshift-cluster-network-operator daemonset=cluster-network-operator Created pod: cluster-network-operator-mtwr6
Dec 27 16:07:53.571 W ns=openshift-sdn pod=ovs-rwv97 node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:07:53.586 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-rwv97
Dec 27 16:07:54.248 W ns=openshift-sdn pod=sdn-kqdtl node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:07:54.253 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-kqdtl
Dec 27 16:07:54.657 I ns=openshift-sdn pod=sdn-kqdtl Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:07:55.164 W ns=openshift-sdn pod=sdn-kqdtl node=ip-10-0-33-43.ec2.internal container=sdn container stopped being ready
Dec 27 16:07:58.570 I ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:07:59.959 I ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530"
Dec 27 16:08:03.259 I ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 Created container
Dec 27 16:08:03.398 I ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 Started container
Dec 27 16:08:03.517 W ns=openshift-sdn pod=sdn-kqdtl node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:08:03.733 I ns=openshift-sdn daemonset=sdn Created pod: sdn-jf9sk
Dec 27 16:08:03.868 I ns=openshift-sdn pod=ovs-rwv97 Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:08:04.283 I ns=openshift-sdn pod=sdn-jf9sk pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:04.283 W ns=openshift-sdn pod=ovs-rwv97 node=ip-10-0-135-20.ec2.internal container=openvswitch container stopped being ready
# DNS operator starts deploying, but SDN is still busy being restarted
# BUG: SDN needs to wait for deployment to be rolled out
# BUG: having to restart the pods is super bad, if we can't fix the need to restart we need to make this only happen when the nodes are drained (later)
#
Dec 27 16:08:04.905 I ns=openshift-dns-operator deployment=dns-operator Scaled up replica set dns-operator-6fbc89d8dc to 1
Dec 27 16:08:04.917 I ns=openshift-dns-operator replicaset=dns-operator-6fbc89d8dc Created pod: dns-operator-6fbc89d8dc-blcxm
Dec 27 16:08:05.092 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Successfully assigned openshift-dns-operator/dns-operator-6fbc89d8dc-blcxm to ip-10-0-161-51.ec2.internal
Dec 27 16:08:05.394 W ns=openshift-sdn pod=ovs-rwv97 node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:08:05.408 I ns=openshift-sdn daemonset=ovs Created pod: ovs-lfrwb
Dec 27 16:08:05.848 I ns=openshift-sdn pod=sdn-jf9sk Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:05.857 I ns=openshift-sdn pod=sdn-jf9sk Created container
Dec 27 16:08:06.032 I ns=openshift-sdn pod=sdn-jf9sk Started container
Dec 27 16:08:06.046 I ns=openshift-sdn pod=ovs-lfrwb pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:06.409 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-g8hg9
Dec 27 16:08:06.410 W ns=openshift-sdn pod=sdn-g8hg9 node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:08:06.751 I ns=openshift-sdn pod=sdn-g8hg9 Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:08:07.123 W ns=openshift-sdn pod=sdn-g8hg9 node=ip-10-0-153-178.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:07.324 I node=ip-10-0-33-43.ec2.internal Starting openshift-sdn.
Dec 27 16:08:07.462 I ns=openshift-sdn pod=ovs-lfrwb Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:07.462 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:07eab41e0452f0478cc614b32da4fbc1676c22c638431f3168d13c73c648f0f3"
# Router starts failing health checks, probably due to SDN restart
Dec 27 16:08:07.643 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:07.795 I ns=openshift-sdn pod=ovs-lfrwb Created container
Dec 27 16:08:07.976 I ns=openshift-sdn pod=ovs-lfrwb Started container
Dec 27 16:08:07.978 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:08.175 W ns=openshift-sdn pod=sdn-wsskg node=ip-10-0-135-20.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:08.351 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-bszj9
Dec 27 16:08:08.352 W ns=openshift-sdn pod=ovs-bszj9 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:08:08.547 I ns=openshift-sdn pod=sdn-wsskg Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:08.739 I ns=openshift-sdn pod=sdn-wsskg Created container
Dec 27 16:08:08.895 I ns=openshift-sdn pod=sdn-wsskg Started container
Dec 27 16:08:09.451 W ns=openshift-sdn pod=sdn-wsskg node=ip-10-0-135-20.ec2.internal container=sdn container restarted
Dec 27 16:08:10.116 W ns=openshift-dns pod=dns-default-h82v2 Liveness probe failed: Get http://10.128.2.2:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:11.156 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:11.450 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:07eab41e0452f0478cc614b32da4fbc1676c22c638431f3168d13c73c648f0f3"
Dec 27 16:08:11.453 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:11.712 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Created container
Dec 27 16:08:11.739 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Started container
Dec 27 16:08:11.805 W ns=openshift-dns pod=dns-default-h82v2 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:12.374 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf node=ip-10-0-135-20.ec2.internal container=cluster-storage-operator container stopped being ready
Dec 27 16:08:12.423 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq node=ip-10-0-135-20.ec2.internal container=cluster-node-tuning-operator container stopped being ready
Dec 27 16:08:12.442 I ns=openshift-dns-operator deployment=dns-operator Scaled down replica set dns-operator-9b99c5887 to 0
Dec 27 16:08:12.451 W ns=openshift-dns-operator pod=dns-operator-9b99c5887-d2zkd node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:08:12.485 I ns=openshift-dns-operator replicaset=dns-operator-9b99c5887 Deleted pod: dns-operator-9b99c5887-d2zkd
# Should probably be a better error message (this is due to SDN restart)
Dec 27 16:08:12.518 I ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:12.539 I ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:12.679 I ns=openshift-dns-operator pod=dns-operator-9b99c5887-d2zkd Killing container with id cri-o://dns-operator:Need to kill Pod
Dec 27 16:08:12.988 W ns=openshift-dns-operator pod=dns-operator-9b99c5887-d2zkd node=ip-10-0-30-101.ec2.internal container=dns-operator container stopped being ready
Dec 27 16:08:13.117 I ns=openshift-core-operators deployment=origin-cluster-osin-operator Scaled up replica set origin-cluster-osin-operator-86df5958b to 1
Dec 27 16:08:13.157 I ns=openshift-core-operators replicaset=origin-cluster-osin-operator-86df5958b Created pod: origin-cluster-osin-operator-86df5958b-qfk4c
Dec 27 16:08:13.169 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Successfully assigned openshift-core-operators/origin-cluster-osin-operator-86df5958b-qfk4c to ip-10-0-33-43.ec2.internal
Dec 27 16:08:13.946 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:08:14.572 I ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:dff3065259f98d617d536a83e571ef9256e4400688339c88d108253a773b4d15" already present on machine
Dec 27 16:08:14.630 I ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf Created container
Dec 27 16:08:14.642 I ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf Started container
Dec 27 16:08:14.969 I ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:43879a8071ff4b24f4d45d137c96cfbf6e1ab4501ddf21fa5ffb52e6babd3252"
Dec 27 16:08:15.150 W ns=openshift-dns pod=dns-default-h82v2 Liveness probe failed: Get http://10.128.2.2:8080/health: dial tcp 10.128.2.2:8080: connect: no route to host
Dec 27 16:08:15.261 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf node=ip-10-0-135-20.ec2.internal container=cluster-storage-operator container restarted
Dec 27 16:08:15.340 I ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:43879a8071ff4b24f4d45d137c96cfbf6e1ab4501ddf21fa5ffb52e6babd3252"
Dec 27 16:08:15.561 I ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq Created container
Dec 27 16:08:15.609 I ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq Started container
Dec 27 16:08:15.913 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91"
Dec 27 16:08:16.469 I ns=openshift-sdn pod=ovs-bszj9 Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:08:16.469 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq node=ip-10-0-135-20.ec2.internal container=cluster-node-tuning-operator container restarted
Dec 27 16:08:16.633 W ns=openshift-sdn pod=sdn-g8hg9 node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:08:16.698 I ns=openshift-sdn daemonset=sdn Created pod: sdn-6wmck
Dec 27 16:08:17.041 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:17.052 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:17.223 I ns=openshift-sdn pod=sdn-6wmck pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:17.364 W ns=openshift-sdn pod=ovs-bszj9 node=ip-10-0-33-43.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:08:17.364 W ns=openshift-sdn pod=ovs-bszj9 node=ip-10-0-33-43.ec2.internal container=openvswitch container stopped being ready
Dec 27 16:08:18.516 I ns=openshift-sdn pod=sdn-6wmck Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:18.628 I ns=openshift-sdn pod=sdn-6wmck Created container
Dec 27 16:08:18.648 I ns=openshift-sdn pod=sdn-6wmck Started container
Dec 27 16:08:19.017 W ns=openshift-dns-operator pod=dns-operator-9b99c5887-d2zkd node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:08:19.156 W ns=openshift-sdn pod=sdn-s5zzm node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:08:19.193 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-s5zzm
Dec 27 16:08:19.662 I ns=openshift-sdn pod=sdn-s5zzm Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:08:19.961 I node=ip-10-0-153-178.ec2.internal Starting openshift-sdn.
Dec 27 16:08:20.462 W ns=openshift-sdn pod=sdn-s5zzm node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:21.448 W ns=openshift-apiserver pod=apiserver-wsjnb Liveness probe failed: Get https://10.129.0.16:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:21.462 W ns=openshift-sdn pod=sdn-s5zzm node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:08:21.471 I ns=openshift-sdn daemonset=sdn Created pod: sdn-z6fp6
Dec 27 16:08:22.014 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Liveness probe failed: Get https://10.129.0.15:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:22.016 I ns=openshift-sdn pod=sdn-z6fp6 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:22.366 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:22.367 W ns=openshift-dns pod=dns-default-h82v2 node=ip-10-0-135-20.ec2.internal container=dns container stopped being ready
Dec 27 16:08:22.367 W ns=openshift-dns pod=dns-default-h82v2 node=ip-10-0-135-20.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:08:22.706 I ns=openshift-dns pod=dns-default-h82v2 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:23.079 W ns=openshift-sdn pod=ovs-bszj9 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:08:23.088 I ns=openshift-sdn daemonset=ovs Created pod: ovs-84v8f
Dec 27 16:08:23.245 I ns=openshift-sdn pod=sdn-z6fp6 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:23.427 I ns=openshift-sdn pod=sdn-z6fp6 Created container
Dec 27 16:08:23.427 I ns=openshift-sdn pod=sdn-z6fp6 Started container
Dec 27 16:08:23.580 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-7r4n5
Dec 27 16:08:23.580 W ns=openshift-sdn pod=sdn-7r4n5 node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:08:23.939 I ns=openshift-sdn pod=sdn-7r4n5 Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:08:24.119 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:08:24.252 W ns=openshift-sdn pod=sdn-7r4n5 node=ip-10-0-161-51.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:24.334 I ns=openshift-dns pod=dns-default-h82v2 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:08:24.398 I node=ip-10-0-12-206.ec2.internal Starting openshift-sdn.
Dec 27 16:08:24.478 I ns=openshift-dns pod=dns-default-h82v2 Created container
Dec 27 16:08:24.522 I ns=openshift-dns pod=dns-default-h82v2 Started container
Dec 27 16:08:24.540 I ns=openshift-dns pod=dns-default-h82v2 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9c116e0a8c006f3c67f074a0372c6b4f977161386a95d41aa0faabbe94a558f0" already present on machine
Dec 27 16:08:24.673 I ns=openshift-dns pod=dns-default-h82v2 Created container
Dec 27 16:08:24.698 I ns=openshift-dns pod=dns-default-h82v2 Started container
Dec 27 16:08:25.170 W ns=openshift-apiserver pod=apiserver-wsjnb Readiness probe failed: Get https://10.129.0.16:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:25.181 I ns=openshift-sdn pod=sdn-controller-zwgm5 Killing container with id cri-o://sdn-controller:Need to kill Pod
Dec 27 16:08:25.278 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91"
Dec 27 16:08:25.329 W ns=openshift-dns pod=dns-default-h82v2 node=ip-10-0-135-20.ec2.internal container=dns container restarted
Dec 27 16:08:25.329 W ns=openshift-dns pod=dns-default-h82v2 node=ip-10-0-135-20.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:08:25.685 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Created container
Dec 27 16:08:25.707 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Started container
Dec 27 16:08:25.719 I ns=openshift-sdn pod=ovs-84v8f Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:26.116 I ns=openshift-sdn pod=ovs-84v8f Created container
Dec 27 16:08:26.159 I ns=openshift-sdn pod=ovs-84v8f Started container
Dec 27 16:08:26.252 W ns=openshift-sdn pod=sdn-jf9sk node=ip-10-0-33-43.ec2.internal container=sdn container stopped being ready
# This is a bug that needs to be filed upstream, but the pod status phase is set to Pending during graceful deletion and it should not be (should go to a terminal phase)
Dec 27 16:08:26.414 W ns=openshift-sdn pod=sdn-controller-zwgm5 node=ip-10-0-33-43.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:08:26.414 W ns=openshift-sdn pod=sdn-controller-zwgm5 node=ip-10-0-33-43.ec2.internal container=sdn-controller container stopped being ready
Dec 27 16:08:26.568 I ns=openshift-sdn pod=sdn-jf9sk Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:26.588 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Readiness probe failed: Get https://10.129.0.15:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:26.701 W ns=openshift-sdn pod=ovs-9qkc6 node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:08:26.803 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-9qkc6
Dec 27 16:08:26.840 I ns=openshift-sdn pod=sdn-jf9sk Created container
Dec 27 16:08:26.862 I ns=openshift-core-operators deployment=origin-cluster-osin-operator Scaled down replica set origin-cluster-osin-operator-6f67b69bfd to 0
Dec 27 16:08:26.868 I ns=openshift-sdn pod=sdn-jf9sk Started container
Dec 27 16:08:27.382 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:27.457 W ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:08:27.469 W ns=openshift-sdn pod=sdn-jf9sk node=ip-10-0-33-43.ec2.internal container=sdn container restarted
Dec 27 16:08:27.835 I ns=openshift-core-operators replicaset=origin-cluster-osin-operator-6f67b69bfd Deleted pod: origin-cluster-osin-operator-6f67b69bfd-wtlzr
Dec 27 16:08:27.835 I ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr Killing container with id cri-o://operator:Need to kill Pod
Dec 27 16:08:27.883 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:27.987 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Killing container with id cri-o://router:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:08:28.016 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:7a32d6d2d8477afab8cea56ee629b777a6c6bf54bd861b326b2e1861a616bd8c" already present on machine
Dec 27 16:08:28.347 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Created container
Dec 27 16:08:28.488 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Started container
Dec 27 16:08:28.498 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:08:28.503 W ns=openshift-sdn pod=sdn-controller-zwgm5 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:08:28.527 W ns=openshift-sdn pod=sdn-7r4n5 node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:08:28.584 I ns=openshift-sdn daemonset=sdn Created pod: sdn-76z7n
Dec 27 16:08:28.633 I ns=openshift-sdn daemonset=sdn-controller Created pod: sdn-controller-hs226
Dec 27 16:08:29.371 I ns=openshift-sdn pod=sdn-76z7n pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:29.371 W ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr node=ip-10-0-30-101.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:08:29.371 W ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:08:29.525 I ns=openshift-core-operators deployment=origin-cluster-osin-operator2 Scaled up replica set origin-cluster-osin-operator2-5d7bf9d7d to 1
Dec 27 16:08:29.582 I ns=openshift-core-operators replicaset=origin-cluster-osin-operator2-5d7bf9d7d Created pod: origin-cluster-osin-operator2-5d7bf9d7d-wqdxw
Dec 27 16:08:29.638 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Successfully assigned openshift-core-operators/origin-cluster-osin-operator2-5d7bf9d7d-wqdxw to ip-10-0-30-101.ec2.internal
Dec 27 16:08:29.704 W ns=openshift-apiserver pod=apiserver-wsjnb The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:29.865 I ns=openshift-sdn pod=sdn-controller-hs226 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:08:30.039 W ns=openshift-dns pod=dns-default-g27zs Liveness probe failed: Get http://10.129.0.2:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:30.103 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:30.395 I ns=openshift-sdn pod=sdn-76z7n Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:30.396 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container stopped being ready
Dec 27 16:08:30.488 I ns=openshift-sdn pod=sdn-76z7n Created container
Dec 27 16:08:30.805 I ns=openshift-sdn pod=sdn-76z7n Started container
Dec 27 16:08:30.895 I ns=openshift-apiserver pod=apiserver-wsjnb Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:30.914 W ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr node=ip-10-0-30-101.ec2.internal pod has been pending longer than a minute
Dec 27 16:08:31.082 W ns=openshift-controller-manager pod=controller-manager-2qgd4 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:31.110 I ns=openshift-sdn pod=sdn-controller-hs226 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:08:31.511 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 node=ip-10-0-33-43.ec2.internal container=console container stopped being ready
Dec 27 16:08:31.582 I ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:31.601 I ns=openshift-sdn pod=sdn-controller-hs226 Created container
Dec 27 16:08:31.634 I ns=openshift-sdn pod=sdn-controller-hs226 Started container
Dec 27 16:08:31.971 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Liveness probe failed: Get https://10.129.0.15:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:32.098 W ns=openshift-sdn pod=sdn-wsskg node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:08:32.223 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-wsskg
Dec 27 16:08:32.449 I ns=openshift-sdn configmap=openshift-network-controller ip-10-0-33-43 became leader
Dec 27 16:08:32.483 I ns=openshift-sdn pod=sdn-wsskg Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:08:32.550 W ns=openshift-sdn pod=sdn-controller-p5xc2 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:08:32.682 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91"
Dec 27 16:08:32.682 I ns=openshift-sdn daemonset=sdn-controller Deleted pod: sdn-controller-p5xc2
Dec 27 16:08:33.366 I ns=openshift-apiserver pod=apiserver-wsjnb pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:08:33.366 I node=ip-10-0-161-51.ec2.internal Starting openshift-sdn.
Dec 27 16:08:33.393 W ns=openshift-sdn pod=sdn-wsskg node=ip-10-0-135-20.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:33.478 I ns=openshift-apiserver pod=apiserver-wsjnb Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:08:33.877 I ns=openshift-apiserver pod=apiserver-wsjnb Created container
Dec 27 16:08:34.184 I ns=openshift-apiserver pod=apiserver-wsjnb Started container
Dec 27 16:08:34.371 I ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea" already present on machine
Dec 27 16:08:34.422 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91"
Dec 27 16:08:34.607 I ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:08:34.620 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 node=ip-10-0-135-20.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:08:34.638 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:08:34.712 I ns=kube-system configmap=openshift-master-controllers controller-manager-qbf49 became leader
Dec 27 16:08:34.809 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Created container
Dec 27 16:08:35.718 I ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 Created container
Dec 27 16:08:35.718 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Started container
Dec 27 16:08:35.901 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j node=ip-10-0-135-20.ec2.internal container=grafana-proxy container stopped being ready
Dec 27 16:08:36.109 I ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 Started container
Dec 27 16:08:36.359 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:08:36.378 I ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Created container
Dec 27 16:08:36.776 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-697b68b5c4-q9ncl node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:08:36.953 I ns=openshift-core-operators deployment=origin-cluster-osin-operator2 Scaled down replica set origin-cluster-osin-operator2-697b68b5c4 to 0
Dec 27 16:08:36.953 I ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Started container
Dec 27 16:08:37.229 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-697b68b5c4-q9ncl Killing container with id cri-o://operator:Need to kill Pod
Dec 27 16:08:37.246 W ns=openshift-sdn pod=sdn-wsskg node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:08:37.321 I ns=openshift-core-operators replicaset=origin-cluster-osin-operator2-697b68b5c4 Deleted pod: origin-cluster-osin-operator2-697b68b5c4-q9ncl
Dec 27 16:08:37.321 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 node=ip-10-0-135-20.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:08:37.419 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 node=ip-10-0-33-43.ec2.internal container=console container restarted
Dec 27 16:08:37.603 I ns=openshift-sdn daemonset=sdn Created pod: sdn-lltpr
Dec 27 16:08:37.608 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-697b68b5c4-q9ncl node=ip-10-0-12-206.ec2.internal container=operator container stopped being ready
Dec 27 16:08:37.615 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91" already present on machine
Dec 27 16:08:37.783 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c node=ip-10-0-33-43.ec2.internal container=operator container stopped being ready
Dec 27 16:08:37.998 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Created container
Dec 27 16:08:38.144 W ns=openshift-dns pod=dns-default-g27zs Liveness probe failed: Get http://10.129.0.2:8080/health: dial tcp 10.129.0.2:8080: connect: no route to host
Dec 27 16:08:38.303 I ns=openshift-sdn pod=sdn-lltpr Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:38.303 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Started container
Dec 27 16:08:38.483 I ns=openshift-sdn pod=sdn-lltpr Created container
Dec 27 16:08:38.483 I ns=openshift-sdn pod=sdn-lltpr Started container
Dec 27 16:08:38.665 I ns=openshift-sdn daemonset=sdn Deleted pod: sdn-7v4wg
Dec 27 16:08:38.666 W ns=openshift-sdn pod=sdn-7v4wg node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:08:39.537 I ns=openshift-sdn pod=sdn-7v4wg Killing container with id cri-o://sdn:Need to kill Pod
Dec 27 16:08:39.733 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c node=ip-10-0-33-43.ec2.internal container=operator container restarted
Dec 27 16:08:39.733 I ns=openshift-core-operators deployment=openshift-service-cert-signer-operator Scaled up replica set openshift-service-cert-signer-operator-78447d9bb8 to 1
Dec 27 16:08:40.011 I ns=openshift-core-operators replicaset=openshift-service-cert-signer-operator-78447d9bb8 Created pod: openshift-service-cert-signer-operator-78447d9bb8-szxl4
Dec 27 16:08:40.021 W ns=openshift-core-operators pod=origin-cluster-osin-operator-6f67b69bfd-wtlzr node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:08:40.112 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Successfully assigned openshift-core-operators/openshift-service-cert-signer-operator-78447d9bb8-szxl4 to ip-10-0-12-206.ec2.internal
Dec 27 16:08:40.326 W ns=openshift-sdn pod=sdn-7v4wg node=ip-10-0-30-101.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:08:40.326 W ns=openshift-sdn pod=sdn-7v4wg node=ip-10-0-30-101.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:40.448 I node=ip-10-0-135-20.ec2.internal Starting openshift-sdn.
Dec 27 16:08:41.079 I ns=openshift-sdn pod=ovs-9qkc6 Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:08:41.455 W ns=openshift-controller-manager pod=controller-manager-2qgd4 node=ip-10-0-33-43.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:08:41.567 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:41.579 W ns=openshift-sdn pod=sdn-7v4wg node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:08:41.728 I ns=openshift-sdn daemonset=sdn Created pod: sdn-448jh
Dec 27 16:08:41.828 I ns=openshift-controller-manager pod=controller-manager-2qgd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e" already present on machine
# DEBUG: What is this?
Dec 27 16:08:42.306 W ns=openshift-controller-manager pod=controller-manager-2qgd4 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:08:42.979 W ns=openshift-dns pod=dns-default-g27zs The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:42.981 I ns=openshift-controller-manager pod=controller-manager-2qgd4 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:43.014 W ns=openshift-sdn pod=ovs-9qkc6 node=ip-10-0-161-51.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:08:43.014 W ns=openshift-sdn pod=ovs-9qkc6 node=ip-10-0-161-51.ec2.internal container=openvswitch container stopped being ready
Dec 27 16:08:43.275 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750"
Dec 27 16:08:43.324 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:43.412 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c node=ip-10-0-33-43.ec2.internal container=operator container stopped being ready
Dec 27 16:08:43.510 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:43.521 I ns=openshift-sdn pod=sdn-448jh pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:43.681 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-697b68b5c4-q9ncl node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:08:44.048 W ns=openshift-controller-manager pod=controller-manager-2qgd4 Back-off restarting failed container
Dec 27 16:08:44.220 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:08:44.310 I ns=openshift-sdn pod=sdn-448jh Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6"
Dec 27 16:08:44.677 I ns=openshift-sdn pod=sdn-448jh Created container
Dec 27 16:08:44.771 I ns=openshift-sdn pod=sdn-448jh Started container
Dec 27 16:08:45.914 W ns=openshift-sdn pod=ovs-9qkc6 node=ip-10-0-161-51.ec2.internal pod has been pending longer than a minute
Dec 27 16:08:46.311 W ns=openshift-controller-manager pod=controller-manager-2qgd4 Back-off restarting failed container
Dec 27 16:08:46.387 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Back-off restarting failed container
Dec 27 16:08:46.882 W ns=openshift-apiserver pod=apiserver-wsjnb Readiness probe failed: Get https://10.129.0.20:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:47.291 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Back-off restarting failed container
Dec 27 16:08:47.291 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:47.291 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:08:47.549 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Back-off restarting failed container
Dec 27 16:08:47.642 I node=ip-10-0-30-101.ec2.internal Starting openshift-sdn.
Dec 27 16:08:48.735 W ns=openshift-sdn pod=ovs-9qkc6 node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:08:49.355 W ns=openshift-dns pod=dns-default-xtj2b Liveness probe failed: Get http://10.129.2.2:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:49.356 I ns=openshift-sdn daemonset=ovs Created pod: ovs-z84hz
Dec 27 16:08:49.483 W ns=openshift-dns pod=dns-default-g27zs Liveness probe failed: Get http://10.129.0.2:8080/health: dial tcp 10.129.0.2:8080: connect: no route to host
Dec 27 16:08:49.547 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:08:50.008 I ns=openshift-sdn pod=ovs-z84hz Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:50.363 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:08:50.363 I ns=openshift-sdn pod=ovs-z84hz Created container
Dec 27 16:08:50.363 I ns=openshift-sdn pod=ovs-z84hz Started container
Dec 27 16:08:50.452 W ns=openshift-sdn pod=ovs-l9kq7 node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:08:50.508 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-l9kq7
Dec 27 16:08:50.875 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:08:51.798 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750"
Dec 27 16:08:51.812 W ns=openshift-sdn pod=sdn-76z7n node=ip-10-0-161-51.ec2.internal container=sdn container stopped being ready
Dec 27 16:08:51.988 I ns=openshift-sdn pod=sdn-76z7n Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:08:52.437 I ns=openshift-sdn pod=sdn-76z7n Created container
Dec 27 16:08:52.438 W ns=openshift-sdn pod=sdn-76z7n node=ip-10-0-161-51.ec2.internal container=sdn container restarted
Dec 27 16:08:52.723 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Created container
Dec 27 16:08:52.723 I ns=openshift-sdn pod=sdn-76z7n Started container
Dec 27 16:08:52.925 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Started container
Dec 27 16:08:53.283 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:53.645 I ns=openshift-core-operators deployment=openshift-service-cert-signer-operator Scaled down replica set openshift-service-cert-signer-operator-7df87bf9b4 to 0
Dec 27 16:08:53.826 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-7df87bf9b4-6bvd4 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:08:53.986 I ns=openshift-core-operators replicaset=openshift-service-cert-signer-operator-7df87bf9b4 Deleted pod: openshift-service-cert-signer-operator-7df87bf9b4-6bvd4
Dec 27 16:08:53.987 W ns=openshift-dns pod=dns-default-g27zs node=ip-10-0-33-43.ec2.internal container=dns container stopped being ready
Dec 27 16:08:53.987 W ns=openshift-dns pod=dns-default-g27zs node=ip-10-0-33-43.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:08:54.347 I ns=openshift-dns pod=dns-default-g27zs Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:54.348 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-7df87bf9b4-6bvd4 Killing container with id cri-o://operator:Need to kill Pod
Dec 27 16:08:54.348 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:08:55.403 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-7df87bf9b4-6bvd4 node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:08:55.582 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-7df87bf9b4-6bvd4 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:08:55.702 I ns=openshift-dns pod=dns-default-g27zs Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:08:55.711 W ns=openshift-console pod=console-operator-5d979b9854-fct6l The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:55.820 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Scaled up replica set openshift-cluster-kube-apiserver-operator-588dc8c5dd to 1
Dec 27 16:08:55.942 I ns=openshift-cluster-kube-apiserver-operator replicaset=openshift-cluster-kube-apiserver-operator-588dc8c5dd Created pod: openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn
Dec 27 16:08:55.944 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn Successfully assigned openshift-cluster-kube-apiserver-operator/openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn to ip-10-0-33-43.ec2.internal
Dec 27 16:08:56.087 W ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:56.155 I ns=openshift-dns pod=dns-default-g27zs Created container
Dec 27 16:08:56.190 I ns=openshift-dns pod=dns-default-g27zs Started container
Dec 27 16:08:56.197 I ns=openshift-dns pod=dns-default-g27zs Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9c116e0a8c006f3c67f074a0372c6b4f977161386a95d41aa0faabbe94a558f0" already present on machine
Dec 27 16:08:56.444 W ns=openshift-dns pod=dns-default-xtj2b The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:08:56.523 W ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm node=ip-10-0-161-51.ec2.internal container=dns-operator container stopped being ready
Dec 27 16:08:57.430 I ns=openshift-controller-manager pod=controller-manager-2qgd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e" already present on machine
Dec 27 16:08:57.511 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal container=console-operator container stopped being ready
Dec 27 16:08:57.515 I ns=openshift-console pod=console-operator-5d979b9854-fct6l Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:57.839 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Pod sandbox changed, it will be killed and re-created.
Dec 27 16:08:57.856 I ns=openshift-dns pod=dns-default-g27zs Created container
Dec 27 16:08:57.883 I ns=openshift-controller-manager pod=controller-manager-2qgd4 Created container
Dec 27 16:08:57.913 I ns=openshift-dns pod=dns-default-g27zs Started container
Dec 27 16:08:57.936 I ns=openshift-controller-manager pod=controller-manager-2qgd4 Started container
Dec 27 16:08:58.398 I ns=openshift-sdn pod=ovs-l9kq7 Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:08:58.403 W ns=openshift-dns pod=dns-default-xtj2b Liveness probe failed: Get http://10.129.2.2:8080/health: dial tcp 10.129.2.2:8080: connect: no route to host
Dec 27 16:08:58.404 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:58.500 I ns=openshift-console pod=console-operator-5d979b9854-fct6l pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:44c0f063219f9c6c5190c8c0f9ab9e997c6e168a9c91a7edd88f0099736c7798"
Dec 27 16:08:58.500 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:08:58.560 W ns=openshift-sdn pod=ovs-l9kq7 node=ip-10-0-12-206.ec2.internal container=openvswitch container stopped being ready
Dec 27 16:08:58.669 I ns=openshift-console pod=console-operator-5d979b9854-fct6l Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:44c0f063219f9c6c5190c8c0f9ab9e997c6e168a9c91a7edd88f0099736c7798"
Dec 27 16:08:59.170 W ns=openshift-controller-manager pod=controller-manager-2qgd4 node=ip-10-0-33-43.ec2.internal container=controller-manager container restarted
Dec 27 16:08:59.185 I ns=openshift-console pod=console-operator-5d979b9854-fct6l Created container
Dec 27 16:08:59.219 W ns=openshift-dns pod=dns-default-g27zs node=ip-10-0-33-43.ec2.internal container=dns container restarted
Dec 27 16:08:59.219 W ns=openshift-dns pod=dns-default-g27zs node=ip-10-0-33-43.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:08:59.234 I ns=openshift-console pod=console-operator-5d979b9854-fct6l Started container
Dec 27 16:08:59.580 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal container=console-operator container restarted
Dec 27 16:08:59.857 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:07eab41e0452f0478cc614b32da4fbc1676c22c638431f3168d13c73c648f0f3" already present on machine
Dec 27 16:08:59.881 W ns=openshift-sdn pod=ovs-l9kq7 node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:08:59.961 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:066f93a61a131fcddf7ee3333be0eb4faadb61f946c06f3a2ddee8deacba033e"
Dec 27 16:09:00.137 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Created container
Dec 27 16:09:00.151 I ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm Started container
Dec 27 16:09:00.260 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91" already present on machine
Dec 27 16:09:00.470 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:00.554 W ns=openshift-dns-operator pod=dns-operator-6fbc89d8dc-blcxm node=ip-10-0-161-51.ec2.internal container=dns-operator container restarted
Dec 27 16:09:00.722 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Created container
Dec 27 16:09:00.765 I ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c Started container
Dec 27 16:09:00.908 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:066f93a61a131fcddf7ee3333be0eb4faadb61f946c06f3a2ddee8deacba033e"
Dec 27 16:09:01.005 W ns=openshift-core-operators pod=origin-cluster-osin-operator-86df5958b-qfk4c node=ip-10-0-33-43.ec2.internal container=operator container restarted
Dec 27 16:09:01.033 I ns=openshift-core-operators configmap=cluster-osin-operator2-lock b7b6b10e-09f1-11e9-a7fd-0a580a80001c became leader
Dec 27 16:09:01.217 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn Created container
Dec 27 16:09:01.260 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-588dc8c5dd-sxqzn Started container
Dec 27 16:09:01.730 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:01.919 I openshift-apiserver OpenShift API started failing: Get https://ci-ln-y7gvwf2-703b0-api.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=3s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:02.447 I ns=openshift-apiserver pod=apiserver-wsjnb pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:02.499 I ns=openshift-apiserver pod=apiserver-wsjnb Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:02.856 I ns=openshift-apiserver pod=apiserver-wsjnb Created container
Dec 27 16:09:02.897 I ns=openshift-apiserver pod=apiserver-wsjnb Started container
Dec 27 16:09:03.471 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:03.978 W ns=openshift-apiserver pod=apiserver-7h7td Liveness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:03.993 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:04.597 I ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:04.678 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Liveness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:04.751 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Liveness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:04.780 I ns=openshift-sdn pod=sdn-controller-p5xc2 Killing container with id cri-o://sdn-controller:Need to kill Pod
Dec 27 16:09:05.333 W ns=openshift-apiserver pod=apiserver-wsjnb Readiness probe failed: Get https://10.129.0.20:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:05.341 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:05.725 W ns=openshift-dns pod=dns-default-xtj2b Liveness probe failed: Get http://10.129.2.2:8080/health: dial tcp 10.129.2.2:8080: connect: no route to host
Dec 27 16:09:05.725 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:05.725 W ns=openshift-sdn pod=sdn-controller-p5xc2 node=ip-10-0-30-101.ec2.internal container=sdn-controller container stopped being ready
Dec 27 16:09:05.740 W ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:05.880 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:09:05.886 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Scaled down replica set openshift-cluster-kube-apiserver-operator-5c44566d56 to 0
Dec 27 16:09:05.886 I ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:06.057 W ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:09:06.188 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n node=ip-10-0-33-43.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:09:06.251 I ns=openshift-cluster-kube-apiserver-operator replicaset=openshift-cluster-kube-apiserver-operator-5c44566d56 Deleted pod: openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7
Dec 27 16:09:06.407 W ns=openshift-sdn pod=sdn-controller-p5xc2 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:09:06.457 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 node=ip-10-0-33-43.ec2.internal container=machine-config-controller container stopped being ready
Dec 27 16:09:06.504 I ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7 Killing container with id cri-o://operator:Need to kill Pod
Dec 27 16:09:06.508 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:06.929 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:07.018 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:07.228 I ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:07.230 W ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc node=ip-10-0-33-43.ec2.internal container=configmap-registry-server container stopped being ready
Dec 27 16:09:07.248 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:09:07.337 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:09:07.345 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:07.503 I ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea" already present on machine
Dec 27 16:09:07.580 W ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7 node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:09:07.723 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:09:07.737 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Killing container with id cri-o://router:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:07.810 I ns=openshift-dns pod=dns-default-xtj2b Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:07.860 W ns=openshift-dns pod=dns-default-xtj2b node=ip-10-0-161-51.ec2.internal container=dns container stopped being ready
Dec 27 16:09:07.860 W ns=openshift-dns pod=dns-default-xtj2b node=ip-10-0-161-51.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:09:08.029 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:7a32d6d2d8477afab8cea56ee629b777a6c6bf54bd861b326b2e1861a616bd8c" already present on machine
Dec 27 16:09:08.029 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:08.066 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Created container
Dec 27 16:09:08.316 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:08.316 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx node=ip-10-0-33-43.ec2.internal container=apiservice-cabundle-injector-controller container stopped being ready
Dec 27 16:09:08.405 I ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:08.593 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Started container
Dec 27 16:09:08.704 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:09:09.150 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:09.152 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:09:09.331 I ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n Created container
Dec 27 16:09:09.512 I ns=openshift-sdn pod=sdn-z6fp6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:09:09.673 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z node=ip-10-0-33-43.ec2.internal container=configmap-cabundle-injector-controller container stopped being ready
Dec 27 16:09:09.853 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:09.853 I ns=openshift-dns pod=dns-default-xtj2b Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:09:09.881 I ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n Started container
Dec 27 16:09:10.033 I ns=openshift-sdn pod=sdn-z6fp6 Created container
Dec 27 16:09:10.034 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:09:10.231 I ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc pulling image "quay.io/operatorframework/configmap-operator-registry:latest"
Dec 27 16:09:10.231 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n node=ip-10-0-33-43.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:09:10.395 I node=ip-10-0-33-43.ec2.internal Starting openshift-sdn.
Dec 27 16:09:10.395 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:09:10.395 I ns=openshift-sdn pod=sdn-z6fp6 Started container
# This appears to be the last of the SDN containers, so after 3 minutes we've restarted everything on the cluster as a result of the update
# The SDN controller is also restarted, which might cause a set of subsequent errors.
# BUG: It looks like the SDN daemonset is updated multiple times (or multiple pods are created) which completely disrupts everything else
Dec 27 16:09:10.395 I ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:10.575 I ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750" already present on machine
Dec 27 16:09:10.576 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# BUG: Needs to be turned into a image reference
Dec 27 16:09:10.577 I ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc Successfully pulled image "quay.io/operatorframework/configmap-operator-registry:latest"
Dec 27 16:09:10.577 I ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:11.085 I ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:36f76a2b29b7329dd882f1111680ef6cd20faab0288cd8dbb1476211bf82aec5" already present on machine
Dec 27 16:09:11.085 I ns=openshift-dns pod=dns-default-xtj2b Created container
Dec 27 16:09:11.477 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 node=ip-10-0-33-43.ec2.internal container=service-serving-cert-signer-controller container stopped being ready
Dec 27 16:09:11.488 I ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750" already present on machine
Dec 27 16:09:11.488 W ns=openshift-dns pod=dns-default-xtj2b node=ip-10-0-161-51.ec2.internal container=dns container restarted
Dec 27 16:09:11.488 W ns=openshift-dns pod=dns-default-xtj2b node=ip-10-0-161-51.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:09:11.531 I ns=openshift-dns pod=dns-default-xtj2b Started container
Dec 27 16:09:11.867 I ns=openshift-dns pod=dns-default-xtj2b Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9c116e0a8c006f3c67f074a0372c6b4f977161386a95d41aa0faabbe94a558f0" already present on machine
Dec 27 16:09:11.868 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:09:12.061 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:12.079 I ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc Created container
Dec 27 16:09:12.107 I ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc Started container
Dec 27 16:09:12.149 I ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx Created container
Dec 27 16:09:12.270 I ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx Started container
Dec 27 16:09:12.277 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:09:13.034 I ns=openshift-dns pod=dns-default-xtj2b Created container
Dec 27 16:09:13.077 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx node=ip-10-0-33-43.ec2.internal container=apiservice-cabundle-injector-controller container restarted
# BUG: This event is not providing a lot of info, it should either tell me something I need to know or be quiet
Dec 27 16:09:13.168 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:09:13.182 I ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z Created container
Dec 27 16:09:13.200 I ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z Started container
Dec 27 16:09:13.229 I ns=openshift-dns pod=dns-default-xtj2b Started container
Dec 27 16:09:13.241 W ns=openshift-operator-lifecycle-manager pod=olm-operators-k9jdc node=ip-10-0-33-43.ec2.internal container=configmap-registry-server container restarted
Dec 27 16:09:13.337 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Created container
Dec 27 16:09:13.431 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Started container
Dec 27 16:09:13.432 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:13.448 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z node=ip-10-0-33-43.ec2.internal container=configmap-cabundle-injector-controller container restarted
Dec 27 16:09:13.577 I ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750" already present on machine
# Almost certainly related to SDN restart, but now it causes API failures (pod is not taken out of rotation first)
Dec 27 16:09:13.922 W ns=openshift-apiserver pod=apiserver-7h7td Liveness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:14.002 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:14.022 I ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 Created container
Dec 27 16:09:14.139 I ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 Started container
Dec 27 16:09:14.632 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Liveness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:14.723 I ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 Created container
Dec 27 16:09:14.741 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Liveness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:14.778 I ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 Started container
Dec 27 16:09:15.225 W ns=openshift-apiserver pod=apiserver-wsjnb Readiness probe failed: Get https://10.129.0.20:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:15.362 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:09:16.708 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:16.712 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 node=ip-10-0-33-43.ec2.internal container=service-serving-cert-signer-controller container restarted
Dec 27 16:09:17.194 I ns=openshift-console configmap=console-operator-lock c0b65c42-09f1-11e9-a572-0a580a81020c became leader
Dec 27 16:09:17.603 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 node=ip-10-0-161-51.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:09:17.749 W ns=openshift-monitoring pod=alertmanager-main-2 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:17.915 I ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea" already present on machine
Dec 27 16:09:18.093 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:09:18.351 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:18.530 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:09:18.655 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy-self container stopped being ready
Dec 27 16:09:18.655 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-state-metrics container stopped being ready
Dec 27 16:09:18.662 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:18.948 W ns=openshift-cluster-kube-apiserver-operator pod=openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:09:19.069 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager container stopped being ready
Dec 27 16:09:19.069 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:09:19.069 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=config-reloader container stopped being ready
Dec 27 16:09:19.446 W ns=openshift-monitoring pod=prometheus-k8s-0 The pod's network interface has been lost and the pod will be stopped.
# BUG: more images that need to be properly referenced
Dec 27 16:09:19.446 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
Dec 27 16:09:19.510 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 node=ip-10-0-33-43.ec2.internal container=machine-config-controller container restarted
Dec 27 16:09:19.527 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:09:19.570 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Container image "quay.io/coreos/kube-state-metrics:v1.4.0" already present on machine
Dec 27 16:09:19.658 I ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:19.709 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:09:19.733 I ns=openshift-monitoring pod=alertmanager-main-2 Pod sandbox changed, it will be killed and re-created.
# BUG: more images that need to be properly referenced
Dec 27 16:09:20.108 I ns=openshift-monitoring pod=prometheus-k8s-0 Container image "openshift/prometheus:v2.5.0" already present on machine
Dec 27 16:09:20.189 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Pod sandbox changed, it will be killed and re-created.
Dec 27 16:09:20.316 W ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:09:20.458 W ns=openshift-monitoring pod=prometheus-k8s-0 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/7660/ns/ipc\" caused \"lstat /proc/7660/ns/ipc: no such file or directory\""
Dec 27 16:09:20.472 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# BUG: more images that need to be properly referenced
Dec 27 16:09:20.478 I ns=openshift-monitoring pod=prometheus-k8s-0 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:09:20.484 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container stopped being ready
Dec 27 16:09:20.647 W ns=openshift-monitoring pod=prometheus-k8s-0 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:09:20.667 I ns=openshift-monitoring pod=prometheus-k8s-0 Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
Dec 27 16:09:20.707 I node=ip-10-0-161-51.ec2.internal Starting openshift-sdn.
Dec 27 16:09:20.888 W ns=openshift-monitoring pod=prometheus-k8s-0 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
# BUG: more images that need to be properly referenced
Dec 27 16:09:20.906 I ns=openshift-monitoring pod=prometheus-k8s-0 Container image "quay.io/coreos/prom-label-proxy:v0.1.0" already present on machine
Dec 27 16:09:21.138 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Back-off restarting failed container
Dec 27 16:09:21.214 I ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 Pod sandbox changed, it will be killed and re-created.
# BUG: more images that need to be properly referenced
Dec 27 16:09:21.529 I ns=openshift-monitoring pod=alertmanager-main-2 Container image "openshift/prometheus-alertmanager:v0.15.2" already present on machine
Dec 27 16:09:21.701 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy container stopped being ready
Dec 27 16:09:21.701 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prom-label-proxy container stopped being ready
Dec 27 16:09:21.701 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus container stopped being ready
Dec 27 16:09:21.701 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:09:21.701 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=rules-configmap-reloader container stopped being ready
Dec 27 16:09:21.735 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container stopped being ready
Dec 27 16:09:21.984 I ns=openshift-monitoring pod=alertmanager-main-2 Created container
Dec 27 16:09:21.987 I ns=openshift-monitoring pod=alertmanager-main-2 Started container
# BUG: more images that need to be properly referenced
Dec 27 16:09:21.991 I ns=openshift-monitoring pod=alertmanager-main-2 Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine
Dec 27 16:09:22.088 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:22.092 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:09:22.144 I ns=openshift-monitoring pod=alertmanager-main-2 Created container
Dec 27 16:09:22.168 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
Dec 27 16:09:22.186 I ns=openshift-monitoring pod=alertmanager-main-2 Started container
Dec 27 16:09:22.244 I ns=openshift-monitoring pod=alertmanager-main-2 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:09:23.439 I ns=openshift-cluster-kube-apiserver-operator configmap=openshift-cluster-kube-apiserver-operator-lock c4fab100-09f1-11e9-a267-0a580a810019 became leader
Dec 27 16:09:23.439 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Back-off restarting failed container
Dec 27 16:09:23.440 I ns=openshift-monitoring pod=alertmanager-main-2 Created container
Dec 27 16:09:23.440 I ns=openshift-monitoring pod=alertmanager-main-2 Started container
Dec 27 16:09:23.441 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container stopped being ready
Dec 27 16:09:23.452 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Created container
Dec 27 16:09:23.453 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Started container
Dec 27 16:09:23.453 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:23.453 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Status for operator openshift-cluster-kube-apiserver-operator changed
Dec 27 16:09:23.453 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:23.517 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:23.715 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Liveness probe failed: Get http://10.130.0.7:8080/healthz: dial tcp 10.130.0.7:8080: connect: no route to host
# BUG: more images that need to be properly referenced
Dec 27 16:09:23.715 I ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 Container image "quay.io/coreos/prometheus-operator:v0.26.0" already present on machine
Dec 27 16:09:24.008 I ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 Created container
Dec 27 16:09:24.092 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:24.093 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy-main container stopped being ready
Dec 27 16:09:24.113 W ns=openshift-apiserver pod=apiserver-7h7td Liveness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:24.370 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:24.370 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:24.463 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: dial tcp 10.130.0.16:8443: connect: no route to host
Dec 27 16:09:24.463 I ns=openshift-apiserver pod=apiserver-7h7td Killing container with id cri-o://openshift-apiserver:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:24.463 I ns=openshift-apiserver pod=apiserver-7h7td pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:24.828 I ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 Started container
Dec 27 16:09:24.828 I ns=openshift-apiserver pod=apiserver-7h7td Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:25.009 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Liveness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:25.188 I ns=openshift-apiserver pod=apiserver-7h7td Created container
Dec 27 16:09:25.188 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Killing container with id cri-o://console:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:25.189 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:09:25.189 I ns=openshift-apiserver pod=apiserver-7h7td Started container
Dec 27 16:09:25.710 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:25.710 W ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 node=ip-10-0-161-51.ec2.internal container=prometheus-operator container stopped being ready
Dec 27 16:09:25.892 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:09:25.892 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container stopped being ready
Dec 27 16:09:26.798 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Back-off restarting failed container
Dec 27 16:09:26.800 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-config-reloader container stopped being ready
Dec 27 16:09:27.264 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager container restarted
Dec 27 16:09:27.264 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:09:27.264 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=config-reloader container restarted
Dec 27 16:09:27.740 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:09:27.740 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:09:27.827 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:27.975 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy-main container restarted
Dec 27 16:09:28.041 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Created container
Dec 27 16:09:28.092 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Started container
Dec 27 16:09:28.250 W ns=openshift-monitoring pod=prometheus-operator-7b5b4b567b-mnbb2 node=ip-10-0-161-51.ec2.internal container=prometheus-operator container restarted
Dec 27 16:09:28.680 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-config-reloader container restarted
Dec 27 16:09:28.694 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:09:28.741 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:29.537 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Updated ConfigMap/kube-apiserver-pod -n openshift-kube-apiserver because it changed
Dec 27 16:09:29.587 I ns=openshift-service-cert-signer configmap=openshift-service-serving-cert-signer-cabundle-injector-lock c9506e45-09f1-11e9-86d8-0a580a81001e became leader
# DEBUG: What?
Dec 27 16:09:30.298 W ns=openshift-sdn daemonset=ovs Error creating: Timeout: request did not complete within allowed duration
Dec 27 16:09:30.324 I ns=openshift-service-cert-signer configmap=openshift-service-serving-cert-signer-apiservice-injector-lock c8edb9ab-09f1-11e9-97ad-0a580a81001c became leader
Dec 27 16:09:30.482 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# DEBUG: This could still be the SDN restart, or the apiserver could have crashed a second time (possibly due to being unable to connect to etcd)
Dec 27 16:09:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:09:30.914 I ns=openshift-core-operators configmap=cluster-osin-operator-lock c900069f-09f1-11e9-844e-0a580a810017 became leader
Dec 27 16:09:31.053 I ns=openshift-service-cert-signer configmap=openshift-service-serving-cert-signer-serving-ca-lock c9e2d3d4-09f1-11e9-8cd7-0a580a81001f became leader
Dec 27 16:09:31.922 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:33.474 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:33.956 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:34.498 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:09:35.133 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/kube-apiserver-pod-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:35.349 I ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea" already present on machine
Dec 27 16:09:35.545 I ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Created container
Dec 27 16:09:35.567 I ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 Started container
Dec 27 16:09:36.078 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 node=ip-10-0-161-51.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:09:36.547 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/config-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:36.547 W ns=openshift-sdn daemonset=sdn-controller Error creating: Timeout: request did not complete within allowed duration
Dec 27 16:09:37.671 I ns=openshift-apiserver pod=apiserver-wsjnb pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:37.816 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:37.816 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/aggregator-client-ca-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:37.816 I ns=openshift-apiserver pod=apiserver-wsjnb Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:37.979 I ns=openshift-apiserver pod=apiserver-wsjnb Created container
Dec 27 16:09:37.996 I ns=openshift-apiserver pod=apiserver-wsjnb Started container
Dec 27 16:09:38.697 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:39.152 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:09:39.156 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Liveness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:39.600 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
# DEBUG: this operator uses host network, so the fact that it is timing out trying to hit the apiserver is worrying (could be the hostnetwork kube-proxy issue)
Dec 27 16:09:39.600 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:09:39.775 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/client-ca-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:40.049 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/etcd-serving-ca-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:40.239 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Created container
Dec 27 16:09:40.244 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Started container
Dec 27 16:09:40.261 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Container image "quay.io/coreos/kube-state-metrics:v1.4.0" already present on machine
Dec 27 16:09:40.604 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:40.604 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Created container
Dec 27 16:09:40.604 I ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb Started container
Dec 27 16:09:41.304 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/kubelet-serving-ca-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:41.305 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy-self container restarted
Dec 27 16:09:41.305 W ns=openshift-monitoring pod=kube-state-metrics-dcf7dc56d-hgwrb node=ip-10-0-161-51.ec2.internal container=kube-state-metrics container restarted
Dec 27 16:09:41.666 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:42.184 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:09:42.545 I ns=openshift-dns pod=dns-default-zv48s Killing container with id cri-o://dns:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:42.545 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created ConfigMap/sa-token-signing-certs-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:42.545 I ns=openshift-dns pod=dns-default-zv48s Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:09:42.938 I ns=openshift-dns pod=dns-default-zv48s Created container
Dec 27 16:09:43.264 I ns=openshift-dns pod=dns-default-zv48s Started container
Dec 27 16:09:43.334 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=kube-rbac-proxy container restarted
Dec 27 16:09:43.334 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prom-label-proxy container restarted
Dec 27 16:09:43.334 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus container restarted
Dec 27 16:09:43.334 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:09:43.334 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=rules-configmap-reloader container restarted
Dec 27 16:09:43.422 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:43.492 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created Secret/aggregator-client-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:43.919 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:09:44.030 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:44.916 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created Secret/etcd-client-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:45.184 W ns=openshift-apiserver pod=apiserver-wsjnb Readiness probe failed: Get https://10.129.0.20:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:45.719 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created Secret/kubelet-client-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:45.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:09:46.922 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created Secret/serving-cert-5 -n openshift-kube-apiserver because it was missing
Dec 27 16:09:46.922 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Revision 4 created because configmap/kube-apiserver-pod has changed
Dec 27 16:09:46.951 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Status for operator openshift-cluster-kube-apiserver-operator changed
Dec 27 16:09:47.071 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: dial tcp 10.128.2.7:1936: connect: no route to host
Dec 27 16:09:48.323 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: dial tcp 10.130.0.18:8443: connect: no route to host
Dec 27 16:09:48.733 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:09:50.474 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:50.701 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Updating node "ip-10-0-12-206.ec2.internal" from revision 4 to 5
Dec 27 16:09:50.736 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Status for operator openshift-cluster-kube-apiserver-operator changed
Dec 27 16:09:50.856 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:09:52.170 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:09:52.230 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Created Pod/installer-5-ip-10-0-12-206.ec2.internal -n openshift-kube-apiserver because it was missing
Dec 27 16:09:53.443 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:09:53.527 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# DEBUG: Why do the installer pods require pod networking?
Dec 27 16:09:53.577 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(5f427f4897740e73496ef1dc2de33a42ec2f70bccd05a302c8b3639d36a2e67d): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:09:53.641 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:09:54.088 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:09:54.107 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Killing container with id cri-o://olm-operator:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:54.216 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:09:54.774 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Created container
Dec 27 16:09:54.888 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Started container
Dec 27 16:09:54.962 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:09:55.357 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Killing container with id cri-o://catalog-operator:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:09:55.552 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:09:55.642 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Created container
Dec 27 16:09:55.661 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Started container
Dec 27 16:09:56.021 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:09:58.539 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:09:58.610 I ns=openshift-apiserver pod=apiserver-7h7td pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:09:58.711 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Created container
Dec 27 16:09:58.790 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Started container
Dec 27 16:09:59.032 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:10:00.114 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:10:00.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:10:01.097 I ns=openshift-apiserver pod=apiserver-7h7td Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:10:01.533 W ns=openshift-apiserver pod=apiserver-7h7td Readiness probe failed: Get https://10.130.0.18:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:01.533 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:10:01.557 I ns=openshift-apiserver pod=apiserver-7h7td Created container
Dec 27 16:10:01.609 I ns=openshift-apiserver pod=apiserver-7h7td Started container
Dec 27 16:10:01.652 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:10:01.682 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:01.699 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Created container
Dec 27 16:10:01.719 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:10:01.747 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Started container
Dec 27 16:10:01.767 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Readiness probe failed: Get http://10.130.0.7:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:01.907 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:10:03.683 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Readiness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:04.016 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:10:04.017 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:04.227 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:10:04.912 I ns=openshift-monitoring pod=alertmanager-main-2 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:10:05.078 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Liveness probe failed: Get http://10.130.0.6:8080/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:05.084 I ns=openshift-monitoring pod=alertmanager-main-2 Created container
Dec 27 16:10:05.152 I ns=openshift-monitoring pod=alertmanager-main-2 Started container
Dec 27 16:10:05.258 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:10:05.360 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:05.419 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:10:05.807 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(8538cc449eb504745531fb63238a1f4105720b03a60f7ac504a41eede4dfb8d1): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:10:10.339 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:10:10.454 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:10:10.469 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:10.952 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:10:11.528 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:10:13.312 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
# DEBUG: SDN pod failed, no clear reason why
Dec 27 16:10:13.492 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:10:13.831 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:10:14.007 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:14.188 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:10:15.561 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:15.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:10:16.033 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:10:16.791 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Created container
Dec 27 16:10:16.872 I ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Started container
Dec 27 16:10:17.074 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:10:20.784 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:10:20.840 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:20.861 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(9d17ad68000817eab9f334b2782a9e946a6dfde7e050cc51b624165182d90ddd): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:10:22.338 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:10:23.186 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:10:23.391 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Created container
Dec 27 16:10:23.414 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Started container
Dec 27 16:10:23.557 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:10:24.044 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:24.044 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:10:25.494 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Liveness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:25.922 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Killing container with id cri-o://console:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:10:25.938 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:10:26.286 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Created container
Dec 27 16:10:26.295 I ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Started container
Dec 27 16:10:27.392 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:10:28.473 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc" already present on machine
Dec 27 16:10:28.638 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Created container
Dec 27 16:10:28.817 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:10:28.998 I ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Started container
Dec 27 16:10:29.190 I ns=openshift-apiserver pod=apiserver-7h7td pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:10:29.518 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:10:29.527 I ns=openshift-sdn pod=sdn-z6fp6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:10:29.693 I ns=openshift-sdn pod=sdn-z6fp6 Created container
Dec 27 16:10:29.874 I ns=openshift-sdn pod=sdn-z6fp6 Started container
Dec 27 16:10:30.173 I ns=openshift-apiserver pod=apiserver-7h7td Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:10:30.718 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:10:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:10:31.435 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:10:31.844 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:10:31.844 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:10:32.118 I ns=openshift-apiserver pod=apiserver-7h7td Created container
Dec 27 16:10:32.131 I ns=openshift-apiserver pod=apiserver-7h7td Started container
Dec 27 16:10:32.146 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:34.006 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:34.575 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:10:34.697 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:10:35.634 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:10:36.343 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:10:37.017 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(aa679299264fb0f301acd9e13dd4e201d7d24b10bc18bbd034bc42fabcfb6c83): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:10:38.993 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:10:40.038 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:10:40.548 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:10:40.548 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:10:43.726 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:10:44.049 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:10:47.432 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:10:49.661 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(050fbdb1c59133cd95deafee86ccd46ab8759d0ad4e62b4dac9df11e0be1b181): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:10:50.322 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:10:52.317 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:10:54.090 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:10:54.922 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:11:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:11:01.635 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:11:01.920 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:11:02.462 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(e36dcef5ef4e9319a73bd696261d33fc49ad8900008b83ffe5e7a1be1dbe9ea9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:11:02.462 I ns=openshift-monitoring pod=alertmanager-main-2 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:11:02.507 I ns=openshift-monitoring pod=alertmanager-main-2 Created container
Dec 27 16:11:02.539 I ns=openshift-monitoring pod=alertmanager-main-2 Started container
Dec 27 16:11:03.118 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:11:03.793 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:11:04.205 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:07.725 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:11:10.551 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:11:10.871 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:11:13.958 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:14.512 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:11:15.912 E openshift-apiserver OpenShift API is not responding to GET requests
# DEBUG: This continues for the next twenty minutes on various nodes, it may be due to node readiness (but we get no events) or it could be because of networking?
Dec 27 16:11:15.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:11:16.823 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(ececbaa8cae210c5b58beb81e3cc37cb614d3a89adfda50a65fef428e3a7f12f): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:11:24.032 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:11:24.300 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:24.339 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:11:25.502 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:11:27.431 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:11:30.763 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(7a103d4579877583528aa43507cb7d2eec6ec04630f601544f0a6610a5eaee55): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:11:30.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:11:30.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:11:31.648 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:11:31.827 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:11:34.341 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:11:34.361 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:34.397 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:11:34.487 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:11:37.327 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:11:40.555 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:11:43.627 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(174903b298349fa018264b29167c0fe7b34be03fa59fd7deff65eee2c33d78f0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:11:43.957 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:45.248 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:11:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:11:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:11:46.181 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:11:46.445 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:11:46.625 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:11:52.512 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:11:52.786 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:11:52.834 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:11:53.102 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Created container
Dec 27 16:11:53.239 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Started container
Dec 27 16:11:53.924 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:11:53.945 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:11:55.637 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:11:57.754 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(47b579f86effdc25d3506a9fe9a390aa23e2b857c4dfdbc2d1bb3864a964da7e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:11:59.724 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:12:00.523 I ns=openshift-sdn pod=sdn-z6fp6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:12:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:12:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:12:01.058 I ns=openshift-sdn pod=sdn-z6fp6 Created container
Dec 27 16:12:01.291 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:12:01.580 I ns=openshift-sdn pod=sdn-z6fp6 Started container
Dec 27 16:12:04.321 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:12:04.976 I ns=openshift-dns pod=dns-default-zv48s Killing container with id cri-o://dns:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:12:05.015 I ns=openshift-dns pod=dns-default-zv48s Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:12:05.220 I ns=openshift-dns pod=dns-default-zv48s Created container
Dec 27 16:12:05.265 I ns=openshift-dns pod=dns-default-zv48s Started container
Dec 27 16:12:05.746 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:12:10.558 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:12:14.068 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:12:14.400 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(5cd83b69ee3fbd6891e0844c5510a248236aa607ded7c697fdc63214ab969d32): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:12:15.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:12:15.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:12:16.461 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:12:16.805 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator "aggregator-proxy-client-cert-key" in "openshift-cluster-kube-apiserver-operator" requires a new target cert/key pair
Dec 27 16:12:23.039 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:12:27.394 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:12:27.409 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:12:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:12:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:12:31.160 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(9e06b4afc3345cbe4bbbdccf92537b786a0264cc746b2bec45437736c7179b49): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:12:39.232 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:12:40.603 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:12:42.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(57806d298ef9d7625eb7389bd0e1d85e86620cface91828e3cb0e901bcb47931): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:12:45.188 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:12:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:12:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:12:53.568 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:12:53.568 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:12:53.577 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:12:55.820 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(f0597c0e564089ce35fbd328297b8c81f71bc51f2a8f31af5839c7106e1879f9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:13:00.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:13:00.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:13:01.319 W ns=openshift-monitoring pod=prometheus-k8s-0 Back-off restarting failed container
Dec 27 16:13:01.991 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:13:02.383 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:13:04.832 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:13:10.618 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(3d744d6df63151632ab621d4406001fd1616d8582134bee91a6ae83a99e6c77a): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:13:10.638 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:13:13.435 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:13:13.741 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:13:14.408 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:13:15.359 I ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:7a32d6d2d8477afab8cea56ee629b777a6c6bf54bd861b326b2e1861a616bd8c" already present on machine
Dec 27 16:13:15.451 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:13:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:13:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:13:16.515 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:13:16.874 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:13:16.874 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:13:19.681 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:13:23.673 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(8d70b0548f475306d0eb11d352f43203587221075119e8521934a637655e6e29): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:13:24.407 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:13:27.094 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:13:27.158 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:13:28.496 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:13:28.619 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:13:30.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:13:30.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:13:32.389 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:13:34.611 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:13:37.923 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(98d7df9a5a10d43fc7f9a931c4b396532dd05a96497efd480238038537569fb2): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:13:41.107 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:13:41.492 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:13:42.079 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea"
Dec 27 16:13:42.278 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Created container
Dec 27 16:13:42.330 I ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Started container
Dec 27 16:13:43.332 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:13:43.513 I ns=openshift-sdn pod=sdn-z6fp6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:13:43.695 I ns=openshift-sdn pod=sdn-z6fp6 Created container
Dec 27 16:13:43.798 I ns=openshift-sdn pod=sdn-z6fp6 Started container
Dec 27 16:13:44.110 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:13:44.462 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:13:45.608 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:13:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:13:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:13:52.922 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(25cdc8ccc543b8ed9cd7edc8952e8e74b7d1e11e6a0ce02ed57e057ce7b3311e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:13:54.394 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:14:00.512 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Readiness probe failed: Get https://10.130.0.16:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:14:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:14:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:14:02.019 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:14:04.713 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(f1e9a907b95ff73de015770c8b6778b829cd3bed51a0de9365959c985ad010b4): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:14:10.922 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:14:14.364 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:14:15.048 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:14:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:14:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:14:16.160 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:14:16.617 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:14:20.629 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(f74c0e67df7e31c127e587ed9dd0be636fb4fe869b70b102ff06f47a67a5f39c): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:14:24.931 I ns=openshift-dns pod=dns-default-zv48s Killing container with id cri-o://dns:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:14:25.921 I ns=openshift-dns pod=dns-default-zv48s Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:14:25.935 I ns=openshift-dns pod=dns-default-zv48s Created container
Dec 27 16:14:25.938 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:14:30.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:14:30.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:14:32.795 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(c76ae503ab567b8f43d6d1276a5c7611f2efbf13ee271b6bdbe27321075462a0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:14:40.923 W ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Failed to create new replica set "openshift-cluster-kube-scheduler-operator-9cbc6d5db": Timeout: request did not complete within allowed duration
Dec 27 16:14:42.524 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:14:43.650 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:14:45.392 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:14:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:14:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:14:46.005 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:14:46.728 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(3d19d464ec98795e8488f6cd8d82f09861fee36c9cff7daffa6c36ae5d4ad28e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:14:46.919 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:14:50.514 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Scaled up replica set openshift-cluster-kube-scheduler-operator-9cbc6d5db to 1
Dec 27 16:14:50.681 I ns=openshift-cluster-kube-scheduler-operator replicaset=openshift-cluster-kube-scheduler-operator-9cbc6d5db Created pod: openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25
Dec 27 16:14:50.681 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Successfully assigned openshift-cluster-kube-scheduler-operator/openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 to ip-10-0-30-101.ec2.internal
Dec 27 16:14:53.491 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:14:53.802 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:14:54.349 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:14:54.708 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:14:56.757 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:14:57.093 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Created container
Dec 27 16:14:57.119 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Started container
Dec 27 16:14:59.419 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Liveness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:14:59.829 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:15:00.221 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Scaled down replica set openshift-cluster-kube-scheduler-operator-5bfd6877d6 to 0
Dec 27 16:15:00.228 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:15:00.853 I ns=openshift-cluster-kube-scheduler-operator replicaset=openshift-cluster-kube-scheduler-operator-5bfd6877d6 Deleted pod: openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7
Dec 27 16:15:00.853 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 Killing container with id cri-o://kube-scheduler-operator-container:Need to kill Pod
Dec 27 16:15:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:15:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:15:02.290 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 node=ip-10-0-30-101.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:15:02.290 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 node=ip-10-0-30-101.ec2.internal container=kube-scheduler-operator-container container stopped being ready
Dec 27 16:15:02.960 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(bf359c64e1810f4833d32e71bb578087d8893835fc8aa67cbb02bdea050860e8): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:03.411 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Scaled up replica set openshift-cluster-kube-controller-manager-operator-579c999dff to 1
Dec 27 16:15:03.591 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:15:03.637 I ns=openshift-cluster-kube-controller-manager-operator replicaset=openshift-cluster-kube-controller-manager-operator-579c999dff Created pod: openshift-cluster-kube-controller-manager-operator-579c9999lff4
Dec 27 16:15:03.737 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-579c9999lff4 Successfully assigned openshift-cluster-kube-controller-manager-operator/openshift-cluster-kube-controller-manager-operator-579c9999lff4 to ip-10-0-33-43.ec2.internal
Dec 27 16:15:06.278 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-579c9999lff4 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:15:07.853 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-579c9999lff4 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:15:08.217 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-579c9999lff4 Created container
Dec 27 16:15:08.328 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-579c9999lff4 Started container
Dec 27 16:15:11.666 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Scaled down replica set openshift-cluster-kube-controller-manager-operator-6c496f7bb7 to 0
Dec 27 16:15:11.813 W ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:15:12.231 I ns=openshift-cluster-kube-controller-manager-operator replicaset=openshift-cluster-kube-controller-manager-operator-6c496f7bb7 Deleted pod: openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r
Dec 27 16:15:12.231 I ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r Killing container with id cri-o://operator:Need to kill Pod
Dec 27 16:15:13.462 W ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:15:13.515 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:15:14.105 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Scaled up replica set openshift-cluster-openshift-apiserver-operator-688fb8d6d9 to 1
Dec 27 16:15:14.329 I ns=openshift-cluster-openshift-apiserver-operator replicaset=openshift-cluster-openshift-apiserver-operator-688fb8d6d9 Created pod: openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt
Dec 27 16:15:14.474 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Successfully assigned openshift-cluster-openshift-apiserver-operator/openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt to ip-10-0-12-206.ec2.internal
Dec 27 16:15:15.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:15:15.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:15:15.948 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(269d42a01d196194c006246238ce76076d1a9a3b8ca94122b57d94d0a68c8da8): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:16.015 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(008fda0d2b81dbec075864d7ad7e1f38cec4ae928a13c6526457b49dd14f7686): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:18.224 I ns=openshift-cluster-kube-scheduler-operator configmap=openshift-cluster-kube-scheduler-operator-lock 98c9cef2-09f2-11e9-9cff-0a580a80001d became leader
Dec 27 16:15:18.390 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:15:18.489 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:15:18.931 W ns=openshift-cluster-kube-controller-manager-operator pod=openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:15:23.211 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Updated ConfigMap/kube-scheduler-pod -n openshift-kube-scheduler because it changed
Dec 27 16:15:23.725 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:15:23.725 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:15:25.099 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:15:26.717 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:15:27.399 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Created ConfigMap/kube-scheduler-pod-2 -n openshift-kube-scheduler because it was missing
Dec 27 16:15:27.784 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Created ConfigMap/config-2 -n openshift-kube-scheduler because it was missing
Dec 27 16:15:28.808 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Created Secret/scheduler-kubeconfig-2 -n openshift-kube-scheduler because it was missing
Dec 27 16:15:29.043 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Revision 2 created because configmap/kube-scheduler-pod has changed
Dec 27 16:15:29.075 I ns=openshift-cluster-kube-controller-manager-operator configmap=openshift-cluster-kube-controller-manager-operator-lock 9ed2eb9c-09f2-11e9-bfed-0a580a810020 became leader
Dec 27 16:15:29.195 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:15:29.399 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:15:30.549 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(c6c43ac0882ca2783b6f07b4cfde62f5304b1c96fc61a9baa999a0e2291a7e59): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:30.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:15:30.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:15:32.538 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:15:33.212 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(f86c8ff7f4f7ecb1a28fd63ba9cbecd85b0395701e9c8c9f46bc1c4712746e63): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:33.622 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updated ConfigMap/kube-controller-manager-pod -n openshift-kube-controller-manager because it changed
Dec 27 16:15:36.011 I ns=openshift-kube-scheduler pod=installer-2-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:15:37.904 I ns=openshift-kube-scheduler pod=installer-2-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:15:38.064 I ns=openshift-kube-scheduler pod=installer-2-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:15:38.245 I ns=openshift-kube-scheduler pod=installer-2-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:15:38.815 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created ConfigMap/kube-controller-manager-pod-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:39.477 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:15:39.655 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal invariant violation (bug): static pod should not transition Running->Pending with same UID
Dec 27 16:15:39.655 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=scheduler container stopped being ready
Dec 27 16:15:39.768 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal graceful deletion within 0s
Dec 27 16:15:39.868 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created ConfigMap/config-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:40.137 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:15:41.310 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created ConfigMap/client-ca-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:41.702 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:42.115 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created Secret/cluster-signing-ca-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:42.308 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=installer container stopped being ready
Dec 27 16:15:43.555 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:43.860 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created Secret/controller-manager-kubeconfig-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:43.860 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:15:43.860 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:15:43.935 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created Secret/service-account-private-key-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:44.680 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:44.761 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Created Secret/serving-cert-2 -n openshift-kube-controller-manager because it was missing
Dec 27 16:15:44.925 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:44.926 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Revision 1 created because configmap/kube-controller-manager-pod has changed
Dec 27 16:15:44.967 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:15:45.538 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:15:45.807 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(d64fddd23b1e1535d29fcd8bc386b9f85a691eaab379ee7c2a0e4031962e9e77): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:45.807 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:15:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:15:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:15:46.972 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:15:47.117 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=scheduler container restarted
Dec 27 16:15:47.650 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:15:48.572 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updating node "ip-10-0-30-101.ec2.internal" from revision 1 to 2
Dec 27 16:15:48.609 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:15:48.716 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(6d2231f0864c564d099ad31e52b5737ba0fb9196407f2ed87d5f4b354c970934): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:53.037 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-30-101.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:15:53.597 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-30-101.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:15:54.280 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-30-101.ec2.internal Created container
Dec 27 16:15:54.280 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-30-101.ec2.internal Started container
Dec 27 16:15:54.642 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:15:56.662 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal invariant violation (bug): static pod should not transition Running->Pending with same UID
Dec 27 16:15:56.662 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=kube-controller-manager container stopped being ready
Dec 27 16:15:56.662 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal graceful deletion within 0s
Dec 27 16:15:56.686 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:15:57.481 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=installer container stopped being ready
Dec 27 16:15:57.651 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:15:57.661 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:58.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(8d3068f8cf288267ac605bb6f4585aad4da1c6b301432877b69d12674d1728a4): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:15:59.028 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:15:59.545 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Created container
Dec 27 16:15:59.558 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Started container
Dec 27 16:16:00.450 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:00.451 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:00.506 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:00.861 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:00.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:16:00.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:01.679 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:16:01.726 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Created container
Dec 27 16:16:01.768 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:16:01.786 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=scheduler container restarted
Dec 27 16:16:02.035 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Started container
Dec 27 16:16:02.179 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:16:02.372 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:16:02.513 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=scheduler container stopped being ready
Dec 27 16:16:02.601 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Back-off restarting failed container
Dec 27 16:16:02.621 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:16:02.830 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:16:03.758 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Back-off restarting failed container
Dec 27 16:16:05.002 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(9a49e0822c9a0f0bf1f841cd580593f4d48ec9783a228fa0b45038aa3be2964f): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:07.826 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:16:09.778 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Killing container with id cri-o://scheduler:Need to kill Pod
Dec 27 16:16:11.817 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(4ad7ac4f2edbb1cc8ed178179806100039639074f8cd3de42112811b9eca401e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:12.741 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:16:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:16:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:16.267 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:16:17.123 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:16:17.155 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:17.551 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:18.469 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:16:18.781 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:16:19.112 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Created container
Dec 27 16:16:19.115 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Started container
Dec 27 16:16:19.118 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Back-off restarting failed container
Dec 27 16:16:20.759 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(edbe10086d6b77eb22832505efea399e4e96dd59c4b53649feae16d44073ad15): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:25.818 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(3ff03caadf726c14e83809943f2cbadd07b4a42dbebe8999c81abecf82e4b99c): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:26.148 I ns=kube-system endpoints=kube-scheduler ip-10-0-30-101_3bb721b1-09ec-11e9-bda4-0a7ded5c62ec became leader
Dec 27 16:16:26.905 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Killing container with id cri-o://kube-controller-manager:Need to kill Pod
Dec 27 16:16:27.486 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:27.561 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:27.894 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:16:27.894 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:16:29.101 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=scheduler container restarted
Dec 27 16:16:30.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:16:30.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:30.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:33.085 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Back-off restarting failed container
Dec 27 16:16:36.899 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(1e0619f30d81a15514ba079220cb864be1438da60412675ad68f9f22ab468b8c): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:38.124 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:16:40.973 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(57bd8d7608796ef9b09b00aa54e2003e301cf3ca46b0014e4303e5b517fbbd4d): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:45.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:16:45.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:45.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:16:46.920 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:16:47.773 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:48.209 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:48.209 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Created container
Dec 27 16:16:48.214 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal Started container
Dec 27 16:16:48.308 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:16:48.420 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updated node "ip-10-0-30-101.ec2.internal" from revision 1 to 2
Dec 27 16:16:48.429 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:16:50.423 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updating node "ip-10-0-33-43.ec2.internal" from revision 1 to 2
Dec 27 16:16:50.423 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:16:51.856 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(f4f63c8946a798d8c3e2a1c517465888cb090c4255c1e3f0d58bc6e5372156fb): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:53.509 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:16:53.615 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:16:53.670 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(25cd68b24e3ebb0aef74655d4a0e93953e81c4c18f3552bdd2f6c71f393df2af): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:16:53.860 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:34401816a3868f14d768ea6d1cf35bb620820ed5e6e7829cd334e6296ea967eb"
Dec 27 16:16:54.416 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:16:54.426 I ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:16:56.586 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal graceful deletion within 0s
Dec 27 16:16:56.767 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:16:56.947 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=installer container stopped being ready
Dec 27 16:16:57.640 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:57.670 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:58.000 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:16:58.180 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:16:59.118 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:59.740 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:16:59.814 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:17:00.097 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:17:00.309 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:17:00.456 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updated node "ip-10-0-33-43.ec2.internal" from revision 1 to 2
Dec 27 16:17:00.475 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:17:00.572 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=kube-controller-manager container stopped being ready
Dec 27 16:17:00.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:17:00.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:00.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:00.918 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:17:01.889 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:17:03.245 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Updating node "ip-10-0-12-206.ec2.internal" from revision 1 to 2
Dec 27 16:17:03.529 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:17:03.757 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(0f8280136b707072ed20c22c772d18f4f636fe6e1c4eda94267bfb81eb3dd533): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:03.958 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:17:05.094 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:17:05.680 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(531388f6201357fdc402afd0a53fbbc9334f2f34b9a0e3ccaf33654fcfef2e5e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:08.930 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(d6d9375928b7b51990459cc10f5142f11817a6ddd87fccc61fb92db0f9d145d2): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:10.212 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:17:13.617 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:17:14.056 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:17:14.056 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:17:14.074 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:17:14.899 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:17:14.978 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:17:15.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:17:15.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:15.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:17.395 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:17:17.919 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(f168071f9daebc83f574965e0ddf84e3cfb060eb5a61d9ab80c63ee1e48b75d9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:19.961 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(aef0544d6a7cd4f5a8ed410962defa2ab7aaf582e6ecafe7f483fb5bd88527d5): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:20.631 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(e4d79a2e658e6616b15e8ee653fbe3d5e755cebe026afa373e1fceb0e6ae16f1): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:26.499 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Killing container with id cri-o://kube-controller-manager:Need to kill Pod
Dec 27 16:17:30.259 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Back-off restarting failed container
Dec 27 16:17:30.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:17:30.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:30.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:30.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:32.407 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(9cd8d73595a2805259c0a70491db1fcfdf8603fa498a0b75eb5607406b0032e0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:32.816 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(ea4455ead8067055a538c699f6292ebd9b225f7dc4fc4d3f99096fce3a98e45d): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:33.096 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(ae3c7660f26c3c5ac3b9185b19dda55e31d3667942c34993cd6a6df9a860a2e0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:35.372 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:17:35.997 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:17:45.765 I ns=kube-system configmap=kube-controller-manager ip-10-0-30-101_d85e8944-09f2-11e9-8ad2-0a7ded5c62ec became leader
Dec 27 16:17:45.894 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(fdfdd67f16093481cb132c91632e9479ee7250f3613358681f8322de5bda58d1): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:45.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:17:45.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:45.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:45.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:17:46.182 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(4c4c0de4f46489568099e6e63c9ce72ace7034a9305f145e4d4800ab2420e39a): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:46.743 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:17:46.765 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:17:46.766 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(5397902d57044ef2b78ed7406dbfbfb8118bd3236bf86ca6d16b63da4c3ae8c7): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:47.399 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Created container
Dec 27 16:17:47.665 I ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal Started container
Dec 27 16:17:47.922 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:17:57.703 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(778cbe558b390409d3d45f796156a4f0035b0d869948d34eaba306c47db4a5e9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:17:59.561 W ns=openshift-monitoring pod=prometheus-k8s-0 Back-off restarting failed container
Dec 27 16:17:59.699 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(f43da44bb9c05bd7fa16cd04218a97d9162578da65ce30ba2cb542c52d729acc): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:00.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:18:00.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:00.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:00.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:02.073 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(abf6992be00ddefd428469c22c55f202883bac105f80b41e452a560cdcc36b07): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:06.071 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:18:13.003 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(3e4c7c7feaefeda479c44e964358df6b89594eecd4e2983adbfd764165b08bfc): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:13.065 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(c59cf6b6deed99639d6e8a626cd6dfbc7320c0d91f99f5896e2eef878c080c73): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:15.577 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(399288cc7a51b1beb2266ef0c197679273258b4d722b55e32c737b4db237f0cb): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:18:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:15.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:15.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:16.269 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Back-off restarting failed container
Dec 27 16:18:26.680 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(b152cdfa61e7dce57877f72390a0587fc49cca5ff73548162c28fc6f79edf776): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:27.704 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(6ba60e8c06f92f90be14042a99cfa20fd8736e1d6bd43ad38de2724a6d816fb4): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:28.830 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:18:29.064 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:18:30.915 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:18:30.915 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:30.915 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:30.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:30.915 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:31.903 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(15425afbcaef7b44fd7b769573a3933e9e136cbc08a4225888972aae56d2df2b): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:40.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(03a20d33104f4ab8b2f4f8ccd940c24bebdfa8e473ea6fe8b24290940f661f6b): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:40.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(414a7dc035b8a3b363f10e97f707604840592e23c4ec9b772a8c6477f847476a): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:44.584 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(0adc965f68b1dabeb0fab5157cb91be84c739fc52192e586f0c520a2b4db316c): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:18:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:45.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:45.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:45.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:18:53.407 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(20a99919eb2e07b4eb336fa8742f61760ffcec5dfa02e3d794430a61c2fa4771): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:55.659 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(98b5f5b588a74f874027348a9f79258a89499c039f289d5734e58ec921a3ba92): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:18:59.054 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(4133da7c794ee5766d4363b71aadd79c0a9d778502649138073c0f7428eb2578): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:00.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:19:00.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:00.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:00.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:00.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:04.833 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:19:08.256 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:19:08.256 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(736ad1ca42084d0ab24dae9ba4045416ad43e7c846fb914e57271c294c9e07f8): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:08.378 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:19:09.131 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:19:11.839 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(c79be853a7c0c4c3b40e45a75bf13e983010a7568d6f1bff658d2377bef94ac0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:12.264 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Back-off restarting failed container
Dec 27 16:19:12.461 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(2b777d0e102aedae9d056db7bca44a71db962b0897e3ac881d11067279296aae): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:13.006 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:19:15.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:19:15.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:15.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:15.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:15.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:16.894 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:19:19.846 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(9a551b0042c42c8171f24588a70c13d95e02909e4d1f20c9a75143668e43ae25): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:23.205 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Status for operator openshift-cluster-kube-apiserver-operator changed
Dec 27 16:19:24.605 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(f59d1b842e97cc63518621ab66c3eb5d985472da7d906861c473553a60ecc684): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:26.278 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(6fe00201fa65bba989765360f2af64afd9153cf4791a9af2c8c8e6b14e8f5ee0): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:27.302 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:19:27.812 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:19:30.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:19:30.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:30.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:30.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:30.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:32.962 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(15ecef89ed75d1eebbf3dedb94744ace3baf5f5c50617841ca91c5b56559b15b): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:38.205 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(685ce48980b93fc6c5a1a835dec4364892f59cd11d25602ebb1fe22b0dfdc848): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:38.770 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(073354f2c373fb30dbddad49d04c38673d6a13293e7f43e2bace59f9f1f10599): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:19:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:45.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:45.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:45.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:19:46.758 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(1435166498ced2e659618214d27b976ae972f39e89e2fbbc21f8e682b727fb44): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:50.840 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(3ba3c97df2f2ffdb53ba729e9eff5aea20267e9a3203d766c3321dfc563bbca1): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:51.775 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(cab0dec92f7488ef66a618eb82a69fd743e38c388c90f9a02b878b9ce8e957b6): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:53.823 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:19:54.337 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(756729e02b5080ecb8ddfe12b923fd3da462f8bfb3d64e24b9adabd07d906136): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:19:59.251 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(cbe80e60e7c6c239f8287c10eab4b5c03c0e2a69d92dbebabd6f6bb04b4e8603): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:00.915 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:20:00.915 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:00.915 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:00.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:00.915 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:05.655 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(458cdd113f4b20b223a5fcdf86e3ba29d4fb866a82b66a1f2d3d5459204128f2): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:06.558 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(2869abee15fe93f35c835eceecb0cb56f6e44e975e36c1b08a100c7a102e7fa6): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:07.445 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:20:10.257 I ns=openshift-ingress service=router-default Ensuring load balancer
Dec 27 16:20:11.744 I ns=openshift-ingress service=router-default Ensured load balancer
Dec 27 16:20:14.459 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:20:14.713 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(bc6951bc4ba4786ae809798d7c22d831605ec6b522bb14a58922ed1a871a3887): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:15.915 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:20:15.915 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:15.915 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:15.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:15.915 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:17.683 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(09e005e3d40178a5bcb09818182ea73bdc58afcd35a24f48a75c665da785a595): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:20.707 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(017dde1f82cc05f362e52fe1237de358e3c92bb675a37e1b84ead8b698196dd2): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:29.095 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(2d204475d9b558103a1d7aedcc298896914d64a7b05f6c126b58d10963d5d693): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:30.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:20:30.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:30.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:30.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:30.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:32.646 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(be215ed3cdf727768f506d025e276143066c9232543d4fa25e2e9281ad331247): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:33.555 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(a4cc03601658cf5d6f3f4a03f806bf332c171978f4bac0493d546f6fd1df3dbf): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:40.314 W ns=openshift-sdn daemonset=ovs Error creating: Timeout: request did not complete within allowed duration
Dec 27 16:20:40.314 W ns=openshift-sdn daemonset=sdn-controller Error creating: Timeout: request did not complete within allowed duration
Dec 27 16:20:40.826 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(1fc10df970ffba2bae571298c547f241512c2aa5899e109e48b9fa2a1006e8f4): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:44.614 W ns=openshift-monitoring pod=alertmanager-main-2 Back-off restarting failed container
Dec 27 16:20:44.920 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:20:45.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:20:45.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:45.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:45.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:45.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:20:47.892 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(19b7eb6d7b9c3b4aea599b9ad507cbc49aaad1f20911caaf3a8d90903d581f0e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:48.404 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(3bea7efa0953c424153e0ef937efcb677d5da25c4d9b5408593c672fb7e5d60b): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:54.604 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(6de7a2caa919a6d2eb249b832584b1e813a76d45a0412f61f28c8aa22de98c72): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:20:54.966 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:21:00.692 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(4ad28241d8971c141b15bbb436f63976c0e8d5c65e020e26720ded35b669120d): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:21:00.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:00.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:00.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:00.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:03.866 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(e04d38721f63a380a52a947e1427d4b3a6b9f8d23b7cf16fdeb1f14eb325255e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:07.622 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:21:08.348 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(6a0c27e214e6c2ab522df127dc025c133a36144eaccafccc3dba78f07d8f5f9c): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:09.561 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:21:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:21:15.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:15.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:15.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:17.044 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(e2e887e83f07053244c661335145017844abb1efe4b7d91c60355a0662e01c44): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:23.822 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(a12debdb5be6dba66dfe67b3bb4e3443342ccf8a18d2d858c88c5243d40f19a7): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:26.600 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:21:26.631 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:21:30.594 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:21:30.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:21:30.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:30.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:30.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:30.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:31.412 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(98d07959e70c97039c9ec35fb46499982d5c084cc4a4f0020a444d831c47ac32): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:38.640 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(5862f52a784a4da825a42dd573880a28edcd4c9e9710ca23d6ed147209d80b3d): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:45.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:21:45.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:45.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:45.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:45.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:21:47.694 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(66f372815a9ed85950e6f0500c29ddcbad3d32ed2d3a47ab63cb349ec78351c9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:21:51.278 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(53229ab0ef44ad83ff61415d8682f54be2a54760c577a1992580cdc799da0fe6): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:22:00.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:22:00.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:00.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:00.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:00.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:03.946 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:22:04.181 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(97f825be44ff6c0ef9bdcafa76212ea1ebc6b258e2f431610f2fa9df596da2b2): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:22:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:22:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:15.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:15.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:15.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:18.987 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(216980452c67d41c87aea49e5f5c186dde9d8d9e01f0745e9e0f28669be89f1e): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:22:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:22:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:30.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:30.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:30.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:31.257 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:22:33.244 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(dbfdd737104f5cb4abe2459f16082a63ea418e0deb5ee9bffd7f2b388da9cac9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:22:35.751 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:22:39.177 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:22:45.626 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(524ea0ef26d6d38c43f449cba4a6abe02e4d743436cf38f9d1bcf9c2df0af936): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:22:45.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:22:45.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:45.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:45.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:22:45.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:00.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:23:00.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:00.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:00.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:00.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:01.623 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(8e8d2c3f7ce6baa85a2de4f3ccd986f7119928bd3cb0574bcbd68b17542c5be5): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:23:05.944 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container stopped being ready
Dec 27 16:23:06.153 W ns=openshift-monitoring pod=prometheus-k8s-0 Back-off restarting failed container
Dec 27 16:23:07.620 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Readiness probe failed: Get http://10.128.2.7:1936/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:23:09.411 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:23:15.202 W ns=openshift-monitoring pod=alertmanager-main-2 node=ip-10-0-161-51.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:23:15.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:23:15.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:15.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:15.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:15.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:28.564 W ns=openshift-apiserver pod=apiserver-wsjnb Back-off restarting failed container
Dec 27 16:23:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:23:30.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:30.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:30.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:31.171 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:23:42.972 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container stopped being ready
Dec 27 16:23:45.915 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:23:45.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:45.915 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:45.915 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:23:45.915 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:24:00.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:00.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:00.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:00.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:04.114 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:24:05.424 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:24:08.762 W ns=openshift-dns pod=dns-default-zv48s Back-off restarting failed container
Dec 27 16:24:10.314 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c Back-off restarting failed container
Dec 27 16:24:12.073 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Back-off restarting failed container
Dec 27 16:24:15.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:24:15.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:15.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:15.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:15.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:16.140 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:24:30.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:24:30.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:30.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:30.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:30.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:45.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:24:45.911 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:45.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:45.911 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:45.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:24:47.889 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:25:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:25:00.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:00.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:00.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:00.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:01.848 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(660cb59e159c0a7a3174a8aca1adaae591ef981b274b0f2cc8888a22ccb67e33): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:25:07.212 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:25:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:25:15.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:15.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:15.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:18.503 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:25:19.450 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(4e2c179f2239103a7c1df37e3d0e4610b758ba26818a35d46ad63b0b9f719e74): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:25:30.418 I ns=openshift-cluster-kube-controller-manager-operator deployment=openshift-cluster-kube-controller-manager-operator Status for operator openshift-cluster-kube-controller-manager-operator changed
Dec 27 16:25:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:25:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:30.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:30.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:30.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:37.446 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:25:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:25:45.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:45.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:45.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:25:59.808 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:26:00.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:26:00.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:00.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:00.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:00.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:04.314 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:26:04.323 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:26:04.511 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:26:15.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:26:15.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:15.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:15.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:15.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:16.705 W ns=openshift-console pod=openshift-console-5b88cf9649-h4xs2 Readiness probe failed: Get https://10.129.0.21:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:26:18.445 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:26:18.854 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(90a7d8632a0f875e425ac54fec82f98b91b92e274fa90cb052dea1a5affad05b): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:26:22.361 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container restarted
Dec 27 16:26:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:26:30.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:30.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:30.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:33.907 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 27 16:26:36.603 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:26:38.874 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:26:43.738 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:26:45.839 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:26:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:26:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:45.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:45.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:26:45.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:00.839 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container stopped being ready
Dec 27 16:27:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:27:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:00.911 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:00.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:00.911 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:04.037 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:27:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:27:15.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:15.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:15.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:15.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:17.224 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(bac1f43ece2158c008f3ba0b5e9a1f92f6f3632303c24609f9ceed3cccfcc930): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:27:30.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:27:30.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:30.911 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:30.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:30.911 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:27:45.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:45.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:45.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:45.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:27:59.412 W ns=openshift-monitoring pod=prometheus-k8s-0 Back-off restarting failed container
Dec 27 16:28:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:28:00.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:00.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:00.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:00.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:13.660 W ns=openshift-monitoring pod=prometheus-k8s-0 node=ip-10-0-161-51.ec2.internal container=prometheus-proxy container restarted
Dec 27 16:28:15.915 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:28:15.915 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:15.915 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:15.915 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:15.915 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:22.135 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 Back-off restarting failed container
Dec 27 16:28:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:28:30.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:30.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:30.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:30.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:31.357 W ns=openshift-sdn pod=sdn-z6fp6 Back-off restarting failed container
Dec 27 16:28:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:28:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:45.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:45.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:45.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:28:56.230 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:29:00.240 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:29:00.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:29:00.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:00.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:00.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:00.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:07.311 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk Back-off restarting failed container
Dec 27 16:29:07.404 W ns=openshift-apiserver pod=apiserver-7h7td Back-off restarting failed container
Dec 27 16:29:07.981 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l Back-off restarting failed container
Dec 27 16:29:11.988 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj Back-off restarting failed container
Dec 27 16:29:15.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:29:15.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:15.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:15.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:15.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:22.460 I ns=openshift-cluster-kube-apiserver-operator deployment=openshift-cluster-kube-apiserver-operator Status for operator openshift-cluster-kube-apiserver-operator changed
Dec 27 16:29:30.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:29:30.910 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:30.910 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:30.910 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:30.910 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:39.184 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:29:45.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:29:45.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:45.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:45.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:45.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:29:53.750 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-5-ip-10-0-12-206.ec2.internal_openshift-kube-apiserver_e071170c-09f1-11e9-93ce-0edf97ba3274_0(314cbc811f5f8c96c2741213d1d30897eea8f3a7eb4dca452c42988af26d88ed): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:30:00.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:30:00.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:00.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:00.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:00.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:04.399 W ns=openshift-dns pod=dns-default-zv48s Liveness probe failed: Get http://10.130.0.3:8080/health: dial tcp 10.130.0.3:8080: connect: no route to host
Dec 27 16:30:15.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:30:15.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:15.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:15.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:15.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:21.618 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt_openshift-cluster-openshift-apiserver-operator_a057d75c-09f2-11e9-b3b2-0a7ded5c62ec_0(558096c79c7eba21ba5ad41b4d12e8b4ee08f4cedb9ec64d71c619b47189a6b5): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:30:30.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:30:30.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:30.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:30.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:30.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
# BUG: New round of sdn updates are applied, disrupting all pods, why?
Dec 27 16:30:33.814 I ns=openshift-sdn daemonset=sdn-controller Created pod: sdn-controller-n9jvz
Dec 27 16:30:33.821 I ns=openshift-sdn daemonset=ovs Created pod: ovs-zs7mh
Dec 27 16:30:34.448 I ns=openshift-sdn pod=sdn-controller-n9jvz pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:30:34.460 I ns=openshift-sdn pod=ovs-zs7mh Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:30:34.921 I ns=openshift-sdn pod=ovs-zs7mh Created container
Dec 27 16:30:34.955 I ns=openshift-sdn pod=ovs-zs7mh Started container
Dec 27 16:30:36.156 I ns=openshift-sdn pod=sdn-controller-n9jvz Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:30:36.599 I ns=openshift-sdn pod=sdn-controller-n9jvz Created container
Dec 27 16:30:36.603 I ns=openshift-sdn pod=sdn-controller-n9jvz Started container
Dec 27 16:30:36.607 W ns=openshift-sdn pod=ovs-2rwxh node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:30:36.715 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-2rwxh
Dec 27 16:30:36.727 W ns=openshift-sdn pod=sdn-controller-hdj2k node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:30:36.800 I ns=openshift-sdn daemonset=sdn-controller Deleted pod: sdn-controller-hdj2k
Dec 27 16:30:45.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:30:45.914 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:45.914 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:45.914 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:45.914 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:30:47.562 I ns=openshift-sdn pod=ovs-2rwxh Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:30:48.681 W ns=openshift-sdn pod=ovs-2rwxh node=ip-10-0-153-178.ec2.internal container=openvswitch container stopped being ready
Dec 27 16:30:49.723 W ns=openshift-sdn pod=ovs-2rwxh node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:30:49.862 I ns=openshift-sdn daemonset=ovs Created pod: ovs-vnvn4
Dec 27 16:30:50.525 I ns=openshift-sdn pod=ovs-vnvn4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:30:51.192 I ns=openshift-sdn pod=ovs-vnvn4 Created container
Dec 27 16:30:51.356 I ns=openshift-sdn pod=ovs-vnvn4 Started container
Dec 27 16:30:51.690 W ns=openshift-sdn pod=ovs-2rsgq node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:30:51.777 I ns=openshift-sdn daemonset=ovs Deleted pod: ovs-2rsgq
Dec 27 16:30:51.777 W ns=openshift-sdn pod=sdn-6wmck node=ip-10-0-153-178.ec2.internal container=sdn container stopped being ready
Dec 27 16:30:51.792 I ns=openshift-sdn pod=sdn-6wmck Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:30:52.032 I ns=openshift-sdn pod=sdn-6wmck Created container
Dec 27 16:30:52.066 I ns=openshift-sdn pod=sdn-6wmck Started container
Dec 27 16:30:52.963 W ns=openshift-sdn pod=sdn-6wmck node=ip-10-0-153-178.ec2.internal container=sdn container restarted
Dec 27 16:30:53.751 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Readiness probe failed: Get https://10.131.0.8:5000/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:30:54.827 W ns=openshift-dns pod=dns-default-r77bg Liveness probe failed: Get http://10.131.0.2:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:30:54.827 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:30:55.154 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:30:55.543 W ns=openshift-dns pod=dns-default-r77bg The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:30:55.556 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container stopped being ready
Dec 27 16:30:55.590 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj node=ip-10-0-153-178.ec2.internal container=csi-operator container stopped being ready
Dec 27 16:30:55.870 I ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj Pod sandbox changed, it will be killed and re-created.
Dec 27 16:30:56.008 I ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj Pod sandbox changed, it will be killed and re-created.
Dec 27 16:30:56.771 I ns=openshift-sdn pod=ovs-2rsgq Killing container with id cri-o://openvswitch:Need to kill Pod
Dec 27 16:30:57.109 W ns=openshift-sdn pod=ovs-2rsgq node=ip-10-0-30-101.ec2.internal container=openvswitch container stopped being ready
Dec 27 16:30:57.694 I ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:65257906a48860c744dc1cd7ba790fbed9ed17fe6491993551c51d7dcbace0ac" already present on machine
Dec 27 16:30:57.735 I ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:8fc6598b987000da411e1ba11179e9df429693d7fe0bd1c1c2cef1f740e4a731"
Dec 27 16:30:57.875 I ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj Created container
Dec 27 16:30:57.909 I ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj Started container
Dec 27 16:30:58.512 I ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:8fc6598b987000da411e1ba11179e9df429693d7fe0bd1c1c2cef1f740e4a731"
Dec 27 16:30:58.546 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Liveness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:30:58.589 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj node=ip-10-0-153-178.ec2.internal container=csi-operator container restarted
Dec 27 16:30:58.685 I ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj Created container
Dec 27 16:30:58.716 I ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj Started container
Dec 27 16:30:58.795 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Readiness probe failed: Get https://10.131.0.8:5000/healthz: dial tcp 10.131.0.8:5000: connect: no route to host
Dec 27 16:30:58.846 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Liveness probe failed: Get https://10.131.0.8:5000/healthz: dial tcp 10.131.0.8:5000: connect: no route to host
Dec 27 16:30:59.744 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container restarted
Dec 27 16:31:00.728 W ns=openshift-dns pod=dns-default-r77bg Liveness probe failed: Get http://10.131.0.2:8080/health: dial tcp 10.131.0.2:8080: connect: no route to host
Dec 27 16:31:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:31:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:00.911 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:00.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:00.911 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:02.622 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:03.328 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:03.498 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:03.535 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:04.854 W ns=openshift-apiserver pod=apiserver-8htcf Liveness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:05.886 W ns=openshift-dns pod=dns-default-lptd4 Liveness probe failed: Get http://10.128.0.13:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:05.984 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:06.398 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:06.617 W ns=openshift-dns pod=dns-default-r77bg node=ip-10-0-153-178.ec2.internal container=dns container stopped being ready
Dec 27 16:31:06.617 W ns=openshift-dns pod=dns-default-r77bg node=ip-10-0-153-178.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:31:06.651 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz node=ip-10-0-153-178.ec2.internal container=registry container stopped being ready
Dec 27 16:31:06.865 I ns=openshift-dns pod=dns-default-r77bg Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:06.906 I ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:06.964 I ns=openshift-sdn pod=sdn-controller-hdj2k Killing container with id cri-o://sdn-controller:Need to kill Pod
Dec 27 16:31:07.017 W ns=openshift-sdn pod=sdn-448jh node=ip-10-0-30-101.ec2.internal container=sdn container stopped being ready
Dec 27 16:31:07.183 I ns=openshift-sdn pod=sdn-448jh Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:31:07.626 I ns=openshift-sdn pod=sdn-448jh Created container
Dec 27 16:31:07.645 I ns=openshift-sdn pod=sdn-448jh Started container
Dec 27 16:31:08.097 W ns=openshift-sdn pod=sdn-controller-hdj2k node=ip-10-0-12-206.ec2.internal container=sdn-controller container stopped being ready
Dec 27 16:31:08.167 W ns=openshift-sdn pod=sdn-448jh node=ip-10-0-30-101.ec2.internal container=sdn container restarted
Dec 27 16:31:08.650 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Liveness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:08.801 I ns=openshift-dns pod=dns-default-r77bg Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:31:08.835 I ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:a6c7791ff33871750ddb580eb0784c371d3dd5f359d6ead3934853c40caa2db0" already present on machine
Dec 27 16:31:09.110 I ns=openshift-dns pod=dns-default-r77bg Created container
Dec 27 16:31:09.179 W ns=openshift-sdn pod=ovs-2rsgq node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:31:09.232 I ns=openshift-dns pod=dns-default-r77bg Started container
Dec 27 16:31:09.295 W ns=openshift-sdn pod=sdn-controller-hdj2k node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:31:09.297 I ns=openshift-dns pod=dns-default-r77bg Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9c116e0a8c006f3c67f074a0372c6b4f977161386a95d41aa0faabbe94a558f0" already present on machine
Dec 27 16:31:09.326 I ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Created container
Dec 27 16:31:09.461 I ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz Started container
Dec 27 16:31:09.638 I ns=openshift-dns pod=dns-default-r77bg Created container
Dec 27 16:31:09.681 I ns=openshift-dns pod=dns-default-r77bg Started container
Dec 27 16:31:09.754 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz node=ip-10-0-153-178.ec2.internal container=registry container restarted
Dec 27 16:31:09.984 W ns=openshift-dns pod=dns-default-r77bg node=ip-10-0-153-178.ec2.internal container=dns container restarted
Dec 27 16:31:09.984 W ns=openshift-dns pod=dns-default-r77bg node=ip-10-0-153-178.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:31:10.801 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-scheduler_bdc82a02-09f2-11e9-93ce-0edf97ba3274_0(d352561a94d2eb057a71d0d14e983a65726216f760c27bb23ff32efa89203550): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:31:11.844 I ns=kube-system configmap=openshift-master-controllers controller-manager-2qgd4 became leader
Dec 27 16:31:12.541 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:13.360 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:13.573 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:13.594 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:15.006 W ns=openshift-apiserver pod=apiserver-8htcf Liveness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:15.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:31:15.912 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:15.912 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:15.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:15.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:15.953 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container stopped being ready
Dec 27 16:31:16.012 W ns=openshift-dns pod=dns-default-lptd4 Liveness probe failed: Get http://10.128.0.13:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:16.778 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:16.779 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk node=ip-10-0-153-178.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:31:17.010 W ns=openshift-monitoring pod=alertmanager-main-0 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:17.012 I ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:17.695 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:18.034 I ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:18.034 I ns=openshift-monitoring pod=alertmanager-main-0 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:18.034 W ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:18.034 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj node=ip-10-0-153-178.ec2.internal container=ingress-operator container stopped being ready
Dec 27 16:31:18.215 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager container stopped being ready
Dec 27 16:31:18.215 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:31:18.215 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=config-reloader container stopped being ready
Dec 27 16:31:18.395 I node=ip-10-0-153-178.ec2.internal Starting openshift-sdn.
Dec 27 16:31:18.577 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Liveness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:18.758 I ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:2cc5337189a3690dc111ff8d87a40a12a45010285e438c92d39f03a95571c1ea" already present on machine
Dec 27 16:31:18.907 W ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj node=ip-10-0-153-178.ec2.internal container=prometheus-adapter container stopped being ready
Dec 27 16:31:18.921 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd node=ip-10-0-153-178.ec2.internal container=cluster-monitoring-operator container stopped being ready
Dec 27 16:31:18.937 I ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk Created container
Dec 27 16:31:19.087 I ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk Started container
Dec 27 16:31:19.087 I ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:19.087 I ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:19.267 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Killing container with id cri-o://console:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:31:19.267 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:31:19.718 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk node=ip-10-0-153-178.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:31:19.822 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Created container
Dec 27 16:31:19.870 I ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:90b96750f8e233cbc3f1f876d9e41ce6b284fc0cd9769adb6ea1579e15df71ca" already present on machine
Dec 27 16:31:19.911 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Started container
Dec 27 16:31:20.012 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 node=ip-10-0-30-101.ec2.internal container=console container restarted
Dec 27 16:31:20.012 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 node=ip-10-0-30-101.ec2.internal container=console container stopped being ready
Dec 27 16:31:20.122 I ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj Created container
Dec 27 16:31:20.230 I ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj Started container
Dec 27 16:31:20.246 I ns=openshift-monitoring pod=alertmanager-main-0 Container image "openshift/prometheus-alertmanager:v0.15.2" already present on machine
Dec 27 16:31:20.629 I ns=openshift-monitoring pod=alertmanager-main-0 Created container
Dec 27 16:31:20.703 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj node=ip-10-0-153-178.ec2.internal container=ingress-operator container restarted
Dec 27 16:31:20.808 I ns=openshift-monitoring pod=alertmanager-main-0 Started container
Dec 27 16:31:20.912 I ns=openshift-monitoring pod=alertmanager-main-0 Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine
Dec 27 16:31:21.091 I ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj Container image "quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1" already present on machine
Dec 27 16:31:21.130 I ns=openshift-monitoring pod=alertmanager-main-0 Created container
Dec 27 16:31:21.161 I ns=openshift-monitoring pod=alertmanager-main-0 Started container
Dec 27 16:31:21.174 I ns=openshift-monitoring pod=alertmanager-main-0 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:31:21.227 I ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj Created container
Dec 27 16:31:21.265 I ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj Started container
Dec 27 16:31:21.415 I ns=openshift-monitoring pod=alertmanager-main-0 Created container
Dec 27 16:31:21.432 I ns=openshift-monitoring pod=alertmanager-main-0 Started container
Dec 27 16:31:21.758 W ns=openshift-monitoring pod=prometheus-adapter-7fb498bdc-sbcvj node=ip-10-0-153-178.ec2.internal container=prometheus-adapter container restarted
Dec 27 16:31:22.577 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:23.169 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:31:23.212 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:23.331 I ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:7e7e9074ae67ef2da6d80bb27880af271e0ca2043b563c9773933738a525adfd" already present on machine
Dec 27 16:31:23.397 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:23.483 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:23.485 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq node=ip-10-0-30-101.ec2.internal container=packageserver container stopped being ready
Dec 27 16:31:24.082 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Killing container with id cri-o://packageserver:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:31:24.082 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc"
Dec 27 16:31:24.098 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager container restarted
Dec 27 16:31:24.098 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:31:24.098 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=config-reloader container restarted
Dec 27 16:31:24.727 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc"
Dec 27 16:31:24.922 W ns=openshift-apiserver pod=apiserver-8htcf Liveness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:25.539 I ns=openshift-apiserver pod=apiserver-8htcf Killing container with id cri-o://openshift-apiserver:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:31:25.539 I ns=openshift-apiserver pod=apiserver-8htcf pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:25.545 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Created container
Dec 27 16:31:25.599 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Started container
Dec 27 16:31:25.705 I ns=openshift-apiserver pod=apiserver-8htcf Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:28.914 E kube-apiserver Kube API started failing: Get https://ci-ln-y7gvwf2-703b0-api.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=3s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:30.913 E kube-apiserver Kube API is not responding to GET requests
Dec 27 16:31:30.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:31:30.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:30.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:30.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:30.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:32.912 I kube-apiserver Kube API started responding to GET requests
Dec 27 16:31:32.913 W ns=openshift-dns pod=dns-default-lptd4 Liveness probe failed: Get http://10.128.0.13:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:33.092 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal container=catalog-operator container restarted
Dec 27 16:31:33.104 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq node=ip-10-0-30-101.ec2.internal container=packageserver container restarted
Dec 27 16:31:33.118 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd node=ip-10-0-153-178.ec2.internal container=cluster-monitoring-operator container restarted
Dec 27 16:31:33.315 I ns=openshift-apiserver pod=apiserver-8htcf Created container
Dec 27 16:31:33.613 I ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd Created container
Dec 27 16:31:33.648 I ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd Started container
Dec 27 16:31:33.806 I ns=openshift-apiserver pod=apiserver-8htcf Started container
Dec 27 16:31:33.806 I ns=openshift-sdn daemonset=sdn-controller Created pod: sdn-controller-h5vss
Dec 27 16:31:33.806 I ns=openshift-sdn daemonset=ovs Created pod: ovs-wv9vs
Dec 27 16:31:33.973 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: dial tcp 10.128.0.16:5443: connect: no route to host
Dec 27 16:31:34.154 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:34.167 W ns=openshift-apiserver pod=apiserver-8htcf node=ip-10-0-30-101.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:31:34.167 W ns=openshift-apiserver pod=apiserver-8htcf node=ip-10-0-30-101.ec2.internal container=openshift-apiserver container stopped being ready
Dec 27 16:31:34.335 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.24:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:34.337 I ns=openshift-sdn pod=sdn-controller-h5vss pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:34.668 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:34.668 W ns=openshift-dns pod=dns-default-lptd4 Liveness probe failed: Get http://10.128.0.13:8080/health: dial tcp 10.128.0.13:8080: connect: no route to host
Dec 27 16:31:34.668 I ns=openshift-sdn pod=ovs-wv9vs Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b892f981815a8a3a0756a029087c3183e960d0bb11b5280e447b0d0342ba93f6" already present on machine
Dec 27 16:31:35.032 I ns=openshift-sdn pod=ovs-wv9vs Created container
Dec 27 16:31:35.033 I ns=openshift-sdn pod=ovs-wv9vs Started container
Dec 27 16:31:36.298 I ns=openshift-sdn pod=sdn-controller-h5vss Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:36.913 I ns=openshift-sdn pod=sdn-controller-h5vss Created container
Dec 27 16:31:36.929 I ns=openshift-sdn pod=sdn-controller-h5vss Started container
Dec 27 16:31:38.347 W ns=openshift-apiserver pod=apiserver-8htcf The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:41.316 I ns=openshift-csi-operator configmap=csi-operator-leader csi-operator-6d644cdb5f-nlwgj_e270aaca-09f4-11e9-98aa-0a580a83000b became leader
Dec 27 16:31:42.647 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:42.937 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.24:8443/healthz: dial tcp 10.128.0.24:8443: connect: no route to host
Dec 27 16:31:43.467 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:43.522 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:44.144 W ns=openshift-dns pod=dns-default-lptd4 Liveness probe failed: Get http://10.128.0.13:8080/health: dial tcp 10.128.0.13:8080: connect: no route to host
Dec 27 16:31:44.491 I ns=openshift-dns pod=dns-default-lptd4 Killing container with id cri-o://dns:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:31:44.491 I ns=openshift-dns pod=dns-default-lptd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:31:44.789 I ns=openshift-dns pod=dns-default-lptd4 Created container
Dec 27 16:31:44.880 I ns=openshift-dns pod=dns-default-lptd4 Started container
Dec 27 16:31:45.186 W ns=openshift-dns pod=dns-default-lptd4 node=ip-10-0-30-101.ec2.internal container=dns container restarted
Dec 27 16:31:45.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:31:45.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:45.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:45.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:45.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:31:48.952 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:49.316 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:49.471 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 node=ip-10-0-30-101.ec2.internal container=cluster-autoscaler-operator container stopped being ready
Dec 27 16:31:49.511 I ns=openshift-apiserver pod=apiserver-8htcf Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:49.651 I ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:49.831 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:50.524 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw node=ip-10-0-30-101.ec2.internal container=machine-api-operator container stopped being ready
Dec 27 16:31:50.560 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 node=ip-10-0-30-101.ec2.internal container=kube-scheduler-operator-container container stopped being ready
Dec 27 16:31:50.705 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:50.705 I ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:50.885 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:31:51.199 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:51.224 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:31:51.557 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:31:51.793 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:51.905 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager-proxy container stopped being ready
Dec 27 16:31:51.924 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9667fba1d70aa2757ec571b2b4d3e6b052be0aed024a58b56853cb59699c0b75"
Dec 27 16:31:51.948 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:31:52.021 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:52.140 I ns=openshift-apiserver pod=apiserver-8htcf pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:52.153 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:31:52.198 W ns=openshift-monitoring pod=alertmanager-main-0 Back-off restarting failed container
Dec 27 16:31:52.314 I ns=openshift-apiserver pod=apiserver-8htcf Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:31:52.531 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Readiness probe failed: Get https://10.128.0.23:8443/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:52.681 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9667fba1d70aa2757ec571b2b4d3e6b052be0aed024a58b56853cb59699c0b75"
Dec 27 16:31:53.060 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:31:53.399 W ns=openshift-controller-manager pod=controller-manager-qbf49 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:31:53.399 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:53.809 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Liveness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:53.883 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Readiness probe failed: Get https://10.128.0.16:5443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:31:53.955 I ns=openshift-apiserver pod=apiserver-8htcf Created container
Dec 27 16:31:54.192 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:31:54.306 I ns=openshift-apiserver pod=apiserver-8htcf Started container
Dec 27 16:31:54.647 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:55.465 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal container=olm-operator container restarted
Dec 27 16:31:55.755 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:d8ea70d99ef39234c77c361c7e6cebf0ba52c300aabec56089010f44c35acf79"
Dec 27 16:31:55.755 W ns=openshift-apiserver pod=apiserver-8htcf node=ip-10-0-30-101.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:31:55.855 I ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:5839c85d047754ec8a483d667b20194018709f19d9f36fa38d803f38dc69c468" already present on machine
Dec 27 16:31:57.099 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:d8ea70d99ef39234c77c361c7e6cebf0ba52c300aabec56089010f44c35acf79"
Dec 27 16:31:57.539 I ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:77557a27649cd869ee842edff81c0aa3362632740e6c15f9d2dd3b74d702a6e6" already present on machine
Dec 27 16:31:57.539 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Back-off restarting failed container
Dec 27 16:31:57.554 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal container=openshift-apiserver container restarted
Dec 27 16:31:57.600 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:31:57.760 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt Created container
Dec 27 16:31:58.316 I ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt Started container
Dec 27 16:31:58.401 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:31:58.461 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Back-off restarting failed container
Dec 27 16:31:58.575 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Back-off restarting failed container
Dec 27 16:31:58.827 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt node=ip-10-0-30-101.ec2.internal container=operator container restarted
Dec 27 16:31:59.354 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Back-off restarting failed container
Dec 27 16:32:00.910 I ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 Created container
Dec 27 16:32:00.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:32:00.911 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:00.911 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:00.911 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:00.911 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:00.958 I ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 Started container
Dec 27 16:32:00.975 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Back-off restarting failed container
Dec 27 16:32:01.320 I ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw Created container
Dec 27 16:32:01.357 I ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw Started container
Dec 27 16:32:01.413 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq Killing container with id cri-o://packageserver:Container failed liveness probe.. Container will be killed and recreated.
Dec 27 16:32:01.602 I ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:ed5621d167cb9c0d0d389bf4dddf3f2b0270b60fa41530454c4ac353504157cc"
Dec 27 16:32:02.616 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:86f6104f0b2d69a5c0133fa92f2325bbc78c398ba1000f10927f3a9d3b36e792" already present on machine
Dec 27 16:32:03.123 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 node=ip-10-0-30-101.ec2.internal container=cluster-autoscaler-operator container restarted
Dec 27 16:32:03.152 W ns=openshift-apiserver pod=apiserver-8htcf Readiness probe failed: Get https://10.128.0.31:8443/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 16:32:04.011 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:04.011 W ns=openshift-monitoring pod=grafana-58456d859d-n8s7j Readiness probe failed: dial tcp 10.128.2.8:3000: i/o timeout
Dec 27 16:32:04.230 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Created container
Dec 27 16:32:04.389 I ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 Started container
Dec 27 16:32:04.751 I ns=openshift-controller-manager pod=controller-manager-qbf49 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:04.751 W ns=openshift-dns pod=dns-default-lptd4 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:05.649 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:05.827 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw node=ip-10-0-30-101.ec2.internal container=machine-api-operator container restarted
Dec 27 16:32:05.997 I ns=openshift-monitoring pod=alertmanager-main-0 Container image "openshift/oauth-proxy:v1.1.0" already present on machine
Dec 27 16:32:06.148 I ns=openshift-monitoring pod=alertmanager-main-0 Created container
Dec 27 16:32:06.148 I ns=openshift-monitoring pod=alertmanager-main-0 Started container
Dec 27 16:32:06.164 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq node=ip-10-0-30-101.ec2.internal container=packageserver container restarted
Dec 27 16:32:06.370 I ns=openshift-controller-manager pod=controller-manager-qbf49 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e" already present on machine
Dec 27 16:32:06.510 W ns=openshift-controller-manager pod=controller-manager-qbf49 node=ip-10-0-30-101.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:32:06.814 W ns=openshift-monitoring pod=alertmanager-main-0 node=ip-10-0-153-178.ec2.internal container=alertmanager-proxy container restarted
Dec 27 16:32:06.954 W ns=openshift-console pod=openshift-console-5b88cf9649-wmnp7 node=ip-10-0-30-101.ec2.internal container=console container restarted
Dec 27 16:32:07.040 I ns=openshift-controller-manager pod=controller-manager-qbf49 Created container
Dec 27 16:32:07.065 I ns=openshift-controller-manager pod=controller-manager-qbf49 Started container
Dec 27 16:32:07.327 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:32:08.864 W ns=openshift-controller-manager pod=controller-manager-qbf49 node=ip-10-0-30-101.ec2.internal container=controller-manager container restarted
Dec 27 16:32:09.272 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:3b36bf42c00344584a4974c37ff3db18e2d357fb726c35a17cbce43944632d91" already present on machine
Dec 27 16:32:10.130 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Created container
Dec 27 16:32:10.501 I ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw Started container
Dec 27 16:32:11.006 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:32:11.320 W ns=openshift-core-operators pod=origin-cluster-osin-operator2-5d7bf9d7d-wqdxw node=ip-10-0-30-101.ec2.internal container=operator container restarted
Dec 27 16:32:11.918 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:c2c492a5df65f30e2f485d772a691d6d07ae224baef361a854c49e7c6183a192"
Dec 27 16:32:12.106 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Created container
Dec 27 16:32:12.489 I ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 Started container
Dec 27 16:32:12.592 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-2-ip-10-0-12-206.ec2.internal_openshift-kube-controller-manager_e1adb82a-09f2-11e9-93ce-0edf97ba3274_0(6838744d7feb3bb9a28196c09eec776f1226472b62cc1c2aab0bce2fb1498ba9): failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cni-server.sock: connect: connection refused
Dec 27 16:32:13.775 W ns=openshift-apiserver pod=apiserver-8htcf Back-off restarting failed container
Dec 27 16:32:14.118 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9667fba1d70aa2757ec571b2b4d3e6b052be0aed024a58b56853cb59699c0b75"
Dec 27 16:32:14.996 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9667fba1d70aa2757ec571b2b4d3e6b052be0aed024a58b56853cb59699c0b75"
Dec 27 16:32:15.096 W ns=openshift-apiserver pod=apiserver-8htcf Back-off restarting failed container
Dec 27 16:32:15.416 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Created container
Dec 27 16:32:15.445 W ns=openshift-sdn pod=sdn-z6fp6 node=ip-10-0-12-206.ec2.internal container=sdn container restarted
Dec 27 16:32:15.825 I ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv Started container
Dec 27 16:32:15.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:32:15.913 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:15.913 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:15.913 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:15.913 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:16.749 I ns=openshift-cluster-openshift-apiserver-operator configmap=openshift-cluster-openshift-apiserver-operator-lock f7788d8a-09f4-11e9-936a-0a580a800020 became leader
Dec 27 16:32:16.954 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:32:17.100 W ns=openshift-cluster-kube-scheduler-operator pod=openshift-cluster-kube-scheduler-operator-9cbc6d5db-jxk25 node=ip-10-0-30-101.ec2.internal container=kube-scheduler-operator-container container restarted
Dec 27 16:32:17.914 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv node=ip-10-0-30-101.ec2.internal container=operator container restarted
Dec 27 16:32:17.948 W ns=openshift-apiserver pod=apiserver-7h7td The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:18.152 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:19.436 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:19.713 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:32:19.713 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-controller container stopped being ready
Dec 27 16:32:19.713 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-healthcheck container stopped being ready
Dec 27 16:32:19.713 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=nodelink-controller container stopped being ready
Dec 27 16:32:19.714 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Updated DaemonSetapps./apiserver -n openshift-apiserver because it changed
Dec 27 16:32:19.829 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:19.939 W ns=openshift-controller-manager pod=controller-manager-58r2w The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:20.129 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:32:21.333 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:6562fd93b4332df7a9f8e9cb019c9ed0440ad77ed925702901812d3ddcf294dd" already present on machine
Dec 27 16:32:21.696 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Created container
Dec 27 16:32:21.847 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Started container
Dec 27 16:32:22.029 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:6562fd93b4332df7a9f8e9cb019c9ed0440ad77ed925702901812d3ddcf294dd" already present on machine
Dec 27 16:32:22.210 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Created container
Dec 27 16:32:22.210 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Started container
Dec 27 16:32:22.225 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:77557a27649cd869ee842edff81c0aa3362632740e6c15f9d2dd3b74d702a6e6" already present on machine
Dec 27 16:32:22.629 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Created container
Dec 27 16:32:22.744 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Started container
Dec 27 16:32:23.030 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:32:23.056 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:77557a27649cd869ee842edff81c0aa3362632740e6c15f9d2dd3b74d702a6e6" already present on machine
Dec 27 16:32:23.508 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal graceful deletion within 0s
Dec 27 16:32:23.605 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Created container
Dec 27 16:32:23.710 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:32:23.841 W ns=openshift-apiserver pod=apiserver-8htcf node=ip-10-0-30-101.ec2.internal graceful deletion within 70s
Dec 27 16:32:23.852 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal graceful deletion within 70s
Dec 27 16:32:23.865 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=controller-manager container restarted
Dec 27 16:32:23.865 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-controller container restarted
Dec 27 16:32:23.865 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-healthcheck container restarted
Dec 27 16:32:23.865 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=nodelink-controller container restarted
Dec 27 16:32:23.892 I ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq Started container
Dec 27 16:32:24.012 I ns=openshift-apiserver daemonset=apiserver Deleted pod: apiserver-8htcf
Dec 27 16:32:24.029 I ns=openshift-apiserver daemonset=apiserver Deleted pod: apiserver-7h7td
Dec 27 16:32:24.721 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:32:25.681 W ns=openshift-dns pod=dns-default-lptd4 node=ip-10-0-30-101.ec2.internal container=dns container stopped being ready
Dec 27 16:32:25.681 W ns=openshift-dns pod=dns-default-lptd4 node=ip-10-0-30-101.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:32:25.889 I ns=openshift-dns pod=dns-default-lptd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:32:26.043 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:26.319 W ns=openshift-dns pod=dns-default-lptd4 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:32:26.347 I ns=openshift-dns pod=dns-default-lptd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:9c116e0a8c006f3c67f074a0372c6b4f977161386a95d41aa0faabbe94a558f0" already present on machine
Dec 27 16:32:26.410 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:32:26.689 W ns=openshift-dns pod=dns-default-lptd4 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:32:27.602 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Created container
Dec 27 16:32:27.602 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Started container
Dec 27 16:32:27.844 I ns=openshift-cluster-openshift-apiserver-operator deployment=openshift-cluster-openshift-apiserver-operator Status for operator openshift-cluster-openshift-apiserver-operator changed
Dec 27 16:32:28.063 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:32:28.523 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:98d7bb1d9ade16fade3f3112f141e9f5e8ab9716f79e5f55213ee46a4ca5a5a7"
Dec 27 16:32:29.000 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Created container
Dec 27 16:32:29.507 I ns=openshift-dns pod=dns-default-lptd4 Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:29.550 I ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Started container
Dec 27 16:32:30.410 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Back-off restarting failed container
Dec 27 16:32:30.912 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:32:30.912 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:30.912 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal pod has been pending longer than a minute
Dec 27 16:32:31.010 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=scheduler container restarted
Dec 27 16:32:31.288 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:31.288 I ns=openshift-controller-manager pod=controller-manager-58r2w Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:32.109 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal Back-off restarting failed container
Dec 27 16:32:32.321 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq node=ip-10-0-30-101.ec2.internal container=packageserver container restarted
Dec 27 16:32:32.400 I ns=openshift-dns pod=dns-default-lptd4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8" already present on machine
Dec 27 16:32:32.418 W ns=openshift-controller-manager pod=controller-manager-58r2w node=ip-10-0-12-206.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:32:32.496 I ns=openshift-cluster-kube-scheduler-operator configmap=openshift-cluster-kube-scheduler-operator-lock 00dcde0c-09f5-11e9-a5d8-0a580a800023 became leader
Dec 27 16:32:32.998 I ns=openshift-dns pod=dns-default-lptd4 Created container
Dec 27 16:32:33.013 W ns=openshift-dns pod=dns-default-lptd4 node=ip-10-0-30-101.ec2.internal container=dns container restarted
Dec 27 16:32:33.190 W ns=openshift-dns pod=dns-default-lptd4 Back-off restarting failed container
Dec 27 16:32:33.194 I ns=openshift-cluster-kube-scheduler-operator deployment=openshift-cluster-kube-scheduler-operator Status for operator openshift-cluster-kube-scheduler-operator changed
Dec 27 16:32:33.746 I ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:b532d9351b803b5a03cf6777f12d725b4973851957497ea3e2b37313aadd6750" already present on machine
Dec 27 16:32:33.794 I ns=openshift-dns pod=dns-default-lptd4 Started container
Dec 27 16:32:34.353 W ns=openshift-dns pod=dns-default-zv48s The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:34.633 W ns=openshift-dns pod=dns-default-lptd4 Back-off restarting failed container
Dec 27 16:32:34.677 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 Error: container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/0/ns/ipc\" caused \"lstat /proc/0/ns/ipc: no such file or directory\""
Dec 27 16:32:34.760 W ns=openshift-dns pod=dns-default-lptd4 Back-off restarting failed container
Dec 27 16:32:34.772 W ns=openshift-apiserver pod=apiserver-7h7td node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:32:35.783 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 node=ip-10-0-12-206.ec2.internal container=operator container stopped being ready
Dec 27 16:32:35.966 I ns=openshift-controller-manager pod=controller-manager-58r2w Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-190509@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e" already present on machine
Dec 27 16:32:36.327 I ns=openshift-cluster-openshift-controller-manager-operator configmap=openshift-cluster-openshift-controller-manager-operator-lock 02df6a52-09f5-11e9-ab77-0a580a800024 became leader
Dec 27 16:32:36.668 I ns=openshift-controller-manager pod=controller-manager-58r2w Created container
Dec 27 16:32:37.027 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:37.027 I ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 Container image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:5849e5de586be7ee88fb46f55d2ef46c0d3bffe61748412ae4c8c02be0a82530" already present on machine
Dec 27 16:32:37.061 W ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 node=ip-10-0-33-43.ec2.internal container=cluster-network-operator container stopped being ready
Dec 27 16:32:37.389 I ns=openshift-controller-manager pod=controller-manager-58r2w Started container
Dec 27 16:32:37.389 W ns=openshift-operator-lifecycle-manager pod=certified-operators-mbvrl The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:37.390 I ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:37.390 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal invariant violation (bug): static pod should not transition Running->Pending with same UID
Dec 27 16:32:37.390 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=openshift-kube-apiserver container stopped being ready
Dec 27 16:32:37.571 I ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal pulling image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:32:37.724 I ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h Pod sandbox changed, it will be killed and re-created.
Dec 27 16:32:37.724 I ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-26-225801@sha256:598a3352576b336b1ce2344a1a09a8a0e163feb737c405fb88d2a06b2c075b6e"
Dec 27 16:32:37.725 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz node=ip-10-0-30-101.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:32:37.735 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=installer container stopped being ready
Dec 27 16:32:37.752 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq The pod's network interface has been lost and the pod will be stopped.
Dec 27 16:32:37.905 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h node=ip-10-0-30-101.ec2.internal container=machine-config-operator container stopped being ready
Dec 27 16:32:37.920 I ns=openshift-core-operators configmap=cluster-osin-operator2-lock 03dd509d-09f5-11e9-9b43-0a580a800027 became leader
Dec 27 16:32:37.921 W ns=openshift-controller-manager pod=controller-manager-58r2w node=ip-10-0-12-206.ec2.internal container=controller-manager container restarted
Dec 27 16:32:39.382 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal graceful deletion within 0s
Dec 27 16:32:39.384 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal graceful deletion within 0s
Dec 27 16:32:40.659 W ns=openshift-operator-lifecycle-manager pod=certified-operators-mbvrl node=ip-10-0-30-101.ec2.internal container=configmap-registry-server container stopped being ready
Dec 27 16:32:40.663 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal container=tuned container stopped being ready
Dec 27 16:32:41.550 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal container=tuned container stopped being ready
Dec 27 16:32:41.632 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal container=tuned container stopped being ready
Dec 27 16:32:41.645 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:32:42.108 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:32:42.124 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal container=tuned container stopped being ready
Dec 27 16:32:42.209 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal container=tuned container restarted
Dec 27 16:32:42.311 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal container=tuned container restarted
Dec 27 16:32:42.347 W ns=openshift-cluster-network-operator pod=cluster-network-operator-mtwr6 node=ip-10-0-33-43.ec2.internal container=cluster-network-operator container restarted
Dec 27 16:32:42.782 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-688fb8d6d9-pd5vt node=ip-10-0-12-206.ec2.internal container=operator container restarted
Dec 27 16:32:43.080 W ns=openshift-apiserver pod=apiserver-8htcf node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:32:43.260 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal container=tuned container restarted
Dec 27 16:32:43.383 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:32:43.440 W ns=openshift-kube-controller-manager pod=installer-2-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=installer container stopped being ready
Dec 27 16:32:44.609 W ns=openshift-console pod=openshift-console-5b88cf9649-j5zmj node=ip-10-0-12-206.ec2.internal container=console container restarted
Dec 27 16:32:44.804 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal container=tuned container restarted
Dec 27 16:32:44.936 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz node=ip-10-0-30-101.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:32:45.136 W ns=openshift-operator-lifecycle-manager pod=certified-operators-mbvrl node=ip-10-0-30-101.ec2.internal container=configmap-registry-server container restarted
Dec 27 16:32:45.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:32:46.085 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:32:46.491 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns-node-resolver container stopped being ready
Dec 27 16:32:46.750 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=kube-controller-manager container stopped being ready
Dec 27 16:32:47.143 W ns=openshift-cluster-openshift-apiserver-operator pod=openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:32:47.280 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:32:47.696 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=scheduler container restarted
Dec 27 16:32:48.206 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h node=ip-10-0-30-101.ec2.internal container=machine-config-operator container restarted
Dec 27 16:32:49.262 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:32:50.232 W ns=openshift-dns pod=dns-default-lptd4 node=ip-10-0-30-101.ec2.internal container=dns-node-resolver container restarted
Dec 27 16:32:56.802 W ns=openshift-image-registry pod=registry-ca-hostmapper-m6h8d node=ip-10-0-12-206.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:32:56.803 W ns=openshift-operator-lifecycle-manager pod=packageserver-657699878d-jz4nq node=ip-10-0-30-101.ec2.internal container=packageserver container restarted
Dec 27 16:32:57.413 W ns=openshift-operator-lifecycle-manager pod=rh-operators-pb7dg node=ip-10-0-12-206.ec2.internal container=configmap-registry-server container stopped being ready
Dec 27 16:32:57.612 W ns=openshift-core-operators pod=openshift-service-cert-signer-operator-78447d9bb8-szxl4 node=ip-10-0-12-206.ec2.internal container=operator container restarted
Dec 27 16:33:00.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:33:01.499 W ns=openshift-operator-lifecycle-manager pod=rh-operators-pb7dg node=ip-10-0-12-206.ec2.internal container=configmap-registry-server container restarted
Dec 27 16:33:01.549 W ns=openshift-image-registry pod=registry-ca-hostmapper-m6h8d node=ip-10-0-12-206.ec2.internal container=registry-ca-hostmapper container restarted
Dec 27 16:33:04.673 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:33:15.911 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:33:16.625 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=scheduler container restarted
Dec 27 16:33:19.708 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:33:20.626 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:33:21.125 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 node=ip-10-0-33-43.ec2.internal container=service-serving-cert-signer-controller container stopped being ready
Dec 27 16:33:21.350 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:33:21.571 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal invariant violation (bug): static pod should not transition Running->Pending with same UID
Dec 27 16:33:21.571 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=openshift-kube-apiserver container stopped being ready
Dec 27 16:33:22.085 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx node=ip-10-0-33-43.ec2.internal container=apiservice-cabundle-injector-controller container stopped being ready
Dec 27 16:33:22.370 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=installer container stopped being ready
Dec 27 16:33:23.060 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z node=ip-10-0-33-43.ec2.internal container=configmap-cabundle-injector-controller container stopped being ready
Dec 27 16:33:23.205 W ns=openshift-service-cert-signer pod=service-serving-cert-signer-c44bff5fc-q6zf7 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:33:23.385 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal graceful deletion within 0s
Dec 27 16:33:23.389 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:33:30.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:33:32.294 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal graceful deletion within 0s
Dec 27 16:33:32.331 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:33:33.109 W ns=openshift-service-cert-signer pod=configmap-cabundle-injector-7449dcf778-wzs9z node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:33:33.993 W ns=openshift-kube-scheduler pod=installer-2-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=installer container stopped being ready
Dec 27 16:33:34.052 W ns=openshift-kube-controller-manager pod=openshift-kube-controller-manager-ip-10-0-12-206.ec2.internal node=ip-10-0-12-206.ec2.internal container=kube-controller-manager container restarted
Dec 27 16:33:34.129 W ns=openshift-service-cert-signer pod=apiservice-cabundle-injector-778778fd75-f7pkx node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:33:35.718 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=scheduler container stopped being ready
Dec 27 16:33:37.835 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=scheduler container restarted
Dec 27 16:33:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:33:51.911 W ns=openshift-cluster-node-tuning-operator pod=tuned-68rsp node=ip-10-0-12-206.ec2.internal container=tuned container stopped being ready
Dec 27 16:33:51.921 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal container=tuned container stopped being ready
Dec 27 16:33:52.150 W ns=openshift-cluster-node-tuning-operator pod=tuned-t92zr node=ip-10-0-153-178.ec2.internal container=tuned container stopped being ready
Dec 27 16:33:52.859 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=scheduler container restarted
Dec 27 16:33:53.608 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container stopped being ready
Dec 27 16:33:53.609 W ns=openshift-cluster-node-tuning-operator pod=tuned-t92zr node=ip-10-0-153-178.ec2.internal container=tuned container restarted
Dec 27 16:33:53.895 W ns=openshift-cluster-node-tuning-operator pod=tuned-68rsp node=ip-10-0-12-206.ec2.internal container=tuned container restarted
Dec 27 16:34:00.913 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:34:02.641 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal graceful deletion within 0s
Dec 27 16:34:02.747 W ns=openshift-kube-apiserver pod=openshift-kube-apiserver-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:34:03.818 W ns=openshift-kube-apiserver pod=installer-5-ip-10-0-33-43.ec2.internal node=ip-10-0-33-43.ec2.internal container=installer container stopped being ready
Dec 27 16:34:04.659 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal container=tuned container restarted
Dec 27 16:34:15.914 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:34:23.828 W ns=openshift-controller-manager pod=controller-manager-58r2w node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:34:23.865 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container restarted
Dec 27 16:34:29.205 W ns=openshift-kube-scheduler pod=openshift-kube-scheduler-ip-10-0-30-101.ec2.internal node=ip-10-0-30-101.ec2.internal container=scheduler container restarted
Dec 27 16:34:29.606 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal container=console-operator container stopped being ready
Dec 27 16:34:29.626 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal container=console-operator container restarted
Dec 27 16:34:30.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:34:33.787 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal container=tuned container stopped being ready
Dec 27 16:34:34.639 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container stopped being ready
Dec 27 16:34:45.910 E openshift-apiserver OpenShift API is not responding to GET requests
Dec 27 16:34:48.622 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal container=tuned container restarted
Dec 27 16:34:49.858 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:34:51.210 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv node=ip-10-0-30-101.ec2.internal container=operator container stopped being ready
Dec 27 16:34:53.769 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal container=tuned container stopped being ready
Dec 27 16:34:55.193 W ns=openshift-controller-manager pod=controller-manager-58r2w node=ip-10-0-12-206.ec2.internal container=controller-manager container stopped being ready
# DEBUG: From this point on the apiserver seems to be responsive, but it's not clear why
Dec 27 16:34:56.940 I openshift-apiserver OpenShift API started responding to GET requests
Dec 27 16:34:58.739 W ns=openshift-cluster-openshift-controller-manager-operator pod=openshift-cluster-openshift-controller-manager-operator-5756gnv node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:35:01.272 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container restarted
Dec 27 16:35:02.701 W ns=openshift-controller-manager pod=controller-manager-58r2w node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:35:06.187 W ns=openshift-controller-manager pod=controller-manager-qbf49 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:35:08.880 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal container=tuned container restarted
Dec 27 16:35:09.272 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal container=tuned container stopped being ready
Dec 27 16:35:13.116 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:35:19.770 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:35:22.710 W ns=openshift-operator-lifecycle-manager pod=olm-operator-57c569f758-vfv2l node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:35:24.335 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal container=tuned container restarted
Dec 27 16:35:24.992 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:35:32.716 W ns=openshift-operator-lifecycle-manager pod=catalog-operator-7d5c85c7bf-mbpvk node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:35:37.453 W ns=openshift-controller-manager pod=controller-manager-qbf49 node=ip-10-0-30-101.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:35:45.385 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container restarted
Dec 27 16:35:46.254 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:35:47.525 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 node=ip-10-0-30-101.ec2.internal container=cluster-autoscaler-operator container stopped being ready
Dec 27 16:35:48.735 W ns=openshift-controller-manager pod=controller-manager-qbf49 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:35:51.571 W ns=openshift-controller-manager pod=controller-manager-2qgd4 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:35:58.603 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:35:58.747 W ns=openshift-cluster-api pod=cluster-autoscaler-operator-67b954cd94-wb442 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:35:58.862 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:35:59.602 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw node=ip-10-0-30-101.ec2.internal container=machine-api-operator container stopped being ready
Dec 27 16:36:01.757 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:36:02.624 W ns=openshift-cluster-machine-approver pod=machine-approver-59bc987855-j9lhg node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:36:02.625 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h node=ip-10-0-30-101.ec2.internal container=machine-config-operator container stopped being ready
Dec 27 16:36:03.187 W ns=openshift-cluster-machine-approver pod=machine-approver-59bc987855-j9lhg node=ip-10-0-33-43.ec2.internal container=machine-approver-controller container stopped being ready
Dec 27 16:36:07.495 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:36:07.718 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:36:08.770 W ns=openshift-machine-config-operator pod=machine-config-operator-5695ff5d47-xpq2h node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:36:08.819 W ns=openshift-cluster-api pod=machine-api-operator-5b474d94b7-99cmw node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:36:08.970 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal container=cluster-image-registry-operator container stopped being ready
Dec 27 16:36:12.940 W ns=openshift-cluster-machine-approver pod=machine-approver-59bc987855-j9lhg node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:36:18.018 W ns=openshift-image-registry pod=cluster-image-registry-operator-856b88cbdb-rvs5c node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:36:22.225 W ns=openshift-controller-manager pod=controller-manager-2qgd4 node=ip-10-0-33-43.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:36:23.118 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:36:23.259 W ns=openshift-controller-manager pod=controller-manager-2qgd4 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:36:24.539 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz node=ip-10-0-153-178.ec2.internal container=registry container stopped being ready
Dec 27 16:36:25.769 W ns=openshift-dns pod=dns-default-zv48s node=ip-10-0-12-206.ec2.internal container=dns container restarted
Dec 27 16:36:26.300 W ns=openshift-controller-manager pod=controller-manager-2n5fc node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:36:36.565 W ns=openshift-image-registry pod=image-registry-5cc7c4f4fc-rzsfz node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:36:38.583 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk node=ip-10-0-153-178.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:36:46.577 W ns=openshift-image-registry pod=registry-ca-hostmapper-lk6fk node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:36:49.709 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:36:56.853 W ns=openshift-controller-manager pod=controller-manager-2n5fc node=ip-10-0-12-206.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:37:02.778 W ns=openshift-controller-manager pod=controller-manager-2n5fc node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:37:20.099 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 node=ip-10-0-135-20.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:37:21.163 W ns=openshift-image-registry pod=registry-ca-hostmapper-fx5w4 node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:37:26.121 W ns=openshift-image-registry pod=registry-ca-hostmapper-m6h8d node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:37:57.192 W ns=openshift-image-registry pod=registry-ca-hostmapper-m6h8d node=ip-10-0-12-206.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:38:02.742 W ns=openshift-image-registry pod=registry-ca-hostmapper-m6h8d node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:38:08.216 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:38:19.861 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:38:21.344 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=controller-manager container stopped being ready
Dec 27 16:38:21.344 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-controller container stopped being ready
Dec 27 16:38:21.344 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=machine-healthcheck container stopped being ready
Dec 27 16:38:21.344 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal container=nodelink-controller container stopped being ready
Dec 27 16:38:23.356 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:38:24.802 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 node=ip-10-0-33-43.ec2.internal container=machine-config-controller container stopped being ready
Dec 27 16:38:25.352 W ns=openshift-machine-config-operator pod=machine-config-server-l85ww node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:38:25.858 W ns=openshift-machine-config-operator pod=machine-config-controller-55fcd8945d-xb4h6 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:38:26.219 W ns=openshift-machine-config-operator pod=machine-config-server-l85ww node=ip-10-0-30-101.ec2.internal container=machine-config-server container stopped being ready
Dec 27 16:38:27.192 W ns=openshift-machine-config-operator pod=machine-config-server-l85ww node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:38:30.210 W ns=openshift-machine-config-operator pod=machine-config-server-jqdz9 node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:38:30.740 W ns=openshift-machine-config-operator pod=machine-config-server-jqdz9 node=ip-10-0-33-43.ec2.internal container=machine-config-server container stopped being ready
Dec 27 16:38:31.762 W ns=openshift-machine-config-operator pod=machine-config-server-jqdz9 node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:38:32.728 W ns=openshift-cluster-api pod=clusterapi-manager-controllers-7dd649d6b-5vwcq node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:38:34.917 W ns=openshift-machine-config-operator pod=machine-config-server-lwl26 node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:38:36.423 W ns=openshift-machine-config-operator pod=machine-config-server-lwl26 node=ip-10-0-12-206.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:38:36.423 W ns=openshift-machine-config-operator pod=machine-config-server-lwl26 node=ip-10-0-12-206.ec2.internal container=machine-config-server container stopped being ready
Dec 27 16:38:38.031 W ns=openshift-machine-config-operator pod=machine-config-server-lwl26 node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:38:39.056 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n node=ip-10-0-33-43.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:38:39.888 W ns=openshift-image-registry pod=registry-ca-hostmapper-ngw6n node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:38:43.932 W ns=openshift-machine-config-operator pod=machine-config-daemon-8wl4c node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:38:44.844 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:38:44.874 W ns=openshift-machine-config-operator pod=machine-config-daemon-8wl4c node=ip-10-0-153-178.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:38:56.739 W ns=openshift-machine-config-operator pod=machine-config-daemon-8wl4c node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:38:59.928 W ns=openshift-machine-config-operator pod=machine-config-daemon-s8l47 node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:39:01.370 W ns=openshift-machine-config-operator pod=machine-config-daemon-s8l47 node=ip-10-0-30-101.ec2.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
Dec 27 16:39:01.370 W ns=openshift-machine-config-operator pod=machine-config-daemon-s8l47 node=ip-10-0-30-101.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:39:08.734 W ns=openshift-machine-config-operator pod=machine-config-daemon-s8l47 node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:39:12.422 W ns=openshift-machine-config-operator pod=machine-config-daemon-nhp9t node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:39:12.928 W ns=openshift-machine-config-operator pod=machine-config-daemon-nhp9t node=ip-10-0-33-43.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:39:15.477 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 node=ip-10-0-161-51.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:39:16.474 W ns=openshift-image-registry pod=registry-ca-hostmapper-k6nh6 node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:39:21.500 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:39:22.943 W ns=openshift-machine-config-operator pod=machine-config-daemon-nhp9t node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:39:26.090 W ns=openshift-machine-config-operator pod=machine-config-daemon-wxfbj node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:39:26.640 W ns=openshift-machine-config-operator pod=machine-config-daemon-wxfbj node=ip-10-0-12-206.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:39:32.706 W ns=openshift-machine-config-operator pod=machine-config-daemon-wxfbj node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:39:35.635 W ns=openshift-machine-config-operator pod=machine-config-daemon-pw4m5 node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:39:36.517 W ns=openshift-machine-config-operator pod=machine-config-daemon-pw4m5 node=ip-10-0-161-51.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:39:48.000 W ns=openshift-machine-config-operator pod=machine-config-daemon-pw4m5 node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:39:50.558 W ns=openshift-machine-config-operator pod=machine-config-daemon-k4w55 node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:39:51.438 W ns=openshift-machine-config-operator pod=machine-config-daemon-k4w55 node=ip-10-0-135-20.ec2.internal container=machine-config-daemon container stopped being ready
Dec 27 16:39:52.536 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz node=ip-10-0-30-101.ec2.internal container=registry-ca-hostmapper container stopped being ready
Dec 27 16:39:53.555 W ns=openshift-image-registry pod=registry-ca-hostmapper-qb8wz node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:40:02.834 W ns=openshift-machine-config-operator pod=machine-config-daemon-k4w55 node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:40:05.679 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:40:05.888 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:40:07.143 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj node=ip-10-0-153-178.ec2.internal container=ingress-operator container stopped being ready
Dec 27 16:40:07.203 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd node=ip-10-0-153-178.ec2.internal container=cluster-monitoring-operator container stopped being ready
# BUG: The tuning operator seems to be continually updating (this may be the existing tuned issue or a new one)
Dec 27 16:40:12.219 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:40:13.138 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:40:13.887 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq node=ip-10-0-135-20.ec2.internal container=cluster-node-tuning-operator container stopped being ready
Dec 27 16:40:16.758 W ns=openshift-monitoring pod=cluster-monitoring-operator-56db7f6d97-wd7bd node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:40:16.956 W ns=openshift-ingress-operator pod=ingress-operator-694bd9bf8d-52njj node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:40:20.883 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:40:22.376 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal container=cluster-samples-operator container stopped being ready
Dec 27 16:40:22.953 W ns=openshift-cluster-node-tuning-operator pod=cluster-node-tuning-operator-76c84fc5f5-s6zdq node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:40:23.426 W ns=openshift-cluster-samples-operator pod=cluster-samples-operator-784f74d486-xg9gj node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:40:33.418 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf node=ip-10-0-135-20.ec2.internal graceful deletion within 30s
Dec 27 16:40:33.476 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal graceful deletion within 70s
Dec 27 16:40:34.307 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal container=openshift-apiserver container stopped being ready
Dec 27 16:40:34.671 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf node=ip-10-0-135-20.ec2.internal container=cluster-storage-operator container stopped being ready
Dec 27 16:40:35.248 W ns=openshift-apiserver pod=apiserver-wsjnb node=ip-10-0-33-43.ec2.internal deleted
Dec 27 16:40:35.676 W ns=openshift-cluster-storage-operator pod=cluster-storage-operator-5d8d77976d-bq6kf node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:40:43.681 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal container=tuned container stopped being ready
Dec 27 16:40:50.754 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:40:51.815 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal container=console-operator container stopped being ready
Dec 27 16:40:52.840 W ns=openshift-cluster-node-tuning-operator pod=tuned-twr6k node=ip-10-0-135-20.ec2.internal deleted
Dec 27 16:40:55.737 W ns=openshift-cluster-node-tuning-operator pod=tuned-68rsp node=ip-10-0-12-206.ec2.internal graceful deletion within 30s
Dec 27 16:40:56.531 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:40:57.499 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj node=ip-10-0-153-178.ec2.internal container=csi-operator container stopped being ready
Dec 27 16:40:58.002 W ns=openshift-console pod=console-operator-5d979b9854-fct6l node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:41:06.563 W ns=openshift-csi-operator pod=csi-operator-6d644cdb5f-nlwgj node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:41:26.971 W ns=openshift-cluster-node-tuning-operator pod=tuned-68rsp node=ip-10-0-12-206.ec2.internal container=tuned container stopped being ready
Dec 27 16:41:32.696 W ns=openshift-cluster-node-tuning-operator pod=tuned-68rsp node=ip-10-0-12-206.ec2.internal deleted
Dec 27 16:41:36.010 W ns=openshift-cluster-node-tuning-operator pod=tuned-t92zr node=ip-10-0-153-178.ec2.internal graceful deletion within 30s
Dec 27 16:41:50.864 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:42:06.653 W ns=openshift-cluster-node-tuning-operator pod=tuned-t92zr node=ip-10-0-153-178.ec2.internal container=tuned container stopped being ready
Dec 27 16:42:16.567 W ns=openshift-cluster-node-tuning-operator pod=tuned-t92zr node=ip-10-0-153-178.ec2.internal deleted
Dec 27 16:42:18.690 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal graceful deletion within 30s
Dec 27 16:42:28.910 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:42:49.109 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal container=tuned container stopped being ready
Dec 27 16:42:50.126 W ns=openshift-cluster-node-tuning-operator pod=tuned-cxjc9 node=ip-10-0-161-51.ec2.internal deleted
Dec 27 16:42:53.130 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal graceful deletion within 30s
Dec 27 16:43:24.223 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal container=tuned container stopped being ready
Dec 27 16:43:25.294 W ns=openshift-cluster-node-tuning-operator pod=tuned-b8tmq node=ip-10-0-30-101.ec2.internal deleted
Dec 27 16:43:27.398 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal graceful deletion within 30s
Dec 27 16:43:58.803 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal container=tuned container stopped being ready
Dec 27 16:44:02.954 W ns=openshift-cluster-node-tuning-operator pod=tuned-p2ggp node=ip-10-0-33-43.ec2.internal deleted
# DEBUG: Not good
Dec 27 16:48:17.590 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:48:58.678 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:54:44.368 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 16:55:19.441 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:01:06.141 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:01:39.213 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:07:19.920 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:07:58.953 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:24:01.737 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:24:01.760 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:24:01.925 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:24:01.925 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:26:38.184 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:27:19.266 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:33:00.997 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
Dec 27 17:33:39.089 W ns=openshift-ingress pod=router-default-b79dc5894-t77g4 node=ip-10-0-135-20.ec2.internal container=router container restarted
# Cluster is torn down
Dec 27 17:36:28.545 I openshift-apiserver OpenShift API started failing: Get https://ci-ln-y7gvwf2-703b0-api.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=3s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Dec 27 17:36:28.584 I kube-apiserver received an error while watching events: Internal error occurred: rpc error: code = Unknown desc =
Dec 27 17:36:31.539 E kube-apiserver Kube API started failing: Get https://ci-ln-y7gvwf2-703b0-api.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=3s: dial tcp 35.153.150.156:6443: i/o timeout
Dec 27 17:36:42.538 E kube-apiserver Kube API is not responding to GET requests
That looks like cri-o/cri-o#1927. Sounds like they're working on a fix now.