Skip to content

Instantly share code, notes, and snippets.

@imranrazakhan

imranrazakhan/multimaster.logs

Created October 25, 2019 15:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save imranrazakhan/e006282baf0200d0f384a1bc4510796a to your computer and use it in GitHub Desktop.
Save imranrazakhan/e006282baf0200d0f384a1bc4510796a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
multimaster.logs
Oct 25 16:08:35 xdww002.my.local origin-node[18869]: E1025 16:08:35.035241 18869 event.go:212] Unable to write event: 'Patch https://xdww001.my.local:8443/api/v1/namespaces/kube-system/events/master-etcd-xdww002.my.local.15d0e894c53b9128: dial tcp 10.11.0.178:8443: i/o timeout' (may retry after sleeping)
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205055 I | raft: 1f1b1f1969afb782 is starting a new election at term 366
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205078 I | raft: 1f1b1f1969afb782 became candidate at term 367
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205090 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 367
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205103 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 367
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: E1025 14:08:35.425646 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.HostSubnet: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/hostsubnets?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.458163068Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: I1025 14:08:35.558185 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:38793->10.11.0.194:53: i/o timeout"
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.747473516Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:288: Failed to list *v1.Service: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/services?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.985546858Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.004009 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.004803 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.005821 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.006926 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.007911 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.009100 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.010082 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.011169 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.012403 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.013423 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.104128 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.139627501Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.13964985Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.176417352Z caller=notifier.go:473 component=notifier alertmanager=https://10.128.0.25:9094/api/v1/alerts count=1 msg="Error sending alert" err="Post https://10.128.0.25:9094/api/v1/alerts: dial tcp 10.128.0.25:9094: connect: no route to host"
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: I1025 14:08:36.324719 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34328->10.11.0.194:53: i/o timeout"
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.351791 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-controller-manager: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.986033671Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.004835 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.005574 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.006647 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.007722 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.008742 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.009798 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.010896 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.011925 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.013097 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.014117 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local origin-node[18869]: W1025 16:08:37.089450 18869 status_manager.go:482] Failed to get status for pod "master-etcd-xdww002.my.local_kube-system(691e4b894fec0cd2e377c5bdab4d51f5)": Get https://xdww001.my.local:8443/api/v1/namespaces/kube-system/pods/master-etcd-xdww002.my.local: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:37.352568812Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: I1025 14:08:37.558669 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:58699->10.11.0.194:53: i/o timeout"
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.571103 1 webhook.go:90] Failed to make webhook authenticator request: Post https://xdww002.my.local:8443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.571123 1 authentication.go:62] Unable to authenticate the request due to an error: Post https://xdww002.my.local:8443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:37,648 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.SocketTimeoutException: connect timed out
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: time="2019-10-25T14:08:37.849046896Z" level=info msg=response go.version=go1.10.3 http.request.host="10.129.0.18:5000" http.request.id=7f29018c-ff06-4a69-9dcf-59a9247660b9 http.request.method=GET http.request.remoteaddr="10.129.0.1:35030" http.request.uri=/healthz http.request.useragent=kube-probe/1.11+ http.response.duration="38.82µs" http.response.status=200 http.response.written=0 instance.id=f5871a25-5a0c-4fca-9b4f-734a14005449
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:37.986508853Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.005545 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.006407 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.007480 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.008636 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.009644 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.010680 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.011810 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.012882 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.014047 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.015031 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205140 I | raft: 1f1b1f1969afb782 is starting a new election at term 367
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205165 I | raft: 1f1b1f1969afb782 became candidate at term 368
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205179 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 368
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205192 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 368
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.224713 W | rafthttp: health check for peer bbda256326db0d46 could not connect: dial tcp 10.11.0.178:2380: i/o timeout
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: I1025 14:08:38.324952 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:33658->10.11.0.194:53: i/o timeout"
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:38.365215677Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host"
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:38,648 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused)
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:38.717964893Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.006231 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.007052 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.008223 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.009254 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.010298 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.011439 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.012502 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.013623 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.014584 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.015645 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.238993 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:39.365789942Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: I1025 14:08:39.558815 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:42478->10.11.0.194:53: i/o timeout"
Oct 25 16:08:39 xdww002.my.local origin-node[18869]: E1025 16:08:39.569016 18869 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://xdww001.my.local:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dxdww002.my.local&limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.569079 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.NetNamespace: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/netnamespaces?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.569970 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.EgressNetworkPolicy: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/egressnetworkpolicies?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:39,649 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused)
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:39.718550669Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.835269 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-controller-manager: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.865195 1 leaderelection.go:234] error retrieving resource lock kube-system/openshift-master-controllers: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/openshift-master-controllers: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.950580 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: Get https://172.30.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.007027 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.007843 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.008872 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.009919 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.011037 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.012016 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.013056 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.014070 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.015245 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.016227 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: I1025 14:08:40.325269 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:60711->10.11.0.194:53: i/o timeout"
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:40.366313792Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: 2019/10/25 14:08:40 Failed to dial backend: 'dial tcp 172.30.0.1:443: connect: connection timed out'
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 14:08:40.597674 19700 reflector.go:136] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: Failed to list *core.Service: Get https://xdww001.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:40,649 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused)
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:40.719101285Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: E1025 16:08:40.878633 18869 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://xdww001.my.local:8443/api/v1/nodes?fieldSelector=metadata.name%3Dxdww002.my.local&limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 14:08:40.878615 19700 reflector.go:136] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: Failed to list *core.Endpoints: Get https://xdww001.my.local:8443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943821 18869 kuberuntime_manager.go:513] Container {Name:api Image:docker.io/openshift/origin-control-plane:v3.11.0 Command:[/bin/bash -c] Args:[#!/bin/bash
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: set -euo pipefail
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: if [[ -f /etc/origin/master/master.env ]]; then
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: set -o allexport
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: source /etc/origin/master/master.env
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: fi
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: exec openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=${DEBUG_LOGLEVEL:-2}
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: ] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:master-config ReadOnly:false MountPath:/etc/origin/master/ SubPath: MountPropagation:<nil>} {Name:master-cloud-provider ReadOnly:false MountPath:/etc/origin/cloudprovider/ SubPath: MountPropagation:<nil>} {Name:master-data ReadOnly:false MountPath:/var/lib/origin/ SubPath: MountPropagation:<nil>} {Name:master-pki ReadOnly:false MountPath:/etc/pki SubPath: MountPropagation:<nil>} {Name:host-localtime ReadOnly:false MountPath:/etc/localtime SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:45,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz/ready,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943899 18869 kuberuntime_manager.go:757] checking backoff for container "api" in pod "master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)"
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943984 18869 kuberuntime_manager.go:767] Back-off 2m40s restarting failed container=api pod=master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: E1025 16:08:40.944003 18869 pod_workers.go:186] Error syncing pod b24b15710309f0062b93e07af49cb464 ("master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)"), skipping: failed to "StartContainer" for "api" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=api pod=master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)"
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.007756 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.008604 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.009667 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.010654 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.011740 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.012868 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.013995 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.014975 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.016175 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.017220 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.366792497Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: I1025 14:08:41.559195 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34154->10.11.0.194:53: i/o timeout"
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.665228351Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:41,874 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.NoRouteToHostException: No route to host (Host unreachable)
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.875147621Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host"
Oct 25 16:08:41 xdww002.my.local origin-node[18869]: E1025 16:08:41.875306 18869 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "xdww002.my.local": Get https://xdww001.my.local:8443/api/v1/nodes/xdww002.my.local?timeout=10s: dial tcp 10.11.0.178:8443: connect: no route to host
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.008530 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.009365 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.010449 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.011434 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.012496 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.013530 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.014754 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.015806 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.016832 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.018003 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 14:08:42.238352 1 summary.go:97] error while getting metrics summary from Kubelet xdww001.my.local(10.11.0.178:10250): Get https://10.11.0.178:10250/stats/summary/: dial tcp 10.11.0.178:10250: connect: connection timed out
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: W1025 14:08:42.238378 1 manager.go:102] Failed to get kubelet_summary:10.11.0.178:10250 response in time
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: I1025 14:08:42.325626 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:45351->10.11.0.194:53: i/o timeout"
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.367292157Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: I1025 16:08:42.594789 18869 prober.go:111] Liveness probe for "master-etcd-xdww002.my.local_kube-system(691e4b894fec0cd2e377c5bdab4d51f5):etcd" failed (failure): cluster may be unhealthy: failed to list members
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: Error: client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint https://10.11.0.194:2379 exceeded header timeout
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: error #0: client: endpoint https://10.11.0.194:2379 exceeded header timeout
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.665730263Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705159 I | raft: 1f1b1f1969afb782 is starting a new election at term 368
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705182 I | raft: 1f1b1f1969afb782 became candidate at term 369
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705193 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 369
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705202 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 369
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.778224 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.87553612Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 14:08:42.978434 1 reflector.go:136] github.com/openshift/origin/pkg/router/template/service_lookup.go:32: Failed to list *core.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.97841345Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.009224 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.010078 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.011192 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.012278 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.013322 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.014356 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.015501 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.016496 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.017654 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.018693 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.096446202Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.170857715Z caller=notifier.go:473 component=notifier alertmanager=https://10.129.0.23:9094/api/v1/alerts count=1 msg="Error sending alert" err="context deadline exceeded"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019/10/25 14:08:43 oauthproxy.go:772: 10.129.0.24:35282 Post https://172.30.0.1:443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 172.30.0.1:443: getsockopt: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.176656197Z caller=notifier.go:473 component=notifier alertmanager=https://10.129.0.23:9094/api/v1/alerts count=2 msg="Error sending alert" err="bad response status 403 Forbidden"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:43.219866 E | etcdserver: publish error: etcdserver: request timed out
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:43.224927 W | rafthttp: health check for peer bbda256326db0d46 could not connect: dial tcp 10.11.0.178:2380: i/o timeout
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.367810928Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: I1025 14:08:43.559523 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34519->10.11.0.194:53: i/o timeout"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 14:08:43.907088 1 reflector.go:205] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:348: Failed to list *v1beta2.StatefulSet: Get https://172.30.0.1:443/apis/apps/v1beta2/namespaces/openshift-monitoring/statefulsets?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.978835778Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 14:08:43.978879 1 reflector.go:136] github.com/openshift/origin/pkg/router/template/service_lookup.go:32: Failed to list *core.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.995978024Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.010103 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.010890 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.011871 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.012913 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.013921 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.015041 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.016104 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.017226 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.018269 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.019366 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.057971495Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout"
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.097047797Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: I1025 14:08:44.325923 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:36292->10.11.0.194:53: i/o timeout"
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.368412397Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment