Skip to content

Instantly share code, notes, and snippets.

@danehans
Last active August 29, 2015 14:07
Show Gist options
  • Save danehans/248b91e3cbe71ef163eb to your computer and use it in GitHub Desktop.
Save danehans/248b91e3cbe71ef163eb to your computer and use it in GitHub Desktop.
kolla-kube-mstr-error
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.535112 00791 iowatcher.go:88] Unable to decode an event from the watch stream: got invalid watch event type: ERROR
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537346 00791 iowatcher.go:88] Unable to decode an event from the watch stream: got invalid watch event type: ERROR
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537475 00791 reflector.go:166] unexpected watch close - watch lasted less than a second and no items received
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537545 00791 reflector.go:105] watch of *api.Pod ended with error: very short watch
APISERVER STAUS
[root@kube-cluster-kube-master-feulivz7l4jn kolla]# systemctl status kube-apiserver
kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)
Active: active (running) since Mon 2014-10-13 05:21:00 UTC; 16h ago
Main PID: 781 (kube-apiserver)
CGroup: /system.slice/kube-apiserver.service
└─781 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=0.0.0...
Oct 13 17:49:58 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:49:58.506785 00781 e...)
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 e...]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 p...]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 e...)
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 e...]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 e...)
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 e...]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 e...)
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 e...]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 p...]
Hint: Some lines were ellipsized, use -l to show in full.
APISERVER LOG (Note: No activity since I got vxlan networking squared away)
[root@kube-cluster-kube-master-feulivz7l4jn kolla]# journalctl -f -l -xn -u kube-apiserver
-- Logs begin at Wed 2014-10-08 09:18:51 UTC. --
Oct 13 17:49:58 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:49:58.506785 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
^C
[root@kube-cluster-kube-master-feulivz7l4jn kolla]# date
Mon Oct 13 21:37:35 UTC 2014
[root@kube-cluster-kube-master-feulivz7l4jn kolla]#
[root@kube-cluster-kube-master-feulivz7l4jn k8s]# systemctl status kube-apiserver -l
kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)
Active: active (running) since Mon 2014-10-13 05:21:00 UTC; 16h ago
Main PID: 781 (kube-apiserver)
CGroup: /system.slice/kube-apiserver.service
└─781 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=0.0.0.0 --port=8080 --machines=10.0.0.6,10.0.0.5,10.0.0.4 --minion_port=10250 --allow_privileged=true
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 21:51:15 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 21:51:15.268416 00781 endpoints_controller.go:73] Failed to find an IP for pod: {{ mariadb 2014-10-13 21:51:11 +0000 UTC 4439 default} map[name:mariadb] {{v1beta1 mariadb 0bee4460-5323-11e4-8f46-fa163ec3ddbf [] [{mariadb kollaglue/fedora-rdo-mariadb [] [{ 0 3306 TCP }] [{DB_ROOT_PASSWORD password}] 0 0 [] <nil> <nil> false }] {0xcabef0 <nil> <nil>}} Running 10.0.0.6 map[]} {{ [] [] {<nil> <nil> <nil>}} Waiting 10.0.0.6 map[]}}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment