kolla-kube-mstr-error

gistfile1.txt

Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.535112 00791 iowatcher.go:88] Unable to decode an event from the watch stream: got invalid watch event type: ERROR
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537346 00791 iowatcher.go:88] Unable to decode an event from the watch stream: got invalid watch event type: ERROR
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537475 00791 reflector.go:166] unexpected watch close - watch lasted less than a second and no items received
Oct 13 21:12:27 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-scheduler[791]: E1013 21:12:27.537545 00791 reflector.go:105] watch of *api.Pod ended with error: very short watch

APISERVER STAUS

[root@kube-cluster-kube-master-feulivz7l4jn kolla]# systemctl status kube-apiserver
kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)
   Active: active (running) since Mon 2014-10-13 05:21:00 UTC; 16h ago
 Main PID: 781 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─781 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=0.0.0...

Oct 13 17:49:58 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:49:58.506785 00781 e...)
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 e...]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 p...]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 e...)
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 e...]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 e...)
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 e...]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 e...)
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 e...]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 p...]
Hint: Some lines were ellipsized, use -l to show in full.

APISERVER LOG (Note: No activity since I got vxlan networking squared away)

[root@kube-cluster-kube-master-feulivz7l4jn kolla]# journalctl -f -l -xn -u kube-apiserver
-- Logs begin at Wed 2014-10-08 09:18:51 UTC. --
Oct 13 17:49:58 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:49:58.506785 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
^C
[root@kube-cluster-kube-master-feulivz7l4jn kolla]# date
Mon Oct 13 21:37:35 UTC 2014
[root@kube-cluster-kube-master-feulivz7l4jn kolla]# 
[root@kube-cluster-kube-master-feulivz7l4jn k8s]# systemctl status kube-apiserver -l
kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)
   Active: active (running) since Mon 2014-10-13 05:21:00 UTC; 16h ago
 Main PID: 781 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─781 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=0.0.0.0 --port=8080 --machines=10.0.0.6,10.0.0.5,10.0.0.4 --minion_port=10250 --allow_privileged=true

Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.074065 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:01 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:01.615936 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:08 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:08.509898 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:11 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:11.076093 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:18 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:18.513587 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:21 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:21.078698 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:28 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:28.518613 00781 etcd_tools_watch.go:152] watch was unable to retrieve the current index for the provided key: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] ("/registry/controllers")
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.080268 00781 endpoints_controller.go:53] Failed to list services: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 17:50:31 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 17:50:31.616637 00781 pod_cache.go:78] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Oct 13 21:51:15 kube-cluster-kube-master-feulivz7l4jn.dmz-pod2.lab kube-apiserver[781]: E1013 21:51:15.268416 00781 endpoints_controller.go:73] Failed to find an IP for pod: {{ mariadb 2014-10-13 21:51:11 +0000 UTC  4439  default} map[name:mariadb] {{v1beta1 mariadb 0bee4460-5323-11e4-8f46-fa163ec3ddbf [] [{mariadb kollaglue/fedora-rdo-mariadb []  [{ 0 3306 TCP }] [{DB_ROOT_PASSWORD password}] 0 0 [] <nil> <nil> false }] {0xcabef0 <nil> <nil>}} Running 10.0.0.6   map[]} {{   [] [] {<nil> <nil> <nil>}} Waiting 10.0.0.6   map[]}}