Skip to content

Instantly share code, notes, and snippets.

@danehans
Created October 14, 2014 01:27
Show Gist options
  • Save danehans/21fc9ab67d5cd4e4fa41 to your computer and use it in GitHub Desktop.
Save danehans/21fc9ab67d5cd4e4fa41 to your computer and use it in GitHub Desktop.
heat kube master issue
kube master does not work after instance is reboot or suspended/resumed.
#interfaces and route are missing
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::f816:3eff:fe83:439b prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:83:43:9b txqueuelen 1000 (Ethernet)
RX packets 101805 bytes 120493424 (114.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 61992 bytes 6210195 (5.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 39970 bytes 2936113 (2.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 39970 bytes 2936113 (2.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# ls /etc/sysconfig/network-scripts/
ifcfg-eth0 ifdown-ippp ifdown-ppp ifup-aliases ifup-ipx ifup-post ifup-wireless
ifcfg-lo ifdown-ipv6 ifdown-routes ifup-bnep ifup-isdn ifup-ppp init.ipv6-global
ifdown ifdown-isdn ifdown-sit ifup-eth ifup-ovs ifup-routes network-functions
ifdown-bnep ifdown-ovs ifdown-tunnel ifup-ippp ifup-plip ifup-sit network-functions-ipv6
ifdown-eth ifdown-post ifup ifup-ipv6 ifup-plusb ifup-tunnel
# linkmanager is missing
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# ls -al /opt
total 8
drwxr-xr-x. 2 root root 4096 Aug 7 2013 .
dr-xr-xr-x. 19 root root 4096 Oct 13 23:59 ..
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status linkmanager.service
linkmanager.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
#openvswitch
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# rpm -qa | grep openvswitch
openvswitch-2.3.0-1.fc20.x86_64
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status openvswitch
openvswitch.service - Open vSwitch
Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled)
Active: inactive (dead)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl enable openvswitch
ln -s '/usr/lib/systemd/system/openvswitch.service' '/etc/systemd/system/multi-user.target.wants/openvswitch.service'
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status openvswitch
openvswitch.service - Open vSwitch
Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled)
Active: inactive (dead)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl start openvswitch
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status openvswitch
openvswitch.service - Open vSwitch
Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled)
Active: active (exited) since Tue 2014-10-14 00:57:43 UTC; 1s ago
Process: 15654 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 15654 (code=exited, status=0/SUCCESS)
#missing and/or blank files
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# cat /etc/environment
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# cat /etc/sysconfig/linkmanager
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status docker.socket
docker.socket - Docker Socket for the API
Loaded: loaded (/usr/lib/systemd/system/docker.socket; disabled)
Active: inactive (dead)
Listen: /var/run/docker.sock (Stream)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; static)
Active: inactive (dead)
Docs: http://docs.docker.com
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status kubelet
kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled)
Active: inactive (dead)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status kube-proxy
kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; disabled)
Active: inactive (dead)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]# systemctl status kube-
kube-.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
[root@kube-cluster-kube-master-wbzqtqmtpaxd ~]#
### Note: everything looks good on minions after reboot or suspend/resume.
@larsks
Copy link

larsks commented Oct 14, 2014

Docker and linkmanager (and hence openvswitch) never run on the master. Similarly, /etc/environment is not configured on the master.

These are only installed on minions. Take a look at the user-data scripts in the templates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment