kube-apiserver restart loop

README.md

kube-apiserver restart loop

If the kube-apiserver is in a restart loop, it is possible that one of the etcd servers it is trying to connect to is no longer reachable. It should be able to just move on to the next etcd server but in some rare cases it does not. In those situations you need to remove the bad etcd servers from its startup options as shown below.

1. Get runlike command for kube-apiserverwith the following command:

   docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver
   

   Example output:

      [root@ip-172-31-14-184 ~]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver
      Unable to find image 'axeal/runlike:latest' locally
      latest: Pulling from axeal/runlike
      bdf0201b3a05: Pull complete
      38f114998adb: Pull complete
      21134b1a9e68: Pull complete
      1826484332dc: Pull complete
      a124bc45399b: Pull complete
      e1147593372b: Pull complete
      2bd42cd7a205: Pull complete
      92dd7b6abd56: Pull complete
      ff342074ab25: Pull complete
      c96b6f85e13d: Pull complete
      c6f85279bce2: Pull complete
      69240e7fa944: Pull complete
      c2cab7570ce0: Pull complete
      Digest: sha256:8039674ccacc1ddb851723e49919ff36896d078b35067c8e80462064aacd0068
      Status: Downloaded newer image for axeal/runlike:latest
      docker run --name=kube-apiserver --hostname=ip-172-31-14-184.us-east-2.compute.internal --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --volume=/etc/kubernetes:/etc/kubernetes:z --volumes-from=service-sidekick --network=host --restart=always --label org.label-schema.schema-version="1.0" --label io.rancher.rke.container.name="kube-apiserver" --label org.label-schema.vcs-ref="c5da50e996b51071c14ffe3a83bd0f4ab469b3e0" --label org.label-schema.build-date="2019-06-12T03:01:55Z" --label org.label-schema.vcs-url="https://github.com/rancher/hyperkube.git" --entrypoint /opt/rke-tools/entrypoint.sh --detach=true rancher/hyperkube:v1.14.3-rancher1 kube-apiserver --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-lookup=true --storage-backend=etcd3 --etcd-prefix=/registry --etcd-servers=https://172.31.14.184:2379,https://172.31.4.50:2379,https://172.31.5.114:2379 --bind-address=0.0.0.0 --insecure-port=0 --profiling=false --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-node-port-range=30000-32767 --authorization-mode=Node,RBAC --anonymous-auth=false --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-cluster-ip-range=10.43.0.0/16 --allow-privileged=true --requestheader-group-headers=X-Remote-Group --cloud-provider= --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --secure-port=6443 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.14.184 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra-
   

2. Find the option that lists your etcd servers

   --etcd-servers=https://172.31.14.184:2379,https://172.31.4.50:2379,https://172.31.5.114:2379 
   

3. Remove the dead etcd server(s), delete your old kube-apiserver container then create a new one with the modified runlike command.