self-hosted-proposal.md

Proposal: Self-hosted kubernetes components

Abstract

A Kubernetes deployment has many of the same considerations as a normal application deployment using Kubernetes. It should therefore be possible to deploy Kubernetes with kubernetes. This will make cluster installation and updating vastly simpler. From this point forward, Kubernetes installed by Kubernetes will be referred to as "self-hosted", or "managed".

Background

Our current approach to self-hosting Kubernetes is a "pivot" style installation procedure. Briefly, the following steps summarize a generic outline for this installation procedure (it is assumed that a functional etcd cluster exists somewhere):

Control plane

1. Start a "bootstrap" control plane somewhere (does not have to be local to installation machine -- could be "monokube" style, e.g. control plane in single binary).
2. Let $init_system start a Kubelet with --runonce, --lock-file, --api-servers, and a taint like component=kubelet. This is known as the "bootstrap" Kubelet. This kubelet will use standard Kubernetes authentication methods to authenticate with the bootstrap control plane.
3. Send manifests for Kubernetes components to apiserver (Deployments, Daemonsets, ConfigMaps, Secrets). Within these manifests will be a Kubelet that has a toleration component=kubelet.
4. $init_system runs "bootstrap" Kubelet until "managed (self-hosted)" Kubelet starts up.
5. "managed" Kubelet pulls down the rest of master components
6. "bootstrap" apiserver goes away, new "managed" apiserver takes over

Worker

1. "bootstrap" Kubelet run by $init_system starts with --runonce, --lock-file, --api-servers and a similar taint as above. Similarly, this Kubelet will have any necesarry assets it needs to securely auth against the apiserver.
2. "bootstrap" Kubelet pulls down "managed" Kubelet and exits.
3. "managed" Kubelet pulls down rest of worker pods (kube-proxy, etc).
4. "managed" Kubelet is now ready to begin functioning in the cluster.

Proposal

Aside from the Taints and Tolerations, the only other change needed in Kubernetes to support this is the ability for a Kubelet in runonce mode to contact an apiserver. This work has already started (Issue: kubernetes/kubernetes#23073, PR: kubernetes/kubernetes#23074).

This will allow a "self-hosted" cluster with minimal new concepts introduced into the core Kubernetes code base, and remains flexible enough to work well with future bootstrapping services.

Other discussion

Various similar approaches have been discussed here and here. This also relies on the Kubelet being able to be run inside a container, more discussion on that is here.