Install kubernetes on a centos or fedora atomic host using vagrant and ansible. I got this working and blogged about it: http://www.projectatomic.io/blog/2015/09/clustering-atomic-hosts-with-kubernetes-ansible-and-vagrant/. I'm keeping some notes about upstreaming my changes here.

kubernetes-ansible-atomic.md

The kubernetes ansible/vagrant install method can be used with the libvirt, virtualbox or openstack providers. The Vagrantfile requires that vagrant-openstack-provider be installed whether you're using it or not, so install it:

$ vagrant plugin install vagrant-openstack-provider

Alternatively, you can skip the second step above, but you'll need to comment out the require 'vagrant-openstack-provider' line in the Vagrantfile.

$ git clone https://github.com/jasonbrooks/contrib.git
$ cd contrib/ansible/vagrant

The Vagrantfile in my fork of the kuberenetes contrib repo adds support for setting a distro_type to indicate whether you want to use CentOS 7 (default), Fedora 22 (export DISTRO_TYPE=fedora), Fedora Atomic (export DISTRO_TYPE=fedora-atomic), or CentOS Atomic (export DISTRO_TYPE=centos-atomic). This should work with both the libvirt and virtualbox providers.

NOTE: Plain CentOS 7 and Fedora 22 aren't working for me right now w/ the script, but the atomic versions are working.

Use vagrant to bring up your kube-master and kube-nodes. The default number of nodes is 2, but you can change this by setting a different env variable for NUM_NODES.

$ vagrant up --no-provision

If your Atomic Host image needs updating, you can do it before provisioning, like this:

$ for i in {kube-node-1,kube-master,kube-node-2}; do vagrant ssh $i -c "sudo atomic host upgrade"; done
$ vagrant reload --no-provision

Time to configure kubernetes, but running the ansible playbook on the kube-master:

$ vagrant provision kube-master

Kubernetes should be all set up now:

$ vagrant ssh kube-master
$ kubectl get nodes

Fedora 22 needs this: ansible/ansible-modules-extras#885

kube-apiserver 6443

Why change the kube-apiserver listen port to 6443, from its conventional default of 443?

If you don't and you're running centos atomic, no one can talk to kube-apiserver, and you'll find that kubectl get nodes returns no nodes.

Aug 12 14:18:48 <eparis>        jbrooks: ansible is setting the apiserver to use port 443, but the daemon runs non-root and doesn't have perm to bind port 443   :-(
Aug 12 14:19:07 <eparis>        jbrooks: when you ran it (as root) it had perm.....
Aug 12 14:19:37 <eparis>        jbrooks: looks like you also need to change roles/kubernetes/defaults/main.yml
Aug 12 14:20:00 <eparis>        jbrooks: 6443 will hopefully do it....
Aug 12 14:20:30 <eparis>        jbrooks: i should have 443 working in the next release.
Aug 12 14:21:19 <eparis>        jbrooks: other options would be to cp /usr/lib/systemd/system/kube-apiserver.server /etc/systemd/system;  then edit it so it runs as root; systemctl restart kube-apiserver

TASK: [master | add cap_net_bind_service to kube-apiserver] ******************* 
failed: [kube-master] => {"failed": true}
stderr: Failed to set capabilities on file `/usr/bin/kube-apiserver' (Read-only file system)
usage: setcap [-q] [-v] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]

 Note <filename> must be a regular (non-symlink) file.

msg: Unable to set capabilities of /usr/bin/kube-apiserver

OK, so, in Fedora, the kubernetes pkg gets the capability to bind to low ports in its spec file, so with regular fedora or fedora atomic, there's no issue, kube-apiserver is ready to bind to 443 out of the box. In RHEL (and, therefore, CentOS), kube-apiserver does not have this capability out of the box. In regular centos, the ansible script takes care of adding this capability. In centos atomic, kube-apiserver lives in /usr, which is read-only, so ansible cannot make this change.

There's a bug: https://bugzilla.redhat.com/show_bug.cgi?id=1275379

In the bug, Colin suggests using systemd socket activation to enable kube-apiserver to listen on 443. I haven't managed to get this working. I've tried w/ many variations on:

[Unit]
Description=kube-apiserver Activation Socket

[Socket]
ListenStream=0.0.0.0:443

[Install]
WantedBy=sockets.target

I've also tried various tweaks to the kube-apiserver.service file.

My lack of success may have something to do w/ this:

Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details) or via the traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).

http://0pointer.de/public/systemd-man/systemd.socket.html

Current status

It seems good/best to get RHEL to change its kubernetes package to set the capability out of the box, in the same way that Fedora's pkg does: http://pkgs.fedoraproject.org/cgit/kubernetes.git/commit/?id=857cddbff129e70655e68ced08df11a931cb39e6.

Another possibility would be changing the ansible to make kube-apiserver run as root, only in case of atomic-but-not-fedora. I could send a PR for this. It seems unattractive, though.

Another possibility would be adding to the needed setcap fu to the treecompose-post script for centos atomic: https://github.com/CentOS/sig-atomic-buildscripts/blob/downstream/treecompose-post.sh and https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md.

I could also get the centos kubernetes pkg patched...

And finally, I could say, screw it, kube-apiserver on centos atomic needs its own special nonstandard port.

A PR to make this value more visible, w/ some comment:

kubernetes-retired/contrib#242

It seems good/best to get RHEL to change its kubernetes package to set the capability out of the box,

It's insecure unless one also does: http://pkgs.fedoraproject.org/cgit/kubernetes.git/commit/?id=7869d324 The commit message doesn't explain this.

I can't say when or if a similar change will make it into the RHEL package, but I assert that regardless, the right fix is socket activation.

But, does kube-apiserver support socket activation?