kubernetes ansible setenforce 1

kube-setenforce-1.md

I use these ansible scripts (see also) to bring up kubernetes clusters, on centos and fedora. The scripts are awesome, but they put selinux into permissive mode, because, "tokens and selinux don't work together."

That's unfortunate, because one of the main things I'm doing when I turn up a kubernetes cluster is bug-hunting, and selinux can be a source of bugs.

I commented out the bit that put selinux into permissive mode:

#- name: Set selinux permissive because tokens and selinux don't work together
#  selinux: state=permissive policy={{ ansible_selinux.type }}
#  when: ansible_selinux is defined and ansible_selinux.status == "enabled"

I then brought up a cluster, and found that most/all of the kubernetes-addons refused to work.

Someone asked a question about this on Stack Overflow, and the answer provided was simple enough:

You probably need to run the following command to set the SELinux context correctly in the volumes directory. I have an open issue to make this happen automatically in the future:

sudo chcon -Rt svirt_sandbox_file_t /var/lib/kubelet

To implement this suggested fix, I added this bit to roles/node/tasks/main.yml:

- name: chcon of kubelet rather than set selinux permissive
  command: chcon -Rt svirt_sandbox_file_t /var/lib/kubelet
  when: ansible_selinux is defined and ansible_selinux.status == "enabled"

Following the change, the kube-addons pods all seemed to work, with the exception of the fluentd-elasticsearch pod, which still had selinux denials.

Dan Walsh wrote a blog post about selinux and the fluentd container in which he suggested a policy fix that partly worked (the fluentd container would run, but it couldn't actually read the logs it was supposed to read). A commenter on that post suggested an additional line to add to the policy that made the container work. Here's the policy:

policy_module(container_logger, 1.0)

virt_sandbox_domain_template(container_logger)
##############################
# virt_sandbox_net_domain(container_logger_t)
gen_require(`
 attribute   sandbox_net_domain;
')

typeattribute container_logger_t sandbox_net_domain;
##############################
logging_manage_all_logs(container_logger_t)
docker_read_lib_files(container_logger_t)

To use the policy, paste the above into a file called container_logger.te, and then, on your centos or fedora (swap dnf in for yum) host:

$ sudo yum install selinux-policy-devel
$ make -f /usr/share/selinux/devel/Makefile container_logger.pp
$ sudo semodule -i container_logger.pp

This ought to do it for the container, but for the container+kubernetes case, we need to tell kubernetes to run the fluentd container with the appropriate selinux type. Here's what the yaml for that should look like:

    securityContext:
      seLinuxOptions:
        type: container_logger_t

Here it is again, sandwiched into the otherwise-stock yaml from upstream:

apiVersion: v1
kind: Pod
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.15
    resources:
      limits:
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 200Mi
    securityContext:
      seLinuxOptions:
        type: container_logger_t
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers

In order for this to work, make sure that there's no SecurityContextDeny in the KUBE_ADMISSION_CONTROL= line of the file /etc/kubernetes/apiserver in your kube master.

With the ansible changes and the policy applied to my nodes, and with the modded fluentd yaml, the kube-addons work.

Next steps / Questions:

Is this the best/right way to accomplish this?

At this point... I think so? There's some worthwhile discussion in this github issue. In any case, this is better than no selinux at all, which is the status quo for a kube cluster w/ kube-addons running, AFAICT.

That extra line added to the policy that Dan Walsh blogged about, is that legit?

It makes the container work, otherwise... I don't know. The line is the last one in the policy above, docker_read_lib_files(container_logger_t)

How do you compile selinux policy on atomic, where you don't have selinux-policy-devel installed?

With a container, I figure. I'll work out / write down those steps. I hope soon to see this policy ship w/ centos and fedora, so there'd be no policy building required.

Can't this Just Work?

I'm going to get this into a PR and into the kubernetes/contrib repo. We'll need to modify the yaml for the fluentd container on the fly or get upstream to accept some selinux-foo into the upstream yaml, and we'll need to figure out how to deal with the policy building bit. I'm not sure if what a non-selinux-supporting kube cluster makes of seLinuxOptions stuff -- it'd make sense just to ignore it, but I don't know if that's how it works.