See Docker issue #24834 for the more complete background. The short story is that if lxcfs is installed (which it is by default in Ubuntu 16.04) and you attempt to use the Docker daemon with user namespaces enabled, most likely you will end up with a confusing "container command could not be invoked error" like the one here:
# docker run --rm busybox date
docker: Error response from daemon: Container command 'date' could not be invoked..
LXC has resolved this problem with a more minimal environment in their own PR#124 but it may be a little while before that hits distro updates or the next LXC release. Until you get an updated LXC you might want to understand how to stop the lxcfs cgroup mounts in the interim. The steps below provide that guidance.
If you aren't going to use LXC and/or LXD on this system, then one option is to remove the components so that even if you umount the problematic cgroup mounts, they won't come back on restart.
But first, to verify this is the problem you are facing you can look at the current mounts, and verify that lxcfs
has mounted the cgroups in /run/lxcfs/controllers:
# mount | grep lxc
tmpfs on /run/lxcfs/controllers type tmpfs (rw,relatime,size=100k,mode=700)
cpuset on /run/lxcfs/controllers/cpuset type cgroup (rw,relatime,cpuset)
memory on /run/lxcfs/controllers/memory type cgroup (rw,relatime,memory)
hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,relatime,hugetlb)
pids on /run/lxcfs/controllers/pids type cgroup (rw,relatime,pids)
perf_event on /run/lxcfs/controllers/perf_event type cgroup (rw,relatime,perf_event)
net_cls,net_prio on /run/lxcfs/controllers/net_cls,net_prio type cgroup (rw,relatime,net_cls,net_prio)
devices on /run/lxcfs/controllers/devices type cgroup (rw,relatime,devices)
freezer on /run/lxcfs/controllers/freezer type cgroup (rw,relatime,freezer)
cpu,cpuacct on /run/lxcfs/controllers/cpu,cpuacct type cgroup (rw,relatime,cpu,cpuacct)
blkio on /run/lxcfs/controllers/blkio type cgroup (rw,relatime,blkio)
name=systemd on /run/lxcfs/controllers/name=systemd type cgroup (rw,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
After verifying that this is your issue--lxcfs is mounting the cgroups and a starting container is trying to read from these paths as an unprivileged user (and failing)--then you can take steps to mitigate this until you have an updated LXC installed.
The shortest and easiest step to resolve the issue is to remove the lxcfs package and reboot:
# apt-get remove lxcfs
# shutdown -r now
After a reboot these extra cgroup mounts will not exist and you will be back in operation.
If you don't plan on using LXC/LXD on your system, then another option is to remove all the LXC packages from your system for now. First, I retrieve the list of the LXC packages I have installed on my system:
# dpkg -l lx*
+++-===================================-======================-======================-===========================================================================
un lxc <none> <none> (no description available)
ii lxc-common 2.0.3-0ubuntu1~ubuntu1 amd64 Linux Containers userspace tools (common tools)
un lxc-docker <none> <none> (no description available)
un lxc-docker-virtual-package <none> <none> (no description available)
un lxc1 <none> <none> (no description available)
ii lxcfs 2.0.2-0ubuntu1~ubuntu1 amd64 FUSE based filesystem for LXC
ii lxd 2.0.3-0ubuntu1~ubuntu1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.3-0ubuntu1~ubuntu1 amd64 Container hypervisor based on LXC - client
un lxd-tools <none> <none> (no description available)
It looks like I only have 4 packages installed, so I will stop the lxcfs service and then use apt-get to remove the packages:
# service lxcfs stop
# apt-get remove lxc-common lxcfs lxd lxd-client
The only additional problem is that stopping lxcfs does not get rid of the cgroup mounts, so I have to do it manually if I don't want to wait for a reboot cycle:
# for i in name=systemd blkio cpu,cpuacct freezer devices net_cls,net_prio perf_event pids hugetlb memory cpuset; do umount /run/lxcfs/controllers/$i; done
# umount /var/lxcfs/controllers
To verify docker + user namespaces works properly; start the docker daemon with userns enabled and then attempt to run the same simple container and verify that it works:
# docker run --rm busybox date
Thu Aug 11 20:42:47 UTC 2016