julianlam/provisioning-unprivileged-lxc-containers-via-login-or-script.md

## provisioning-unprivileged-lxc-containers-via-login-or-script.md

      
                Raw

        
              provisioning-unprivileged-lxc-containers-via-login-or-script.md
            
          
    Provisioning and usage of unprivileged LXC containers via indirect login or script
As I've discovered, managing LXC containers is fairly straightforward, but when building out a system for provisioning out user maintained instances of NodeBB, it was imperative that unprivileged LXC containers were used, so that in the event of shell breakout from NodeBB followed by privilege escalation of the saas user, the root user in the LXC container would only be an unprivileged user on the host machine.
During the course of development, I ran into numerous blockers when it came to managing LXC containers in unexpected circumstances. Namely:

Using LXC in a subshell is not directly supported. This usually happens under one of the following two circumstances:

After switching users via su or executing lxc-* commands as another user via sudo
Executing lxc-* commands via a program, application, or script. In my case, a Node.js application.


Get familiar with LXC
Take a gander through this tutorial so you get a better feel of the basic LXC commands: lxc-start, lxc-stop, lxc-destroy, and lxc-create.
We're using a DigitalOcean Ubuntu 14.04 droplet, and we'll be creating Ubuntu 14.04 containers.
Start by logging into your host droplet.
Allow creation of unprivileged containers
Some prep work needs to be done so that unprivileged containers can be created:
echo "lxc.id_map = u 0 100000 65536" > ~/.config/lxc/default.conf
echo "lxc.id_map = g 0 100000 65536" >> ~/.config/lxc/default.conf
echo "lxc.network.type = veth" >> ~/.config/lxc/default.conf
echo "lxc.network.link = lxcbr0" >> ~/.config/lxc/default.conf

# As root...
echo "saas veth lxcbr0 10" >> /etc/lxc/lxc-usernet
Distilled from the guide provided by Stéphane Graber's blog: https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/
Now you'll be able to use all of the lxc-* commands as a non-root user. Nice!
Allow your script to use lxc-* commands
I spent most of a day trying to figure out why my unprivileged user couldn't use lxc-* commands, even though I ran through the steps above.
As it turns out, this was because I was logging into root via ssh, and then switching to the unprivileged user (in my case saas) by running su - saas. This issue contains lots of cgroups-type information that went way over my head. Long story short, when you log in via ssh or directly via console, you are granted a cgroup, but not when you switch users afterwards, so the subshell I was using could not start or list LXC containers. I'd get fun little errors like:
lxc: utils.c: mkdir_p: 193 Permission denied - failed to create directory '/run/user/0/lock/'
lxc: lxclock.c: lxclock: 249 Error opening /tmp/1000/lxc//home/saas/.local/share/lxc/foobar
447: error creating container foobar
and
lxc_container: cgmanager.c: lxc_cgmanager_create: 299 call to cgmanager_create_sync failed: invalid request
lxc_container: cgmanager.c: lxc_cgmanager_create: 301 Failed to create hugetlb:foobar
lxc_container: cgmanager.c: cgm_create: 646 Error creating cgroup hugetlb:foobar
lxc_container: start.c: lxc_spawn: 861 failed creating cgroups
lxc_container: start.c: __lxc_start: 1080 failed to spawn 'foobar'
lxc_container: lxc_start.c: main: 342 The container failed to start.
lxc_container: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options.

If you are seeing these kinds of errors, the most direct solution is to log into the user that is executing the lxc-* commands directly (via ssh or console). If your workflow does not allow this, continue reading.

For the first error, a helpful commenter resolved that nicely:
$ unset XDG_SESSION_ID XDG_RUNTIME_DIR
The second one was tricker, and involved installation of the cmanager-utils package, which allowed me to call the cgm binary, which is a neat way to grant my shell access to the cgroup:
$ sudo cgm create all $USER
$ sudo cgm chown all $USER $(id -u) $(id -g)
$ cgm movepid all $USER $$
If you're wanting to call those commands from a script, put those three lines in their own shell script, change $$ to $1, and you can execute it from your program via exec (or its language equivalent), and pass in your program's pid as its first (and only) argument.
In my case, when my Node.js script is run, one of the first things it does is execute this script using execFile, passing in the current pid as its first argument:
#!/bin/bash
unset XDG_RUNTIME_DIR XDG_SESSION_ID  # Most likely this line is not required, but I added it just in case.
sudo cgm create all $USER
sudo cgm chown all $USER $(id -u) $(id -g)
cgm movepid all $USER $1
Like so:
var execFile = require('child_process').execFile;

execFile(__dirname + '/scripts/cgroups-assign.sh', [process.pid], {
	env: {
		USER: 'myuser'   // I'm not sure if process.getuid would work here, try it and let me know.
	}
});

Enjoy calling lxc-* commands from your script/program!