systemd-nspawn container architecture

readme.md

systemd-nspawn container architecture

This short document will show how to turn systemd-nspawn into a usable containeration system.

Those instructions here should work under Arch Linux and Debian 9

Host requirements

• systemd-nspawn and machinectl (systemd-container package under Debian)
• dnsmasq
• debootstrap

Setup

1. We need to create a network bridge device where systemd-nspawn can connect all the containers. Using systemd-networkd you can create those two files under /etc/systemd/network/.

   /etc/systemd/network/br0.netdev

    # Tell systemd-networkd to create a bridge device
    [NetDev]
    Name=br0
    Kind=bridge
   

   /etc/systemd/network/br0.network

    # Configure ip address for the bridge
    [Match]
    Name=br0
    
    [Network]
    Address=172.23.0.1/24
    LinkLocalAddressing=yes
    IPMasquerade=yes
    LLDP=yes
    EmitLLDP=customer-bridge
   

   After creating those configuration files and running command below you should be good.

    systemctl enable --now systemd-networkd
   

   If you need to make any changes to the network configuration then you can just restart systemd-networkd to reload the config.

    systemctl restart systemd-networkd
   

2. So that all of our containers would always get a static ip address we are gona use dnsmasq.

   Under Debian 9 you will need to use a custom systemd service file for dnsmasq so that we could bypass the Debian customizations.

   /etc/systemd/system/dnsmasq.service

    # Service file from Arch Linux but with changed paths
    [Unit]
    Description=A lightweight DHCP and caching DNS server
    Before=systemd-resolved.service
    After=network.target
    Documentation=man:dnsmasq(8)
    
    [Service]
    Type=dbus
    BusName=uk.org.thekelleys.dnsmasq
    ExecStartPre=/usr/sbin/dnsmasq --test
    ExecStart=/usr/sbin/dnsmasq -k --enable-dbus --user=dnsmasq --pid-file
    ExecReload=/bin/kill -HUP $MAINPID
    
    [Install]
    WantedBy=multi-user.target
   

   You need to tell systemd that the service file has changed

    systemctl daemon-reload
   

   Next we are going to tell dnsmasq to do static leases based on container hostname. In this config file im using 172.23.0.1/24 as the container network.

   /etc/dnsmasq.conf

    domain=.local
    #no-poll # don't constanly poll /etc/resolv.conf
    #resolv-file=/etc/resolv.conf
    no-resolv
    server=8.8.8.8
    server=8.8.4.4
    domain-needed
    bogus-priv
    listen-address=127.0.0.1,172.23.0.1
    dhcp-range=172.23.0.100,172.23.0.200,255.255.255.0,12h
    
    # When systemd-networkd or some other dhcp client requests for a ip address
    # they will be given a address based on their hostname.
    # This section can be moved to seperate file using dhcp-hostsfile option
    dhcp-host=http,172.23.0.2
    dhcp-host=mail,172.23.0.3
    dhcp-host=worpress,172.23.0.4
    dhcp-host=sql,172.23.0.5
   

3. systemd-nspawn@.service must be told to use the br0 bridge For that we have to edit ExecStart line in the service file and add --network-bridge=br0 to the end of it.

   First get the currently used ExecStart value by running

    systemctl cat systemd-nspawn@.service
   

   After that we need to create a override.conf file that changes that file. You can do that either by using command systemctl edit systemd-nspawn@.service or by manualy creating a file called /etc/systemd/system/systemd-nspawn@.service.d/override.conf

   Contents of the override file should be something like this:

    [Service]
    ExecStart=
    ExecStart=<current execstart line> --network-bridge=br0
   

   For Arch Linux the contents would be

    [Service]
    ExecStart=
    ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i --network-bridge=br0
   

Creating containers

That part is quite simple, just create a folder /var/lib/machines/<name> where name is your container hostname and deboostrap into it. machinectl start <name> should boot your container up and after staring systemd-networkd you should have a working network.

I have created some scripts automate container creation part

• create_arch_container.sh - Creates a Arch Linux container
• create_deb_container.sh - Creates a Ubuntu 16.04 container, but you can change it quite easily by changing SUITE argument from xenial to something else

Administrating nspawn containers

• machinectl start <name> - Boot container named name

• machinectl poweroff <name> - Poweroff the container

• machinectl shell <name> [/bin/sh] - Run a command in the container, by default the command is /bin/sh. For Debian/Ubuntu containers you probably want to tell it to use /bin/bash

• machinectl restart <name> - this will probably break nspawn because on startup it will try to register with machined and fail because it takes some time to release a container name in machined. (there is bug report about it somewhere)

• Port forwarding If there is a file /etc/systemd/nspawn/<name>.nspawn then systemd-nspawn will load additional configuration options from it.

  To do port forwarding you can do something like this:

    # /etc/systemd/nspawn/http.nspawn 
    [Network]
    Port=tcp:80:80
    Port=tcp:443:443
  

  You can add many more options from systemd.nspawn(5) man page

Things that are broken under Debian

• dnsmasq

  /etc/init.d/dnsmasq scripts always injects its own options to dnsmasq command line and will break the configuration

• IPTables integration is disabled for systemd (bug #787480)

  This means that this nice IPMasquerade=yes option under systemd-networkd and port=tcp:80:80 port forwarding options do no work and you have to use good old iptables rules like iptables -t nat -A POSTROUTING -s 172.23.0.0/24 -j MASQUERADE to enable NAT and iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -m addrtype --dst-type LOCAL -j DNAT --to-destination 172.23.0.2:80 to do port forwarding.