Running containers using systemd-nspawn

gistfile1.txt

Containers provide a quick way to configure and test new operating system versions, deployment scripts etc in minimal containers which
are merely files stored on your container host.
One aspect that some folk may find daunting is the networking aspect, to make containers useful the default networking which will allow
- Container->Internet connectivity means that the container will not be visible to your local network so say ssh'ing in from say your
desktop machine or launching Xsessions just won't work.
So before you start we need to configure a bridged network on the principle lan interface on your host and then that bridged interface
can be shared by your containers and will allow unfettered inbound and outbound connectivity to your other local hosts and devices.
Unfortunately setting this up can be confusing because each linux distro will have it's own preferred method for controlling the network
and even these will vary across Desktop and Server variants of the same distro...
My approach to this is simple - since I am proposing to use systemd to manage the containers, we can use systemd-networkd to configure
and control the network -

So let's assume you have a suitable host with a suitable version of Ubuntu server or desktop installed and in which case it is likely
that NetworkManager is controlling your network interfaces.
Open a terminal session on your host and install the following packages -
sudo apt install systemd systemd-container
sudo mkdir /etc/systemd/nspawn
Check your current IP configuration to ascertain the current interface name and IP configuration details -
If using DHCP that's all you need to know
If using a static IP then you will need
IP address and subnet
Gateway address
DNS address

Make a note of these for later..

Next we'll create the configuration files to switch over from using your existing interface name to a new bridge name, nothing will
actually happen at this stage since NetworkManager is still running and managing your hosts interfaces..

All systemd-networkd configuration files live in a root owned directory- 
/etc/systemd/network
To configure a bridge we need to create 3 files, one defines the bridge name and interface type, the second associates (binds) the bridge name with a physical interface, and the third sets the bridge interface network parameters -
So 1st lets create the bridge -
sudo su
<your root pwd>
# nano /etc/systemd/network/hostbridge.netdev
[NetDev]
Name=br0
Kind=bridge

Save and exit the file
Next lets create the bind -
# nano /etc/systemd/network/10-bind.network
[Match]
Name=enp1s0f0       <- substitute with your interface name e.g. eth0

[Network]
Bridge=br0
LinkLocalAddressing=no
IPv6AcceptRA=no

Save and exit the file
Finally lets create the interface configuration -
# nano /etc/systemd/network/hostbridge.network
[Match]
Name=br0

[Network]
DHCP=ipv4

Save and exit the file
Note: this assumes you are using DHCP on the host, if you are using a static address then substitute the final DHCP line with -
Address=192.168.0.130/24
Gateway=192.168.0.254
DNS=192.168.0.120

substituting your specific IP addresses for each line from your earlier notes....

Note: If you have multiple network devices on your host but one or more are not connected then by default systemd-networkd will wait upon boot until either all are online or timeout after 2 minutes.
To circumvent this you can tell the daemon to only wait until one interface has become ready before continuing the system startup -

sudo systemctl edit systemd-networkd-wait-online.service
navigate to the 3rd line from the top and add -
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any

CTRL-X to save and exit

This creates an override file in /etc/systemd/system that will enable the boot to proceed as soon as one interface becomes ready, and on my rather old hardware once the machine has moved past POST it boots in under 3 secs.

Now we have the required configuration files we can now turn off NetworkManager and switch over to systemd-networkd -

sudo systemctl stop NetworkManager       <- stop the service
sudo systemctl mask Network-Manager      <- prevent the service starting on boot
sudo systemctl start systemd-networkd    
sudo systemctl enable systemd-networkd   <- enable it to start on next boot

If all worked ok, your network interfaces br0 and physical interface should both be up, and e.g. ping to a foreign address - google or whatever should work ok.
You can check the interfaces using ip a -

enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 00:1b:78:59:2b:2e brd ff:ff:ff:ff:ff:ff
br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:50:82:b4:22:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.240/24 metric 1024 brd 192.168.0.255 scope global dynamic br0
       valid_lft 73419sec preferred_lft 73419sec
    inet6 fe80::f450:82ff:feb4:224d/64 scope link 
       valid_lft forever preferred_lft forever
       
 ...and using systend's network status tool networkctl -
IDX LINK            TYPE     OPERATIONAL SETUP      
  1 lo              loopback carrier     unmanaged
  2 eno1            ether    no-carrier  configuring  <onboard and unused nic
  3 enp1s0f0        ether    enslaved    configured   < port 0 of a dual port PCI nic that is connected
  4 enp1s0f1        ether    no-carrier  configuring  < disconnected 2nd port of PCI nic
  5 br0             bridge   routable    configured   < my virtual bridge
 12 vb-rms-debi4QQB ether    degraded    unmanaged    < an active container using the shared br0 - hence vb prefix

6 links listed.
 
If you have multiple network onterfaces on your host you may experience slow boot times because by default the systemd-wait-online service by default will wait for all available interfaces to come up, and it will wait and eventually timeout after 2 minutes.
To prevent this create an override file and allow startup to proceed if any interface becomes ready -

sudo systemctl edit systemd-networkd-wait-online.service
and place this block within the predefined gap below the template stanza -

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any

save and exit - this will write the override to /etc


So now we have an environment that allows us to install a new container, attach it to the bridge and let it have connectivity to my local lan + devices and the internet.

To build a new container using a Debian or Ubuntu distro you can use a program called debootstrap so -
On the host machine -
sudo apt install debootstrap

The syntax of how to use it is pretty simple -
sudo debootstrap -components=main,universe <codename> <container name> <repository-url>

the components bit just means the core binaries and universe repositories 
<codename> is the distro specific version e.g. bookworm, jammy etc. to see what releases your version of debootstrap supports - 
ls /usr/share/debootstrap/scripts

By default systemd-nspawn looks for containers in -

/var/lib/machines


..so if you have a particular location you'd prefer to locate these then prepend that to the container name and create a soft link in /var/lib/machines pointing to the real machine location so that the systemd tools can find it-

sudo ln -s <real machine path/name> </var/lib/machines/machine-name>


So to create a container called rms-ubuntu-lts based upon Ubuntu Jammy stored in the default location -

sudo debootstrap --include=systemd-container,openssh-server --components=main,universe jammy /var/lib/machines/rms-ubuntu-LTS http://archive.ubuntu.com/ubuntu

The container needs some specific packages  - dbus-broker and systemd-container to work so these are included as add-ons
to the base build

Depending on your internet connectivity and local disk speeds this may take 10 or more minutes to download and install, but you get a running commentary along the way..

This does install the bare minimum to make a bootable system but things like man pages and even the man package itself are not included.
It makes this a pretty minimal but fast process and you can always add stuff later if you wish!


To give an idea as to size a Ubuntu Jammy build + RMS + one nights capture +gdm3 (the Desktop environment) occupies 18GB so still pretty
small.

Post the base system installation debootstrap images require some housekeeping -

By default the only account on the system is root, and it has no password set and also for whatever reason the container inherits the hostname of the parent machine - which can lead to some confusion!!

So next step is to launch a root shell in the container in single user mode -
To exit this shell when finished you hold down the Ctrl key and type 3 ]'s in quick sucession - so CTRl]]]

systemd-nspawn -D <path to container/container name>
Note: you are root so no sudo is necessary

Add a regular user -
adduser rms
newpassword prompt
repeat password
accept defaults

grant user sudo privs - note for Debian distros sudo itself is not installed and at present your container has no active network so cannot access any remote repositories. Workaround is to set a password for root using the passwd cmd and fix later on when the network 
is configured and working.Also Debian have implemented tighter security on containers such that e.g. unprivileged users cannot use -
/usr/bin/ping.
I will document these later.

usermod -aG sudo rms

Change the hostname - best to set this to the container name again to avoid later confusion....

echo "myhostname" >/etc/hostname

NOTE: It is way simpler to have the container hostname agree with the actual container directory name - that way you know what each container ...erm contains!
If your existing container directory doesn't match the contained o/s hostname then after exiting the container you can simply issue a 
sudo mv <oldname> <newname>

This is possible because as far as the host is concerned a container is just a file so can be copied around, moved, cloned or whatever without breaking anything!

set systemd-networkd to start on-boot

systemctl enable systemd-networkd

exit the container -CTRl]]]

The regular way of managing and controlling contaiiners is via the cmdline tool machinectl and it by default will look for containers in 
/var/lib/machines


..and it looks for per-container configuration files in -
/etc/systemd/nspawn with files named in the format <containername>.nspawn

typical contents of a .nspawn would be 

[Network]
Bridge=br0

- this tells systemd that this host will grab an auto generated interface (in practice named host0) which will act as a slave to the parents bridge interface br0


So to launch and background the container -

machinectl start <hostname>

The machinectl cmd requires privileged authentication so it will prompt for your password and then return

a bare machinectl will list currently running containers -
(vRMS) user@rms-test:~/source/RMS$ machinectl
MACHINE     CLASS     SERVICE        OS     VERSION ADDRESSES     
ubuntu-test container systemd-nspawn ubuntu 22.04   192.168.0.247…

If you configured DHCP networking for the container then it should acquire an IP address and register it's hostname in your DNS
so you could e.g. ssh into it by name -

ssh ubuntu-test


You can also login to the container using machinectl -

(vRMS) eharman@rms-test:~/source/RMS$ machinectl login ubuntu-test
==== AUTHENTICATING FOR org.freedesktop.machine1.login ===
Authentication is required to log into a local container.
Authenticating as: ed (eharman)
Password: 
==== AUTHENTICATION COMPLETE ===
Connected to machine ubuntu-test. Press ^] three times within 1s to exit session.

Ubuntu 22.04 LTS ubuntu-test pts/2

ubuntu-test login: 

If you want to use a graphical desktop there are a few extra packages to install and use rdp to display it.
Note: running this in a container does consume some resources and significant network bandwidth -

login to the container
sudo apt install --no-install-recommends ubuntu-desktop
sudo apt install xrdp

start the login greeter screen -
sudo systemctl restart gdm3

Then on your remote system launch the RDP client, enter the destination hostname or IP address and remember to enable the options to
use different credentials if required and then click Connect.
You should then see the default Ubuntu desktop login screen, login using your local credentials and launch GUI apps.

I'm unsure why by default no desktop icons are displayed - it may be because this is a minimal install and so many of the required apps are missing, however you can enable them by installing -

sudo apt install gnome-shell-extension-desktop-icons-ng

then install the app to set preferences

sudo apt install gnome-shell-extension-prefs

To enable Desktop icons open the desktop extensions app and enable Desktop Icons NG (DING)

I haven't quite sorted launching from a click but right mouse and open works albeit showlivestream consumes over 40% util on 4 cores....