These are initially notes to myself about the things I had to look up while taking on a Linux project after a taking a decade or two hiatus from Linux to focus on Solaris, SmartOS, and illumos. Following my footsteps may find this useful.
This is the init system replacement, like SMF was for Solaris.
# systemctl list-unit-files
You can use service name arguments or wildcards to trim the list to just those of interest.
If your terminal supports underlines, you will see underlines separating the units of different types (service, slice, socket, ...).
The latest log for a service start can be seen with
# systemctl status triton-hostid.service
* triton-hostid.service - Set hostid based on system uuid
Loaded: loaded (/lib/systemd/system/triton-hostid.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sat 2020-01-04 20:10:42 UTC; 3h 10min ago
Docs: https://github.com/joyent/linux-live
Process: 372 ExecStart=/usr/triton/bin/set-hostid (code=exited, status=0/SUCCESS)
Main PID: 372 (code=exited, status=0/SUCCESS)
Jan 04 20:10:42 debian-live-20200104T060517Z systemd[1]: triton-hostid.service: Succeeded.
Jan 04 20:10:42 debian-live-20200104T060517Z systemd[1]: Started Set hostid based on system uuid.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Since I'm needing to muck with things pretty early in boot, I found it was easy to hit dependency cycles. You may see some red scroll by quickly on the console as a result of this.
To see it:
root@debian-live-20200104T060517Z:~# systemd-analyze verify default.target
usr-lib-live-mount-medium.mount: Unit is bound to inactive unit dev-sr0.device. Stopping, too.
sysinit.target: Found ordering cycle on systemd-update-utmp.service/verify-active
sysinit.target: Found dependency on systemd-tmpfiles-setup.service/start
sysinit.target: Found dependency on local-fs.target/start
sysinit.target: Found dependency on zfs-mount.service/start
sysinit.target: Found dependency on zfs-import.target/start
sysinit.target: Found dependency on zfs-import-scan.service/start
sysinit.target: Found dependency on triton-hostid.service/start
sysinit.target: Found dependency on basic.target/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job systemd-update-utmp.service/verify-active deleted to break ordering cycle starting with sysinit.target/start
...
That goes on a while until it randomly deletes enough services that the cycle is broken. Of course, that leads to cascading failures during boot because things aren't started or get started in the wrong order.
In my case, I needed to tell systemd to not add the implicit dependencies. In the unit file:
[Unit]
DefaultDependencies=no
...
You would think that this service would start before zfs-import-scan.service, but you would be wrong.
# WRONG!
[Unit]
Description=Set hostid based on system uuid
Documentation=https://github.com/joyent/linux-live
[Service]
...
[Install]
RequiredBy=zfs-import-scan.service
You also need Before.
# OK
[Unit]
Description=Set hostid based on system uuid
Documentation=https://github.com/joyent/linux-live
Before=zfs-import-scan.service
[Service]
...
[Install]
RequiredBy=zfs-import-scan.service
systemctl status and friends use options to less that cause it to require horizontal scrolling to see long lines. That's quite annoying. To fix this for all users:
$ echo 'SYSTEMD_LESS=FRXMK; export SYSTEMD_LESS' | sudo tee /etc/profile.d/systemd.sh
SYSTEMD_LESS=FRXMK; export SYSTEMD_LESS
SYSTEMD_LESS is documented at the end of systemctl(1).
When mount ... is uttered, systemd automatically creates .mount units.
I initially went to look for a mount generator that was running after zfs-mount.service, but that does not exist. Digging into the zfs code, I found that it is calling the mount command. The mount command seems to be completly obvlivious to systemd, which seems reasonable.
I'm cautiously optimistic that networking will not be hard...
# zfs clone triton/63d6e664-3f1f-11e8-aef6-a3120cf8dd9d@final triton/nspawn-1
# systemd-nspawn --boot -D /triton/nspawn-1/root -M nspawn-1 --private-users=pick --private-users-chown --private-network
Spawning container nspawn-1 on /triton/nspawn-1/root.
Press ^] three times within 1s to kill container.
Selected user namespace base 743636992 and range 65536.
systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 9 (stretch)!
...
[ OK ] Started Update UTMP about System Runlevel Changes.
Debian GNU/Linux 9 debian-8-01 console
debian-8-01 login: root
Password:
Login incorrect
debian-8-01 login: root
Linux debian-8-01 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
__ . .
_| |_ | .-. . . .-. :--. |-
|_ _| ;| || |(.-' | | |
|__| `--' `-' `;-| `-' ' ' `-'
/ ; Instance (Debian 9.4 (stretch) 20180404)
`-' https://docs.joyent.com/images/container-native-linux
In another window:
root@debian-live-20200104T060517Z:~# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
nspawn-1 container systemd-nspawn debian 9 -
1 machines listed.
root@debian-live-20200104T060517Z:~# machinectl status
Too few arguments.
root@debian-live-20200104T060517Z:~# machinectl status nspawn-1
nspawn-1(dd6b4b54ef9b4fcebd61c4e3c34a59dd)
Since: Sun 2020-01-05 05:59:54 UTC; 7min ago
Leader: 2348 (systemd)
Service: systemd-nspawn; class container
Root: /triton/nspawn-1/root
OS: Debian GNU/Linux 9 (stretch)
UID Shift: 743636992
Unit: machine-nspawn\x2d1.scope
`-payload
|-2348 /lib/systemd/systemd
|-2363 /lib/systemd/systemd-journald
|-2503 /lib/systemd/systemd-logind
|-2504 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
|-2506 /usr/sbin/rsyslogd -n
|-2509 /usr/sbin/cron -f
|-2535 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:108
|-2718 /usr/sbin/sshd -D
|-2738 /bin/login --
`-3148 -bash
Jan 05 05:59:54 debian-live-20200104T060517Z systemd[1]: Started Container nspawn-1.
A way that is perhaps a bit better integrated:
# zfs list triton/deb9
NAME USED AVAIL REFER MOUNTPOINT
triton/deb9 98.1M 17.9G 512M /var/lib/machines/deb9
In /var/lib/machine/deb9/deb9.nspawn:
[Exec]
Boot=on
PrivateUsers=auto
[Network]
Private=on
The machinectl start command can be used as a front-end for systemd-nspawn start -M .... systemd-nspawn wants to find the root of a container at /var/lib/machines/<name>, but that doesn't really match up with how Triton does images. To trick it into behaving (perhaps poorly):
# cd /var/lib/machines/deb9
# mkdir etc
# touch etc/os-release
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/deb9.nspawn <<EOF
[Exec]
PivotRoot=/root
While the machine does start, we end up with a bunch of mounts in the deb9 directory that should really be in the deb9/root directory. Also, I don't think that /var/lib/machines/deb9/deb9.nspawn is used.
Better is probably to have a different service that fires it up. The following is in /etc/systemctl/system/triton-instance-deb9.service.
[Unit]
Description=deb9
[Service]
LimitNOFILE=100000
ExecStart=/usr/bin/systemd-nspawn -b --machine=deb9 --directory=/var/lib/machines/deb9/root
[Install]
Also=dbus.service
Note that this should probably be a systemd service instance: triton-instance@deb9.service. See systemd.unit.