This is @LucasWerkmeistr's #systemdTOTD thread, gathered using Spooler and reposted here with only some typographic adjustments by me.
Alright everyone, how about some systemd Tip Of The Day goodness?
I'll start with some meta basics: systemd.index(7): list of all systemd manpages (including documentation for C APIs). Explore!
(Reminder: systemd.index(7) refers to manpage systemd.index in section 7, can be viewed with man 7 systemd.index. Section may be omitted.)
systemd.directives(7): list of all systemd config file options, CLI options, journal fields, files, kernel cmdline options, …
What does MemoryHigh=1G mean? What does --firmware-setup do? What's /etc/machine-info? systemd.directives(7) tells you where to look.
systemctl enable --now UNIT: enable and start unit in one command. Also works with disable (stop).
systemctl list-timers: list all systemd timers currently active, with times of last and next activation. (Add --all to list timers not currently running, too.)
systemd manages coredumps for you: coredumpctl list lists coredumps, coredumpctl gdb debugs the last crashed process.
(Your binaries are probably stripped, but it's still better than nothing, and you don't have to go searching for the coredump file first.)
systemd.exec(5) documents many sandboxing options that you can apply to services. I'll focus on them for this week of
ProtectSystem+ProtectHome allow you to make /usr, /boot, /etc, /home+/root read-only or completely inaccessible for a service.
(You should put these in almost all your service files – services should never write to /usr, /boot or /home unless they do system updates.)
GENERAL DISCLAIMER: Always consult the manpage, I oversimplify for Twitter. For example, ProtectSystem+ProtectHome aren't just booleans.
systemctl list-timers: put a service into its own network namespace with no devices except loopback. No phoning home ;) Similarly, PrivateDevices=yes: give a service its own fake /dev file system with null, zero, random etc., but no real devices.
You can see the contents for yourself with:
sudo systemd-run -tq -p PrivateDevices=yes /bin/ls -l /devNoNewPrivileges=yes: ensure that a service (incl children) never gains new privileges (ignore setuid bit, file capabilities, …)
(The kernel feature behind this is prctl(PR_SET_NO_NEW_PRIVS, 1);, see prctl(2) and Documentation/prctl/no_new_privs.txt.)
CapabilityBoundingSet: restrict the capabilities(7) of a root service (e. g. only CAP_SETUID, CAP_SETGID, CAP_SYS_TIME)
For an example, see systemd-timesyncd:
systemctl cat systemd-timesyncdAmbientCapabilities: grant selective capabilities(7) to a non-root service (e. g. CAP_NET_BIND_SERVICE, bind ports < 1024)
And that concludes systemd.exec(5) week! There's a lot more in that manpage, and I might come back to it later, but let's move on for now.
systemd-run(1): run a command like a service. Set properties with -p. Connect to terminal with -t. Tweak environment with -E.
For example, to see the effect of PrivateDevices:
sudo systemd-run -t -p PrivateDevices=yes ls /dev(Add -q to suppress extra info.)
Or to see the default environment of a service:
sudo systemd-run -t env(That's not entirely accurate, since -t adds a TERM. Without it, the output ends up in the journal, and you get a JOURNAL_STREAM instead.)
You can also use systemd-run to run a service later, e. g. --on-active=10m (in ten minutes) or --on-calendar=11:00 (11AM today)
In a way:
at = systemd-run --on-calendar
atq = systemctl list-timers
atrm = systemctl stop
Unit names can be abbreviated. sshd means sshd.service, /var means var.mount, /dev/sda means dev-sda.device.
You can use systemd-escape(1) to escape or unescape unit names.
$ systemd-escape -p --suffix=device /dev/sda
dev-sda.deviceBeware: systemd unit and configuration files do not support end-of-line comments!
# comment
Foo=bar # not a comment!systemctl status, to show the status of a unit (its processes, journal entries, …), also accepts PIDs.
For example, systemctl --user status $$ shows me that my current shell is part of gnome-terminal-server.service.
systemd-inhibit(1) runs a command while delaying or blocking certain actions (sleep, shutdown, …). List locks with --list.
For example, a quick-and-dirty way to temporarily disable sleep on lid close:
systemd-inhibit --what handle-lid-switch sleep infinityGNOME uses this to take responsibility for keypresses away from logind :)
I quite liked the concept of a “topic week” (systemd.exec(5) last time), so let's do that again! This week, journalctl(1).
journalctl -u UNIT: show all log entries for the specified UNIT.
journalctl -u sshd
journalctl -u logrotate.timerSupports all of these abbreviations, of course
journalctl -f: show recent journal entries and then keep printing new ones, like tail -f.
journalctl -b: select boot. -b -3 ≙ third boot before last, -b 5 ≙ fifth boot in journal. -b without arg means current boot.
journalctl --list-boots lists all boots in the journal with ID and offset. If your --list-boots only shows current boot, try creating /var/log/journal/ – see Storage=auto in journald.conf(5) for why.
journalctl --since / --until: restrict to entries in a certain time range. Accepts strings like "3 days ago" or "-10m".
For the full format of time specifications, see systemd.time(7), specifically the PARSING TIMESTAMPS section.
journalctl -e: show just the last few messages and scroll to end of pager. Control amount with -n (default: 1000).
The log output at the end of systemctl status UNIT is roughly equivalent to journalctl -u UNIT -b -e -n 10.
journalctl FIELD=VALUE: filter for messages with VALUE in FIELD. Processes can freely attach fields and values to messages.
For example, in the upcoming Apache 2.6 you'll be able to do this:
journalctl REQUEST_HOSTNAME= REQUEST_URI=/foo/bar (example.com)More info on that here: Apache Module mod_journald
You can list all available fields with journalctl -N, and all values for a field with journalctl -F FIELD. journalctl's awesome tab completion uses that to offer:
journalctl FOO<tab> # complete names
journalctl FOOBAR=<tab> # complete values
Even programs that don't use the journal API leave useful information in the fields:
journalctl SYSLOG_IDENTIFIER=thunderbird.desktop
Attaching the same FIELD=VALUE to thousands of messages is dirt cheap, by the way, so no need to hold back there :)
The journal itself, for example, attaches multiple mostly-constant metadata fields to each message – see systemd.journal-fields(7). How is it possible that fields are so cheap and can be searched for and enumerated nigh-instantaneously? You have one guess ;)
Addendum: there's a dedicated option for a SYSLOG_IDENTIFIER match too, -t (thanks Zbigniew Jędrzejewski-Szmek!).
systemd-cat: run a command with output connected to the journal.
systemd-cat echo hi
echo hi | systemd-catI assume that to achieve SYSLOG_IDENTIFIER=thunderbird.desktop, GNOME simply runs systemd-cat -t thunderbird.desktop thunderbird.
(There's no way to tell from the process list because systemd-cat is just a bit of setup and then exec(), nothing sticks around.)
You can create a journal entry with arbitrary fields from a script using logger --journald[=file] (reads from file or stdin).
Unit files can contain documentation links:
[Unit]
Documentation=
Documentation=man:example(7) (example.com)systemctl help UNIT|PID: show the first manpage in Documentation= for UNIT (or PID's unit).
CORRECTION: All man pages of the unit are shown (concatenated), not just the first one. I was confused by the concatenation. Sorry.
systemctl mask UNIT: completely prevent a unit from being started (stronger than disable). With --now, also stop.
Like enable and disable, this is just a symlink under the hood – in this case, to /dev/null (from /etc or, with --runtime, /run).
systemd-path NAME: print system and user paths, taking into account env vars and some other conditions. Useful for scripts.
Without argument, prints all paths.
--suffix can also be useful:
systemd-path --suffix myprogram user-binaries # install myprogram here
systemctl kill UNIT: Send a signal (--signal SIGNAL, default SIGTERM) to all (or --kill-who=main|control) processes of a unit.
For example, you can easily suspend+resume a unit:
systemctl kill -s STOP UNIT
systemctl kill -s CONT UNIT
(Link)[https://twitter.com/LucasWerkmeistr/status/797570189697183744]
systemd-cgls: show the cgroup tree and the processes of each cgroup. Can be more informative than parent process relationship.
systemd-cgtop: show resource usage of cgroups (not terribly useful unless you have accounting enabled for them).
By default, units only have task accounting. Enable more with e. g. CPUAccounting=yes for one unit or DefaultCPUAccounting=yes globally. Default*Accounting= is configured in /etc/systemd/system.conf; see systemd.resource-control(7) for the different controllers.
There are several signals you can send to PID1 if it doesn't respond via systemctl (e. g. if dbus is hosed) – see systemd(1).
- SIGTERM: daemon-reexec. Serialize state to file system, reexecute self, deserialize state again. (systemctl daemon-reexec)
- SIGUSR1: reconnect to dbus.
- SIGUSR2: dump state in textual form (presumably to kernel console).
- SIGHUP: daemon-reload.
- SIGRTMIN+20/21: enable/disable status messages on console
- SIGRTMIN+22/23: set log level to debug/info
There are loads of signals for starting targets like halt/reboot/rescue, or for immediately halting/rebooting/etc. See manpage for details.
systemctl list-dependencies UNIT: list dependencies of UNIT (duh). Found via @gdamjan on freenode – thanks!
With --reverse, show reverse dependencies (e. g. of /tmp, aka tmp.mount).
With --after/--before, show ordering dependencies.
Journal messages can often be identified by the MESSAGE_ID field, a 128-bit ID. Much better than grepping the message! As a user / administrator / whatever, you can see the MESSAGE_ID in output formats like -o verbose or -o json-pretty…
…and add a match for it to see only those messages:
journalctl MESSAGE_ID=7d4958e842da4a758f6c1cdc7b36dcc5 # systemd's “Starting UNIT…”And as a developer, you can generate new IDs for your messages with journalctl --new-id128, which prints a new ID in several formats.
busctl tree and busctl introspect let you explore dbus from the terminal. Try it out – busctl has AMAZING tab completion.
I'm serious: enter busctl introspect, hit tab a few times, and start exploring. Super easy. (Add --user for the user bus.)
With busctl call, busctl get-property and busctl set-property, you can access dbus from shell scripts.
Example:
busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager GetUnitByPID u $PIDThat's the command to find the systemd unit of a process. Add --user to consult the user systemd instance instead.
systemd-analyze verify FILES…: verify unit files before loading them into the service manager.
systemd-analyze plot > plot.svg: emit an SVG image that shows startup times of units, highlighting units that delayed others
Failed at step NAMESPACE spawning /usr/bin/…: No such file or directory Possible cause: ReadOnlyPaths=/does-not-exist in unit “No such file or directory” is obvious enough, but bridging from NAMESPACE to ReadOnlyPaths et al. can be tricky the first time you see this
All systemd tools read the environment variable SYSTEMD_LOG_LEVEL, e. g. SYSTEMD_LOG_LEVEL=debug journalctl.
(Another tip from #systemd on Freenode – thanks Mic92!)
The available values of SYSTEMD_LOG_LEVEL are the classic syslog(3) log levels: debug, info, notice, warning, err, crit, alert, emerg. You can also add SYSTEMD_LOG_LOCATION={yes,no} to include source code location of message, if you want to investigate further.
Also, SYSTEMD_LOG_TARGET={console,kmsg,journal,syslog,auto,safe,…}. auto = console/journal/kmsg. safe = console/kmsg.
And finally, all systemd services also parse the kernel commandline for these options: systemd.log_{level,location,target}.
systemctl show-environment: show service manager environment. Particularly interesting with --user in a graphical session.
systemd and its daemons load configuration from /usr/lib, /run, /etc, in that order. (Distros without /usr merge: just /lib.)
Distribution packages install into /usr/lib. Dynamic generators write to /run. System administrator overrides both in /etc. You should never, EVER have to edit a file in /usr/lib. Copy it to /etc and edit it there. No more conflicts on package upgrade. Or, better yet, keep the OS-provided file and augment it with overrides in /etc/…/$filename.d/$whatever.conf.
systemctl edit UNIT can be used to create such override files, or (with --full) copy the original file and edit it.
User manager follows similar logic, so $XDG_RUNTIME_DIR/systemd/user is a convenient place to play with unit files. You don't need root since it's just your own instance, and it's in /run so files don't stay around forever if you forget to delete them.
I want to talk about overrides a bit more, since they're pretty useful. Any *.conf file in a FOO.d/ directory in the systemd load path will augment the original FOO file.
For example, you can add sandboxing to sshd.service in /etc/systemd/system/sshd.service.d/sandboxing.conf. (Don't forget the .service suffix! /etc/systemd/sshd.d/sandboxing.conf won't work.)
Overrides of the same name in different load paths override each other, just like regular files. For instance, symlink /etc/systemd/user/dbus.service.d/flatpak.conf to /dev/null to kill /usr/lib/systemd/user/dbus.service.d/flatpak.conf.
systemctl set-property UNIT Foo=x Bar=y… can be used to set properties on services. They are stored in overrides in /etc.
(Or, with --runtime, in /run, so they're lost on reboot – temporary.)
systemd-delta: show overriden and drop-in configuration files. (Without any arguments: show all of them.)
systemd-delta /etc: only changes in /etc (system administrator)
systemd-delta systemd/user: changes to user units (no matter where)
… systemd exposes many cgroup resource control settings (e. g. CPUQuota=50%, MemoryMax=1G). See systemd.resource-control(5).
You can also apply most of them to a running service, e. g. systemctl set-property -p TasksMax=100 apache2.
(For most other properties, the change is only applied the next time they're started.) systemd.resource-control(5) isn't just for services: you can also apply those settings to slices, which group multiple units.
By default, system services go in system.slice, user sessions in user.slice, and containers in machine.slice but you can also configure your own slice units (foo.slice in a unit load path) and add units to it with Slice=foo.slice. Also, instanced services get a per-template slice by default (e. g. system-tor.slice for tor@.service, system-getty.slice for getty@, …). (CORRECTION: an earlier version of that tweet read “instanced units”, but according to manpage that's only for services, not other units.)
You can configure sysctl parameters in /etc/sysctl.d/*.conf files – systemd-sysctl.service will read and apply them on boot.
This follows the same /etc > /run > /usr/lib logic as everything else, so packages can place their files in /usr/lib/sysctl.d/60-mypkg.conf. Which is also its big advantage over plain old /etc/sysctl.conf – that's a single file, packages can't edit it without risk of conflicts. If a package installs /usr/lib/sysctl.d/60-foo.conf, you can override it very simply by symlinking /etc/sysctl.d/60-foo.conf to /dev/null. Or place a different file there, with amended content. Files of the same basename override each other.
I forgot to update this yesterday and it's almost midnight, so let's have several updates in a row about systemd's container support. systemd-nspawn(1) can be used to run a system image on disk as a container. Specify root dir with -D or image file with -i.
By default, it will just set up the namespaces and then drop you into a shell (or run a specified command); add -b to boot a full init. Alternatively to -D or -i, you can specify a machine name with -M. Machines are stored under /var/lib/machines and managed by machinectl(1). systemctl start|stop MACHINE manages such a container as a system service (internally, systemd-nspawn@MACHINE.service).
systemctl clone NAME1 NAME2 clones an image. If /var/lib/machines is btrfs, creates a subvolume (very efficient).
(In fact, if /var/lib/machines is blank and non-btrfs, machinectl will mkfs.btrfs /var/lib/machines.raw and loop mount it there.)
Use systemd-nspawn --ephemeral to run a temporary copy of an image. Again, super efficient if the backing store is btrfs.
For example, I have a Debian-Jessie-Base image that I use like this:
sudo systemd-nspawn -M Debian-Jessie-Base --ephemeral /bin/bash <buildYou can also mark an image as read-only with machinectl read-only IMAGE true|false.
For example, my aforementioned Debian-Jessie-Base image is read-only, so I don't accidentally pollute it when I forget a --ephemeral switch. Every now and then, I'll make it writable again (machinectl read-only DJB false) and update it (systemd-nspawn -M DJB apt update,upgrade). machinectl can also manage images with {pull,import,export}-{tar,raw}.
(I must admit I've never used this feature, because I couldn't find suitable URLs to pull – most distros publish installers, not images.)
machinectl set-limit [NAME] BYTES: limit the size of a single container image, or the total size of all of them.
(Running this command on an empty non-btrfs /var/lib/machines will, as mentioned before, create and loop-mount a btrfs filesystem for it.)
machinectl shell NAME gets you interactive root shell on a running container. Specify another user with USER@NAME (like ssh).
If you omit the machine name (i. e. machinectl shell user@), it defaults to the host system (called .host). Somehow, this turned into “systemd wants to replace su”, which is such an astonishingly bad take I really don't understand how it happened. But that's still the second Google result for “machinectl shell”. It's ridiculous. No, nobody wants to replace plain old su. (The same thing happened to systemd-mount as well, but we'll get to that later. Spoiler: no, mount isn't being replaced either.)
You can also run a command in a container with systemd-run -M NAME COMMAND…. (systemctl, journalctl etc. also support -M.)
So what's the difference between machinectl shell NAME COMMAND… and systemd-run -M NAME COMMAND…? machinectl shell is intended more for interactive usage (COMMAND can specify an alternative shell), systemd-run for scripts. systemd-run can propagate the exit status with --wait (which machinectl shell doesn't). It also offers more configuration options.
systemctl list-units --state failed: list all failed units. (Add --type service to exclude other kinds of units.)
(Since list-units is the default command, you can abbreviate this to systemctl --state failed if you really want to. A bit too terse IMHO.)
systemctl start --no-block UNIT: start a unit and don't wait for startup to complete. Mostly useful for long-running oneshots.
(Oneshot services are ones that run a command and then exit. Mostly useful for timers: man-db, updatedb, logrotate…)
Most systemd commands support globbing (wildcards) for units. E. g. see status of all tor instances: systemctl status tor@*
Or to see all log messages of a socket-activated Accept=yes service: journalctl -u git@*
Doesn't just work for templates, of course. Check status of systemd's own services: systemctl status systemd-*.service
Forget about failed instances of a service: systemctl reset-failed git@*
Send a reminder notification in two hours: systemd-run --user --on-active 2h notify-send "empty the dryer" (there are a million ways to do this, of course, this is just one)
systemd-socket-proxyd(8) allows you to socket activate services that don't support inheriting sockets from their environment. proxyd inherits socket from systemd and forwards everything to a specified address, where the other service listens by itself. See manpage. That in itself isn't terribly exciting, but it gets way better: systemd-socket-proxyd(8) allows you to network isolate a svc that should only listen on one connection, no other communication.
See the manpage for an example. Service+proxyd have PrivateNetwork=yes, proxyd JoinsNamespaceOf= service. systemd opens socket in real network namespace. proxyd inherits it and forwards it to a port in private namespace shared with service. (If the other service can listen on a Unix domain socket instead of an Internet address, you don't even need JoinsNamespaceOf=.)
systemctl list-unit-files --state generated --type service --no-legend | cut -d' ' -f1 # list unmigrated SysV services
ANNOUNCEMENT: systemdTOTD is no longer a daily series, but I'll continue to occasionally post tips under the hashtag as I discover them. If I tweet multiple tips in a day, imagine I'm still playing catch-up ;) but I don't have enough ideas for one tip every day anymore. But for a quick-fire round, here's the last few ideas I still had saved up:
systemd-mount /dev/WHAT [/WHERE]: create a transient mount unit, just like systemd-run creates a transient service (+ opt. timer) unit. So why use systemd-mount over mount? A few possible reasons:
- The mount unit is a systemd unit like any other unit: systemd can manage dependencies (parent mounts), apply common settings, etc. - If you omit the WHERE path, systemd chooses a suitable folder (based on the device label) and creates it. Pure laziness, in other words :)
- With --discover (on by default if WHERE is omitted), systemd also reads some more metadata, e. g. for the unit description.
- But most importantly: if device is removable, systemd automatically creates an automount with short idle timeout instead of normal mount. This means that the device is always unmounted when it's not used for a while, making it safe to remove without ejecting it first. (Automounts are also a systemd unit type, so if you think this sounds cool, you can also do this for other mounts you have :) )
systemd-mount --list: list mountable devices with path, model, type, label, UUID, and other info in one nice table.
I'm definitely going to use this instead of blkid in the future when I need UUIDs for /etc/fstab – so much easier to use and remember :D
SystemCallFilter=: add a blacklist or whitelist of syscalls to a service. See systemd.exec(5) for details.
Blocked system calls can either kill the service immediately (uncatchable SIGSYS) or return a specified error number. And since specifying individual syscalls is tedious, systemd also defines some sets you can use, e. g.:
SystemCallFilter=~@mount @raw-io
The tilde indicates that the filter is a blacklist, not a whitelist, and the at means that the name is a system call set. Most services systemd ships use a blacklist of sets (see for example systemd-{journald,logind,importd}). Specifying a whitelist of individual syscalls would be more restrictive, but more likely to break if systemd's dependencies change behavior.
systemd-inhibit wget # download a large file without automatic suspend on idle like
journalctl --since 06:30 --until 07:00 # what happened during today's unattended upgrade?
systemd-analyze set-log-level debug && systemd-analyze set-log-target console # change log behavior of systemd daemon itself
systemd-analyze syscall-filter @basic-io @file-system # print syscalls in a syscall group, useful to debug a SystemCallFilter
You can use systemd-tmpfiles to clean up temporary files (systemd-tmpfiles-clean.timer runs daily) Link
systemctl --failed # shorter alias for systemctl [list-units] --state=failed (list-units is the default command)
--failed even predates the more general --state, but was deprecated + undocumented for a time; restored in v233 with bef19548 systemctl set-property user.slice MemorySwapMax=0 # keep user sessions in RAM (requires cgroups v2)
echo 1 | sudo tee /sys/fs/cgroup/memory/user.slice/memory.swappiness # similar effect in cgroups v1, in conjunction with echo 100 | sudo tee /sys/fs/cgroup/memory/system.slice/memory.swappiness # …raise swappiness of all system services, or alternatively echo 100 | sudo tee /sys/fs/cgroup/memory/system.slice/rdf2hdt.service/memory.swappiness # …raise swappiness of a single service.
alias iotop='sudo systemd-run -qt -p DynamicUser=yes -p AmbientCapabilities=CAP_NET_ADMIN iotop' # restrict iotop capabilities
for a commented and more thoroughly sandboxed version, see Link
.link files: configure physical network devices. Processed by udev.
.netdev files: configure virtual network devices. Created by systemd-networkd.
.network files: configure networks applying to devices. Processed by systemd-networkd. Apply to links, refer to netdevs.
.link: set MTU, enable Wake-on-LAN
.netdev: create bridge
.network: enable DHCP, connect to bridge
(I've had a hard time understanding the difference between those files, I hope this makes sense? See also Link and Link
# list units by IP traffic
systemctl --no-legend |
while read -r unit _; do
bytes=$(systemctl show -p IPEgressBytes --value -- "$unit")
[[ $bytes == 18446744073709551615 ]] && bytes=-1
((bytes > 0)) && printf '% 15d %s\n' "$bytes" "$unit"
done | sort -rn
Requires DefaultIPAccounting=yes in systemd-system.conf(5) or IPAccounting=yes on individual units. (The screenshot shows a system with systemd v237; on v238+, you will also get aggregated statistics for slice units.)
TIL: systemd reads unit files not just from /usr/lib/systemd (and /run, /etc, and some others), but also from /usr/local/lib/systemd, compatible with the standard prefix in a GNU-style build. systemd debugging tip: temporarily increase the log level.
level=$(systemd-analyze log-level)
sudo systemd-analyze log-level debug
sudo systemctl start whatever.service
sudo systemd-analyze log-level "$level"
journalctl -b -eif you want, you can get a bit fancier with the journalctl arguments, e. g.
journalctl -b _SYSTEMD_UNIT=whatever.service + _SYSTEMD_UNIT=init.scope UNIT=whatever.service
or even
journalctl -b CODE_FILE=../systemd-stable/src/basic/mount-util.sr
(exact path will depend on distro)
systemd debugging tip: you can simply strace PID1.
sudo strace -p1 -f -yy # terminal 1
sudo systemctl start whatever.service # terminal 2systemd is single-threaded and usually spends most of its idle time in a blocking epoll_wait() syscall, so there's typically not much unrelated noise in the strace output. if you want to automate this without using two terminals:
sudo sh -c 'strace -p1 -f -yy -o/tmp/trace & systemd-run …; kill %1'
(this time using systemd-run instead of systemctl start – both might be useful depending on situation)
systemd-inhibit --what=sleep:idle:handle-lid-switch --who="$USER" --why=Music --mode=block vlc *.mp3 # use your laptop as a portable music player without it going to sleep when you shut the lid
can also be useful to charge another device via USB while you're not using the laptop (use sleep infinity for the command in that case)