systemd http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#1729

mail.md

Message #1729 received at 727708@bugs.debian.org (full text, mbox): From: Russ Allbery rra@debian.org To: 727708@bugs.debian.org Subject: Re: Bug#727708: init system other points, and conclusion Date: Sun, 29 Dec 2013 16:10:10 -0800

We seem to be at the point of the process where at least those of us who did early investigation are stating conclusions. I think I have enough information to state mine, so will attempt to do so here.

This is probably going to be rather long, as there were quite a few factors that concerned me and that I wanted to investigate.

The brief summary is that I believe Debian should adopt systemd as its default init system on Linux. There are two separate conceptual areas in which I think systemd offers substantial advantages over upstart, each of which I would consider sufficient to choose systemd on its own. Together, they make a compelling case for systemd. This position would have substantial implications for upgrade paths and for non-Linux ports; I'll discuss a bit of that below, but most of it in the separate branch of this bug report that Ian opened on that topic.

Below, I first discuss the other choices before us besides systemd and upstart. Then I look at a straight technical comparison between the two init systems, and finally look at issues of maintenance, community, ecosystem, and portability. The three main criteria on which I was evaluating both systems were technical capabilities, surrounding ecosystem, and portability. The latter two turned out to be deeply entangled, so I discuss them together.

1. Other Choices

First, other choices besides systemd and upstart.

There were three replacement init systems proposed to the Technical Committee to replace sysvinit, plus the existing status quo. The third option, OpenRC, is a more conservative and less revolutionary change than either systemd or upstart. It continues to use the existing sysvinit init process but replaces the startup script management with a more robust shell library and additional features.

I think the OpenRC developers are great people and I wish them all the success in the world with their project, but I just don't think it's ambitious enough for Debian's needs. If we're going to the effort of replacing init systems and changing our startup scripts, a bare minimum requirement for me is that we at least address the known weaknesses of the sysvinit mechanism, namely:

• Lack of integration with kernel-level events to properly order startup.
• No mechanism for process monitoring and restarting beyond inittab.
• Heavy reliance on shell scripting rather than declarative syntax.
• A fork and exit with PID file model for daemon startup.

My impression of OpenRC is that it is not attempting to solve these issues in the same way that systemd and upstart are. To the extent that these issues are on the OpenRC roadmap, it's not as far along as either systemd or upstart is. It's difficult to evaluate since the OpenRC documentation is rather sparse and lacks the comprehensive manual available to both systemd and upstart, which is itself a sign of a lack of project maturity.

I don't think that switching to OpenRC offers enough clear benefit over the status quo.

That raises the other obvious option: sticking with sysvinit. I've made my position on this fairly clear in other threads, so I won't reiterate it here at length. The short version is that I turned to other tools to manage daemons years ago because sysvinit was simply inadequate, and my feeling on that hasn't changed. The model of fork and exit without clear synchronization points is inherently racy, the boot model encoded into sysvinit doesn't reflect a modern system boot, and maintaining large and complex init scripts as conffiles has been painful for years. Nearly every init script, including the ones in my own packages, have various edge-case bugs or problems because it's very hard to write robust service startup in shell, even with the excellent helper programs and shell libraries that Debian has available. A quick perusal of /etc/init.d/skeleton and the complex case statements and careful attention to status codes required for a proper init script makes this case clear.

I think the choice of a default init system for Linux is a choice between systemd and upstart. We would be doing ourselves and our users a disservice to stick with the status quo, or even a moderate update of the status quo to add a simpler service definition. The limitations have been well-known for years, and I think it's telling that most other operating systems, even fairly conservative ones, have moved away from the System V init script model.

The last option that was before us was supporting multiple init systems. I consider this a variation on a transition plan, with a possibly infinite time horizon, and will discuss this separately when I talk about transition plans.

2. Core Service Management Functionality

As reported to this bug, I did a fairly extensive evaluation of both upstart and systemd by converting one of my packages, which provides a network UDP service, to a native configuration with both systems. While doing so, I tried to approach each init system on its own terms and investigate what full, native support of that init system would look like, both from a Debian packaging perspective and from an upstream perspective. I also tried to work through the upgrade path from an existing init script with an external /etc/default configuration file and how that would be handled with both systemd and upstart.

I started this process with the expectation that systemd and upstart would be roughly evenly matched in capabilities. My expectation was that I would uncover some minor differences and variations, and some different philosophical approaches, but no deeply compelling differentiation.

To my surprise, that's not what happened. Rather, I concluded that systemd has a substantial technical advantage over upstart, primarily in terms of available useful features, but also in terms of fundamental design.

2.1. General Impressions

systemd feels like a software package that has been used and pounded on in a wide variety of real-world situations, and has grown the flexibility and adaptibility that is required to make a wide variety of use cases work. upstart, on the other hand, has a minimal design and a ready escape to shell scripting, which may have discouraged directly tackling a broader array of use cases. Regardless, there are a bunch of cases that systemd handles cleanly with simple configuration that would require shell script fragments or other workarounds in Ubuntu, which in turn makes the startup configurations less reliable and harder to debug.

I was quite impressed throughout the process of developing systemd unit files. Every time I realized I needed some piece of functionality to configure the daemon properly, systemd already had it.

2.2. Major Functionality Gaps

Here are the major pieces of functionality that I think would have to be added to upstart for rough feature parity:

• Socket activation, by which I don't mean lazy start of daemons, although it enables that, but init management of socket setup so that daemons can start in parallel.

  This has been discussed elsewhere on the thread, but I want to note here that systemd's approach is bold and innovative. We've had multiple discussions in Debian lists in the past where people have felt somewhat depressed or discouraged about Debian's lack of innovation or unwillingness to tackle sweeping improvements. After having studied and implemented socket activation, I think this is one of those opportunities, and we should not pass it by.

  There are a variety of advantages to socket activation that have been discussed elsewhere, and I'm not going to repeat them all here. But one I want to call out is the advantage for an enterprise systems administration environment. Right now, in order to configure bind addresses or IPv6 behavior for my services, I have to dig into the individual configuration syntax or command-line flags of each separate daemon, and often there's no easy way to set these parameters without making intrusive changes to daemon startup. Socket activation lets me manage all of this through a simple configuration override that I drop into /etc via (for example) Puppet, and the syntax is the same for every service that uses it. It would obviously take quite some time to get there, but that's a really nice vision of the future, and one that would make a real difference for Debian use cases I care about.

  upstart has a socket activation protocol, but it would need an almost-complete redesign in order to be used the way that systemd's can be used. It doesn't support passing multiple sockets (required for complex daemons, some IPv6 scenarios, and binding to multiple but not all interfaces), it doesn't support IPv6 at all, it doesn't support UDP sockets, and its configuration syntax is inadequate to represent the parameters that would be useful in a real-world case. It also doesn't separate the socket configuration from the daemon configuration, which makes it harder for a local systems administrator to control binding behavior without changing other properties of daemon initialization.

• Integrated daemon status. This one caught me by surprise, since the systemd journal was functionality that I expected to dislike. But I was surprised at how well-implemented it is, and systemctl status blew me away. I think any systems administrator who has tried to debug a running service will be immediately struck by the differences between upstart:

  lbcd start/running, process 32294

  and systemd:

  lbcd.service - responder for load balancing Loaded: loaded (/lib/systemd/system/lbcd.service; enabled) Active: active (running) since Sun 2013-12-29 13:01:24 PST; 1h 11min ago Docs: man:lbcd(8) http://www.eyrie.org/~eagle/software/lbcd/ Main PID: 25290 (lbcd) CGroup: name=systemd:/system/lbcd.service └─25290 /usr/sbin/lbcd -f -l

  Dec 29 13:01:24 wanderer systemd[1]: Starting responder for load balancing... Dec 29 13:01:24 wanderer systemd[1]: Started responder for load balancing. Dec 29 13:01:24 wanderer lbcd[25290]: ready to accept requests Dec 29 13:01:43 wanderer lbcd[25290]: request from ::1 (version 3)

  Both are clearly superior to sysvinit, which bails on the problem entirely and forces reimplementation in every init script, but the systemd approach takes this to another level. And this is not an easy change for upstart. While some more data could be added, like the command line taken from ps, the most useful addition in systemd is the log summary. And that relies on the journal, which is a fundamental design decision of systemd.

  And yes, all of those log messages are also in the syslog files where one would expect to find them. And systemd can also capture standard output and standard error from daemons and drop that in the journal and from there into syslog, which makes it much easier to uncover daemon startup problems that resulted in complaints to standard error instead of syslog. This cannot even be easily replaced with something that might parse the syslog files, even given output forwarding to syslog (something upstart currently doesn't have), since the journal will continue to work properly even if all syslog messages are forwarded off the host, stored in some other format, or stored in some other file. systemd is agnostic to the underlying syslog implementation.

• Security defense in depth. Both upstart and systemd support the basics (setting the user and group, process limits, and so forth). However, systemd adds a multitude of additional defense in depth features, ranging from capability limits to private namespaces or the ability to deny a job access to the network. This is just a simple matter of programming on the upstart side, but it still contributes to the general feature deficit; the capabilities in systemd exist today. I'm sure I'm not the only systems administrator who is expecting security features and this sort of defense in depth to become increasingly important over the next few years.

  Here again, I think we have an opportunity for Debian to be more innovative and forward-looking in what we attempt to accomplish in the archive by adopting frameworks that let us incorporate the principles of least privilege and defense in depth into our standard daemon configurations.

There are also a plethora of minor features and tuning available in systemd but not in upstart. None of this is as significant as the points mentioned above, and none of it is as difficult to implement, but it's not currently implemented, and I think it speaks to systemd having been tested against a broader array of use cases.

2.3. Event vs. Dependency Model

There is one UI design difference between systemd and upstart that's less clear-cut, but which I think will surprise people. systemd is built around familiar dependencies between services, and starts services in dependency order. There are some twists, such as allowing a service to create a reverse dependency (make another service depend on it), but it's the basic design that's familiar to any packager, or to users of languages like Puppet. upstart, on the other hand, uses a message bus model: services are started when particular events are received, and dependencies are expressed by listing the events required to trigger startup (or some other action).

Conceptually, both of these designs are equivalent. They both construct a DAG that's used to order service startup. However, upstart complicates matters by having two types of messages on its message bus: signals and methods (technically, there are also hooks, but the distinction doesn't matter for this point). Signals behave like the typical asynchronous message bus event, or like a dependency: they trigger services to start, but the service issuing the signal does not care whether anyone listens or not. Methods do not; methods are effectively synchronous calls and the service issuing a method event waits until the method event has been acted on before continuing.

The UI problem with this approach is that it creates a pitfall with rather noticable consequences. If someone ever confuses a signal event and a method event and starts a service on a method event instead, it is then very easy to block startup of some fundamental system service because its method event never completes due to deadlock. This is made somewhat more likely by the fact that method events are the default in initctl emit commands, whereas signal events require a flag.

Again, this is not a fundamental issue with either system; either representation is mathematically convertable into the other. But it's difficult to mess up dependencies in quite the same way. One can create cycles, but unless one is modifying the dependencies of core services, it's hard to create a cycle that involves a core service. upstart provides a way to shoot oneself in the foot by blocking startup of a core service by listening to the wrong type of event. This model doesn't, so far as I could find, offer any clear advantages over a dependency structure in compensation.

2.4. Configuration File Model

There is one place where I came into this evaluation preferring the upstart design over the systemd design, and came away with a continued preference, but a more mild one: the configuration file model. systemd uses an /etc overrides /lib model, where all unit configurations are installed in /lib and only local overrides and some configuration goes into /etc. upstart uses the (more familiar to Debian) model where the daemon configuration is a conffile in /etc.

Both approaches have real advantages, but I think the upstart approach has slightly more. The systemd model means that one no longer has to add various guards to daemon configurations to allow for the possibility that the package has been uninstalled but not purged. Those continue to be necessary with upstart (and continue to be written in shell; systemd actually has a nicer language for doing this, even though it's not needed). However, the upstart approach makes it easier to preserve and merge local changes with upstream changes. In the systemd model, the local administrator has line-by-line granularity on overrides of systemd unit configurations, which while solving much of the problem does not help with the specific case of wanting to change the flags passed to the daemon. If the package later changes the flags in some orthogonal way, it's easy for the system to miss that change. This is something that, under systemd, will probably require development of new tools to warn the adminsitrator of what's happened. upstart avoids this problem by having the whole configuration be managed as a conffile.

I think the upstart approach is better, but I think the drawbacks of the systemd approach could be mostly overcome with some fairly simple Debian tools.

2.5. Summary

I think the technical comparison between upstart and systemd as both projects exist today substantially favors systemd, at both the feature and design level. When picking between both products as they currently exist on the basis of their current capabilities and future adaptibility, I have no qualms about picking systemd.

3. Ecosystem and Portability

One of the primary concerns from the start of this conversation has been around portability of any new init system. One advantage of the extreme simplicity of sysvinit is that it's extremely portable; this advantage continues to be shared by OpenRC. Both of the more-functional init systems are Linux-specific. However, upstream attitudes towards portability differ. This ties directly into the development models of both systemd and upstart, the community momentum, and the larger surrounding ecosystem.

3.1. Ecosystem Reality Check

One of the points that I think may have been obscured in the discussion, but which is important to highlight, is that basically all parties have agreed that Debian will adopt large portions of systemd. systemd is an umbrella project that includes multiple components, some more significant than others. Most of those components are clearly superior to anything we have available now on Linux platforms and will be used in the distribution going forward.

In other words, this debate is not actually about systemd vs. upstart in the most obvious sense. Rather, the question, assuming one has narrowed the choices to those two contenders, is between adopting all the major components of systemd including the init system, or adopting most of the major components of systemd but replacing the init system with upstart. Either way, we'll be running udev, logind, some systemd D-Bus services, and most likely timedated and possibly hostnamed for desktop environments.

I think this changes the nature of the discussion in some key ways. We're not really talking about choosing between two competing ecosystems. Rather, we're talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better.

Now, I am generally on the side that says loose coupling of ecosystems is an inherent good. However, I don't agree that it's such an inherent good that we should disassemble things just for the sake of having disassembled things. At feature parity, and absent any compelling reason to swap components, I think we should take the path of least resistance and use the integrations that other people have already developed. Debian has more than enough hard integration problems to solve without creating new ones for ourselves unnecessarily. But that's the key word: unnecessarily. If we do have a reason for doing this, we should seriously consider it.

Therefore, I believe the burden of proof is on upstart to show that it is a clearly superior init system along some axis, whether that be functionality or portability or flexibility or maintainability, to warrant going to the effort of disassembling a part of the systemd ecosystem and swapping in our own component.

3.2. Portability

This is a difficult topic to clearly discuss, since it is, in essence, all future speculation at this point.

I should state up front that, in making these sorts of decisions around free software projects, I have a relatively high future discount rate. In other words, I give substantially less credit to something that does not exist now but could exist in the future. I don't discount it to zero, but I do discount it relatively strongly. Others may not.

I do this because free software projects and volunteer projects are inherently unpredictable. The free software world is stuffed to the gills with roadmaps that never actually happened, through no fault of any of the people involved. It's easy to agree that something would be a good idea, and another matter to actually drive it through to completion.

Right now, neither systemd nor upstart work on non-Linux platforms. Therefore, right now, adopting either of them means that we either jettison our non-Linux ports or adopt a transition plan that retains support for sysvinit scripts. Right now, there is minimal difference between the two projects in terms of portability; they both make extensive use of Linux-specific APIs and have hooks for Linux-specific actions.

However, there is a porting effort for upstart to kFreeBSD underway, and the current upstart maintainers have indicated more interest in portability than the systemd maintainers. That's been a point of significant friction over systemd (and was, in the past, also a point of friction with the previous upstart upstream, although that's subsequently changed). So there is a real advantage for upstart here, but it's one that has to be discounted because it's potential future work that could happen, but which is certainly not guaranteed to happen.

Another point worth considering here is that the best way, from Debian's perspective, of porting either project to kFreeBSD or the Hurd is to implement the currently Linux-specific interfaces on those platforms in some fashion. (An inotify and epoll API that uses kqueue under the hood, for example.) To the extent that this is possible, it benefits both upstart and systemd equally, as well as many other programs in the archive that are written to currently Linux-specific APIs. This is an approach that's been common for years in different porting scenarios; I use it myself to maintain compatibility with both MIT Kerberos and Heimdal in the Kerberos-related packages I maintain.

Finally, note the ecosystem point above. To maintain feature parity across Debian's ports, there already appears to be widespread agreement that components of systemd will have to be ported, particularly logind and possibly some of the other services. Now, that's not quite the same thing as porting the init system: it's possible those components use fewer Linux-specific interfaces (I've not checked), it's possible that alternative implementations of the same functionality can be provided (which IIRC is what happened with udev in some fashion), and not being able to run major desktop environments is not the same thing as not being able to boot. But I do think it blunts some of the porting argument. The non-Linux ports are going to have to port, fork, or replace systemd components anyway, regardless of the choice of init system, or drop out of feature parity with the Linux ports.

So, in short, I consider portability to be a possible benefit of upstart, but I'm inclined to discount that advantage for several reasons. One, it's not yet actually materialized and still may not, and two, systemd porting looks like it's going to be on the table regardless. I therefore think that we should deal with this issue through how we structure a transition plan, rather than taking it as a reason to choose upstart over systemd. More on that in another message.

3.3. Project Momentum

One of the reasons why I'm leery of the future portability argument for upstart, and one of the reasons why I'm leery of upstart in general, is that I'm quite worried upstart will prove to be a blind alley.

I think there are several reasons to be concerned here. None of them is persuasive in isolation, but taken together I think they raise significant cause for concern:

• Red Hat adopted upstart but never did a wholescale conversion, and then abandoned upstart in favor of systemd. Obviously, one should not put too much weight on this; Red Hat is a commercial company that has a wealth of reasons for its actions that do not apply to Debian. But I think it's still worth noting that the only non-Ubuntu major adopter of upstart backed away from it.

• upstart is older than systemd but has significantly fewer features. Now, the danger of this sort of metric is that features can be added as "padding" without any real significance or advantage. But having spent serious time with both systems, I don't believe that's the case here. systemd is not adding extraneous features; rather, it's adding significant, useful functionality and real-world adaptability, and upstart is trailing despite being an older project.

• systemd has a broader community. SuSE and Red Hat are both converting, there is significant interest across the general Linux community, major upstreams of Debian such as GNOME and KDE are adopting systemd support (and in some cases even requiring it), and systemd is tackling significant problems, such as logind, that everyone agrees need to be solved. By comparison, upstart is effectively used only by Ubuntu, and there isn't the same sort of enthusiasm or attempts to tackle broad problems happening at present in the upstart community so far as I can see. This is reasonable if upstart is mature and mostly complete software, but that was not my personal experience.

• There appears to be some direct tension between GNOME upstream and upstart, not mostly due to upstart itself but because of corporate direction at Canonical. Again, this can easily be overstated. But I do think that Debian will want to continue to support GNOME going forward, and doing that with upstart will clearly require more work within the project than doing that with systemd. This is another case where we shouldn't shy away from the work if it's necessary, but we also shouldn't adopt unnecessary work.

Over the past few months, I've also put out some feelers to other colleagues, and the uniform reaction I got in response is that systemd is a better technical solution than upstart. I think this speaks to the general momentum around systemd, and will directly affect our ease of integration in the future. I know that after my personal experience with both projects, I'm excited to add systemd support to my projects as upstream, and not particularly enthused about upstart from an upstream perspective since it doesn't offer me any clear benefits.

3.4. Summary

I'm concerned that, if we adopt upstart, in two or three years we'll end up wanting to do the same thing that Red Hat did, back out, and switch to systemd. That would be a huge amount of wasted effort. Even worse would be to end up in that situation and decide that the conversion is too much work, and then just settle for an init system that is harder to integrate and provides less functionality.

I remain unconvinced of the long-term growth curve of the upstart project. I don't think it's going to be abandoned completely, at least unless Ubuntu decides to switch (which seems unlikely at the moment) or Canonical dissolves (which also seems unlikely). I do think there's a significant danger that it will stagnate and fall behind in terms of desired features, particularly since this appears to already be happening. I don't have faith in the path that takes upstart from where it is now to something with feature parity with systemd as it is now, let alone something that's clearly better than systemd. And I think Debian as a project should be aiming for better, not merely sufficient.

The portability issues are significant. However, I don't think they provide a clear advantage to upstart. It's possible that they will in the future, at which point the ecosystem argument becomes much more difficult and much narrower. But the fact remains that we'll be using large components of systemd across the distribution anyway, which means that swapping out the init system doesn't add as much portability as one might hope, and increases our integration burden.

I think we should make wise decisions about which areas we want to invest project effort. I dislike investing significant project effort in catch-up efforts that, when complete, merely get us back to where we would have been if we'd chosen a different solution. I don't think that's wise stewardship of project resources. I want to see Debian focus its efforts on places where we can make a real difference, where we can be leaders. That means adopting the best-of-breed existing solutions and building on top of them, not reinventing wheels and thereby starting from a trailing position.

4. Conclusion

If I'm correct in my analysis of the community and ecosystem dynamics, I think upstart needs to show that it is a significantly better technical choice than systemd in order to warrant the additional project work that will be required to build on top of upstart. Given feature parity, I believe we should adopt systemd so that we can focus our efforts on interesting new problems rather than on redoing integrations that other people have already done.

My personal analysis did not show that upstart was significantly better than systemd, or even at feature parity. Rather, I believe it is currently trailing systemd substantially in multiple areas, some of which will require significant design changes.

Given that, I believe systemd is the clear choice, despite the portability issues that we will incur by choosing it. However, I think that means we need to be very careful about how we handle a transition. I intend to comment on that in a separate message (which will probably be tomorrow given how long writing this message took).

-- Russ Allbery (rra@debian.org) http://www.eyrie.org/~eagle/