gistfile1.md

Why We No Longer Write Init Scripts

In early 2014, we started switching our in-house services from System V startup (the standard service startup mechanism on RHEL up to RHEL 6) to supervisord, and ceased writing new SysV init scripts. The technical rationale is a bit involved; the end result simplifies development and improves parity between production and development environments, while also being easier to manage and deploy. To explain how this works, let's back up a step and talk about classic Unix daemonization.

Unix Daemons the Old Way, and Why SysV Init Sucks

The classic Unix daemonization system has each daemon (service) started by a dedicated command, which sets up the daemon as a child of PID 1, and then exits. You can see this in practice: run nginx and note that NginX starts in the background, and then returns control of the terminal to your shell. To make this happen, the startup program has to do a very complicated series of system calls. From the systemd documentation:

1. Close all open file descriptors except stdin, stdout, stderr (i.e. the first three file descriptors 0, 1, 2). This ensures that no accidentally passed file descriptor stays around in the daemon process. On Linux, this is best implemented by iterating through /proc/self/fd, with a fallback of iterating from file descriptor 3 to the value returned by getrlimit() for RLIMIT_NOFILE.
2. Reset all signal handlers to their default. This is best done by iterating through the available signals up to the limit of _NSIG and resetting them to SIG_DFL.
3. Reset the signal mask using sigprocmask().
4. Sanitize the environment block, removing or resetting environment variables that might negatively impact daemon runtime.
5. Call fork(), to create a background process.
6. In the child, call setsid() to detach from any terminal and create an independent session.
7. In the child, call fork() again, to ensure that the daemon can never re-acquire a terminal again.
8. Call exit() in the first child, so that only the second child (the actual daemon process) stays around. This ensures that the daemon process is re-parented to init/PID 1, as all daemons should be.
9. In the daemon process, connect /dev/null to standard input, output, and error.
10. In the daemon process, reset the umask to 0, so that the file modes passed to open(), mkdir() and suchlike directly control the access mode of the created files and directories.
11. In the daemon process, change the current directory to the root directory (/), in order to avoid that the daemon involuntarily blocks mount points from being unmounted.
12. In the daemon process, write the daemon PID (as returned by getpid()) to a PID file, for example /run/foobar.pid (for a hypothetical daemon "foobar") to ensure that the daemon cannot be started more than once. This must be implemented in race-free fashion so that the PID file is only updated when it is verified at the same time that the PID previously stored in the PID file no longer exists or belongs to a foreign process. Commonly, some kind of file locking is employed to implement this logic.
13. In the daemon process, drop privileges, if possible and applicable.
14. From the daemon process, notify the original process started that initialization is complete. This can be implemented via an unnamed pipe or similar communication channel that is created before the first fork() and hence available in both the original and the daemon process.
15. Call exit() in the original process. The process that invoked the daemon must be able to rely on that this exit() happens after initialization is complete and all external communication channels are established and accessible.

Obviously, this sequence is rarely followed perfectly. Each daemon has its own subtle variations, and leaves out or mis-implements different steps.

SysV init's service startup layer is built on top of this system. Each service is delivered with an init program (traditionally, a shell script) that accepts start, stop, restart, status commands as arguments and performs the appropriate PID file and signalling dance for the affected service. Each init script is assumed to correctly address the idiosyncrasies of the service it manages, and is assumed to follow the same convention as daemons themselves of returning control to the invoking process once the service has started sufficiently. Unfortunately, they're code, and they have to deal with a complex edge of the Unix ecosystem, so they're often subtly buggy in practice.

For example, Puppet, our configuration management system, self-daemonizes on startup, but fails to control its umask and working directory. This has direct effects on the reproducibility of Puppet manifests: Puppet's own environment, including things it inherited from its startup environment, cascades down into Puppet-initiated exec tasks, which in turn can leave file permissions in surprising states on disk.

Most distros provide a higher-level abstraction, implemented in /sbin/service, which attempts to normalize the environment before invoking init scripts, but

1. not everyone uses service habitually, and
2. band-aids on top of band-aids are a mess generally

New-Style Daemons and the DRY Principle

It turns out Unix already has a very sensible process control system that doesn't require a fifteen-step checklist to correctly initialize a background process. "New-style" init systems implement daemon startup in the init subsystem directly, and assume that daemon programs do not self-daemonize. This eliminates the entire checklist and ensures that daemonization and daemon management happens consistently for every service. Linux distros – including RHEL, starting in RHEL 7 – are switching to new-style init systems by default, as it's proven more reliable and simpler for developers to reason about nearly universally.

Fortunately, we don't have to wait for RHEL 7: supervisord is a meta-daemon that implements new-style daemon management while running as a classical daemon itself.

New-style daemons start up like normal, non-daemonizing programs, making them easy to run manually for development with the same command line that will be used to start their production instance. For example, gunicorn can be started as a new-style daemon by leaving out the --daemon option from its command line and config file, making it behave identically when run directly and when run as a daemon. This helps keep the production configuration similar to the development configuration, making it easier to reproduce issues and to predict how your service will behave out in the wild.