gistfile1.txt

Josh Curl is a software engineer at Rancher Labs.

Architecture of a Fully Containerized Linux Distro

Overview of RancherOS
RancherOS is the first operating system to run all system services as containers. This is done by replacing traditional init systems with Docker as PID 1. This allows RancherOS to keep a very minimal footprint; the latest ISO is only 41MB.

RancherOS runs two instances of Docker: one run as PID 1, called “System Docker”, and another instance of Docker that runs in a container, called “User Docker”. System Docker is intended to only run essential system services such as udev and NTP. The bulk of a user’s containers are expected to be run in User Docker.

Console and SSH
To log into RancherOS, SSH is run in a "console" container. By default this container is based on a very minimal Busybox-like image. The default console image can be switched out for many alternatives such as Ubuntu, Debian, and Fedora images. The concept of using a container as a console provides a lot of potential. It's even possible to build a custom console image that includes exactly the user's preferred set of tools.

RancherOS System Services
RancherOS is customized primarily through the use of “system services”, which are simply containers that run either as a one-off process during the boot processes or as a long running daemon. As an example, suppose you were running RancherOS on ESXi where you'd likely want to be running open-vm-tools,. 

RancherOS ships with a default set of system services. This set includes services that most users would expect to work out of the box, such as udev and a console. None of these default system services are inherently tied to RancherOS and so a user could remove or swap out any of these defaults. For example, if a user wanted to switch what NTP daemon was used they could simply use a different Docker image.

RancherOS Configuration
Most modern Linux distros support cloud-init, a service for initializing cloud instances. It reads a “cloud-config” file which is a fairly simple file format for performing bootstrap operations such as installing packages, writing files, or running arbitrary commands. RancherOS supports a subset of the cloud-config format and extends it with many fields specific to RancherOS.

Unlike other distros, cloud-config files plays a much more essential role in RancherOS. Rather than just playing the role of provisioning, RancherOS is fully driven by a centralized cloud-config compatible configuration format. It contains fields necessary to customize nearly every aspect of RancherOS.

Upgrading a Containerized Operating System

To discuss upgrades for RancherOS as a whole, it is first necessary to discuss how an individual service is upgraded. When a system service is started for the first time, its associated container hasn’t yet been created. As part of the creation process RancherOS will hash the configuration for the system service and store this value as a label on the container. This value acts a way to track whether a system service is in sync with its current configuration. When starting a system service after modifying its configuration RancherOS will generate another hash value and detect that it doesn’t match that of the existing container. At this point the existing container is removed and a new one is created.

As mentioned above, all system services that are included in RancherOS are determined by a default configuration file. This file is updated as part of the upgrade process. Any default system service that was modified will be recreated the next time it's started, typically on a reboot.

Where RancherOS is Headed

Flexible Customization

By providing a cloud-config file to RancherOS it's possible to fully configure which system services are run. Any Docker images that aren’t already present will be dynamically downloaded and run as part of the boot process. Although this is the workflow that most users would expect, there are many use cases where it would be useful to include assets such as Docker images along with a RancherOS ISO or other build artifact.
Inside the Container

Inside the Container

As mentioned earlier, the image used for default RancherOS system services is minimal and Busybox based. We currently make use of the buildroot packaging system to generate this image. While this has worked well so far, our long term vision includes building a minimalistic alternative this.

Although it's still in the very early phases, we're working on a package manager designed to be used exclusively within containers. Our target use cases are much smaller than traditional packages managers; rather than including operations for distribution upgrades or package removal a user would simply recreate their container. This allows us to to create packages that are less complex and earlier to maintain.

In addition, our packages are designed to be built in a fully containerized manner. A simple Dockerfile is all that is needed to build a new package. The package itself is generated by analyzing which files were changed in a particular container layer. The resulting package is then distributed entirely as container image layers. This allows standard container registries to act as a repository for packages.