Substitution Agents enable alternative mechanisms for binary substitution for Nix. They run independently from Nix and provide a superset of the Nix Binary Cache API. In addition to supporting endpoints that return the availability of store paths (/$storePathHash.narinfo), Agents can be requested to actually perform a substitution and make the corresponding path available in the Nix store.
The main use case is to enable on-demand, zero-copy sharing of store paths in container/VM environments while preserving flexibility. The Agent API allows hosts to share its store paths in a controlled manner without incurring extra storage costs.
Nix is becoming increasingly popular in containerized and virtualized environments. Cloud Development Environments like Gitpod.io can provide consistent developer experiences with the help of Nix, and Continuous Integration platforms can make use of Nix to create reproducible artifacts. Currently, store paths can be made available in the guest environment in one of the following ways:
- Create a pre-built image containing all required paths (pkgs.dockerTools.buildImage)
- Have the guest download them on-demand from a Binary Cache server (Cachix, Attic)
- Mount the entire host Nix store onto the guest (nixos-container, NixOS Tests)
The first two strategies are portable across platforms, but incur extra storage and/or bandwidth costs due to duplication. The third option enables "zero-copy" sharing but severely limits flexibility: Untrusted guests cannot configure their own binary caches, and inter-tenant isolation can be tricky to achieve since guests can see everything that has been built.
Achieving on-demand, zero-copy sharing of store paths is difficult due to a multitude of mechanisms (bind mounts, virtiofs, NFS, etc.) that are in use. Platforms implementing such a scheme may also have provider-specific setups that require special handling. It's clear that implementing it directly in Nix isn't a scalable approach.
To solve this issue, we propose the concept of Substitution Agents which run independently from Nix and are able to perform substitutions directly in the guest Nix store.
Nix can be configured to use a Substitution Agent, identified with an URI with the scheme agent+http:// or agent+unix://. This
When the Agent substitutes a path, it's considered Agent-managed in the Nix store database. In order to delete the path from the store or replace it with a non-Agent-managed version, nix or nix-daemon must invoke the Agent API to perform this action.
A Substitution Agent is identified with an URI with the scheme agent+http:// or agent+unix://. Since Agent-managed paths are bound to the Agent, the endpoint must always be known for store operations related to Agent-managed paths to be carried out. In the case of nix-daemon, the Agent endpoint may not be configured dynamically by the client for a single invocation.
The Substitution Agent is responsible for keeping Agent-managed paths available throughout the lifetime of the system. It may start very early in the boot process in order to make Agent-managed paths available prior to stage-2 init. It may also run externally, such as in a sidecar container. The only requirement is that it must be able to affect the system's view of the Nix store.
Since Substitution Agents can affect the view of the Nix store, access to the Agent API is root-equivalent and therefore must be restricted to nix-daemon or authorized users. This may be implemented via filesystem permissions (Unix Domain Socket) or IP restrictions and access tokens (TCP).
On the other hand, users configuring Nix to access a rouge/unapproved Substitution Agent will not give them any more access than what the Agent is able to.
Nix should include a mechanism to allow a Substitution Agent to be configured externally. This may be an environment variable or a well-known path in the filesystem.
A sample use-case is to inject the Substitution Agent endpoint as an environment variable when starting the container for a CI job.
For scenarios where required store paths are known in advance, a pre-generated script containing Agent API calls can be run to bootstrap the environment without the need for installing Nix.