This document contains some ideas for additions to the Nix language.
The Nix package manager, Nixpkgs and NixOS currently have several problems:
-
Poor discoverability of package options. Package functions have function arguments like
enableFoo
, but there is no way for the Nix UI to discover them, let alone to provide programmatic ways to change them. Likewise, there is a global configuration mechanism (config.*
) but the Nix UI doesn't know about it. -
Ad-hoc override/configuration mechanisms in Nixpkgs:
.override
.,overrideDerivation
,super: self:
, overlays,config.*
. -
Very inefficient evaluation mechanism. To get metadata about a package,
nix-env
needs to evaluate a derivation, just to be able to extract itsname
attribute and other metadata. Quite apart from being inefficient, it's not guaranteed that evaluation succeeds (e.g. a package might have anassert stdenv.isLinux
, so on macOS you can't even see the package).Also, derivations and package function need to hang on to their inputs to make
.override
and.overrideDerivation
work. This makes it impossible to garbage-collect inputs. -
The functional style of package definitions is verbose and of questionable value. That is, package functions need to specify dependencies in three places: 1) at the use site, e.g.
buildInputs = [ libfoo ]
; 2) as a function argument, e.g.{ libfoo }: ...
; and 3) at the call site, e.g.import package.nix { inherit libfoo; }
. The last one can be elided usingcallPackage
but that's an ugly hack. -
The NixOS module system is implemented as a library function, which causes oddities like the need for a
mkIf
function (due to a lack of laziness in the language when merging attribute sets). Evaluation is also pretty expensive. -
Inconsistent configuration styles between NixOS (the module system) and Nixpkgs (function arguments, and an ad hoc
config
attribute set). -
nix-env
has no notion of package plugins. For example, there is no way to installflashplayer
into thefirefox
package. Nix doesn't know thatfirefox
is a wrapper script generator that takes a list of plugins and thatflashplayer
is something that can be plugged into that list. Nixpkgs has numerous other examples of such "composable" packages, e.g. TeX (texlive.combine
) and GHC (ghcWithPackages
).Such compositions can be expressed in Nix expressions (e.g.
ghcWithPackages (pkgs: [pkgs.mtl])
), but how to do so is not discoverable except by searching the Internet or grepping the Nixpkgs sources.
It's worth noting that the Nix language is intended as a DSL for package and configuration management, but it has no notions of "packages" or "configurations".
The solution to these problems is to add a language feature named configurations to make Nix more suitable as an actual DSL for package/configuration management. Configurations are essentially an implementation of the NixOS module system as a language feature. The goal is that this provides us with a better override mechanism for packages and can replace the current module system with a more performant implementation.
A configuration is essentially a nested attribute set: it contains values (plus some metadata) indexed by an attribute path. Configurations are composed out of one or module configuration modules. A configuration module is a syntactic construct that vaguely looks like a recursive attribute set using angle brackets:
module1 = <
foo = 123;
bar = true;
a.b.c = if bar then foo else foo * 2;
>;
This configuration module by itself describes a configuration with
three options named foo
, bar
and a.b.c
. Selecting an attribute
from a configuration module causes it to be evaluated into a
configuration (which is an opaque value, not an attribute set!). E.g.
module1.a.b.c => 123
Note that option values can refer to other options in the
configuration, so the value of a.b.c.
can depend of the values of
foo
and bar
. (In contrast to the NixOS module system, you don't
have to write config.
in front of every option use.)
Configuration modules can be composed with other configuration modules to create more complex configurations. This is done by extending another module. For example:
module2 = <
extends module1;
bar = false;
>;
Now,
module2.a.b.c => 246
Thus, modules can override the values of options declared in another module.
TBD: the alternative is this quasi-functional style:
module2 = module1 <
bar = false;
>;
i.e. a configuration is extended by applying a module to it. This is
particularly nice in conjunction with derivation builders (see
below). This also allows extending a configuration with a plain old
attrset, e.g. module2 = module1 { bar = false; }
.
A crucial property here is that forward references are not allowed. That is, the option values in a particular module cannot reference options that have not been declared in that module or one of its ancestors (i.e. the modules it extends). For example, this doesn't work:
module3 = <
foo = bar;
>;
module4 = <
extends module3;
bar = 1;
>;
Here the evaluation of module3
will fail because there is no bar
in scope. This is in contrast to the NixOS module system, which allows
any module to reference any other option, which makes it hard to
reason about the dependencies of a module.
A few restrictions: option names cannot be computed dynamically. For example, this is not allowed:
bad1 = <
foo.${bar} = ...;
>;
Also, options cannot be a prefix of a previously defined option, as in
bad2 = <
extends module1;
a.b = { c = 100; }; # wrong
a.b.c = 100; # right
>;
[Annotations? Attributes?]
Configuration options, like NixOS options, are not just values. They can have some metadata such as documentation, a type, default values etc. The syntax of an option is as follows:
<ATTR-PATH> (| <FIELD-NAME> <EXPR>?)* (= <EXPR)?;
For example:
module5 = <
networking.firewall.enable
| doc "Whether to enable the firewall."
| type types.bool
| default false;
networking.firewall.allowedTCPPorts
| doc "TCP ports to be opened in the firewall."
| type types.list types.int
= [];
>;
TODO: syntax bikeshedding.
The following fields are supported:
-
value
: An option value at priority level 100.= <X>
is sugar for | value `. -
default
: An option value at priority level 1500. Thus defaults are ignored (not merged) when there are any higher-priority values. -
doc
: A description of the option in some sensible format. -
example
: An example value. May be repeated. Unlike in the NixOS module system, there is noliteralExample
because the parser can store the source text of the example field. (The example needs to be a syntactically validExpr
, but it doesn't need to evaluate.) -
type
: The type of the option. This should be a configuration withmerge
andcheck
members. TODO: to support future incremental evaluation, maybemerge
should be a fold, rather than a function that takes a list of all option values. -
if
: A Boolean value that determines whether the option value is to be used, e.g.< environment.systemPackages | if enableFoo = [ foo ]; >
-
prio
: The numerical priority of the value. When computing the final value of the option, all values that have a lower priority that the highest-priority value(s) are discarded. -
label
,before
andafter
: A string value used to topologically sort option values prior to merging. The NixOS module system uses a numeric value for this, but strings are nicer for things like ordering of the activation script:< activationScript | label "users-groups" = "... populate /etc/{passwd, group} ... "; # This activation script fragment needs to run after users/groups # have been created. activationScript | after "users-groups" = "install -d -o fnord -g /var/foo"; >
-
final
: Specifies that the value of the option cannot be changed by later modules. -
scope
: An attrset added to the lexical scope of the evaluation of values of this option, equivalent to puttingwith <scope>;
around every value. Typical use case: allowing differentpkgs
sets to be in scope for different options for cross-compilation (e.g. you can have optionshostBuildInputs
andnativeBuildInputs
with different package sets in scope).
Any other field name gives a syntax error. Only the initial definition
of an option can specify doc
, type
and example
.
Note: there is no merge
field. If you want another merge
policy
you should adapt the type.
TODO: flip priorities? "Higher priority" is ambiguous.
Probably it's also nice to have some global fields for a configuration module. E.g.
-
doc
: A description of the configuration. -
globalScope
: Likescope
, but for all options in this configuration. Typical use case would beglobalScope = pkgs
. -
fieldScope
: Likescope
, but for the evaluation of option fields other than values/defaults. For example,fieldScope = types
would allow getting rid oftypes.
in option types.
Configuration modules could have an if
construct to enable/disable a
group of values more concisely. This is similar to mkIf
in the NixOS
module system, which pushes the conditional down into the options.
<
enableFoo
| type bool
= false;
if enableFoo {
environment.systemPackages = [ foo ];
systemd.units.foo = ...;
};
# The above is equivalent to:
environment.systemPackages | if enableFoo = [ foo ];
systemd.units | if enableFoo = { foo = ...; };
>
However, this might not be very useful if we get rid of most enable
options (see below).
Currently Nix uses functions as the mechanism to abstract over common
derivation patterns. For example, stdenv.mkDerivation
,
buildPythonPackage
and fetchurl
are all functions that take some
inputs and return a derivation. The problem with this approach is that
the act of applying a function consumes its inputs, leaving us without
the ability to override inputs or to query the inputs. This led to
hacks like .override
, .overrideDerivation
, passthru
, meta
and
so on. Also, functions lack a documentation mechanism.
We can replace such functions with configuration modules that produce
a drv
output option. For example, here is a very bare-bones
derivation configuration module:
builders.derivation = <
# Interface
name
| doc "Name of the derivation, used in the Nix store path."
| type types.str
| example "openssl";
version
| doc "Version of the derivation, used in the Nix store path."
| type types.str
| example "1.0.2"
= "";
builder
| doc "Command to be executed to build the derivation."
| type types.path
| example "${bash}/bin/sh";
args
| doc "Arguments passed to the builder."
| type types.unique (types.list types.str)
= [];
outputs
| doc "Symbolic names of the outputs of this derivation."
| type types.list types.str
= "out";
env
| doc "Structured values passed to the builder."
| type types.attrsOf types.str
= { inherit outputs; };
# Implementation
drv
| doc "The resulting store derivation."
| final
= builtins.derivation ({
name = "${name}-${version}";
inherit builder args;
} // env);
>;
We can then build other abstractions on top of each other:
# This is basically stdenv.
builders.generic = <
extends builders.derivation;
# Interface
buildInputs
| doc "Dependencies of this derivation."
| type types.list types.package # FIXME
= [];
phases
| doc "Names of build phases."
| type types.list types.str
| example ["build" "install"];
# Implementation
builder = "${bash}/bin/bash";
args = [ "-c" stdenv/generic/setup.sh ];
env = { inherit phases buildInputs; };
...
>;
# A package is a derivation that can be installed by nix-env.
builder.package = <
description
| doc "A short (one-line) description of the package."
| type types.str;
homepage = ...;
longDescription = ...;
supportedPlatforms
| doc "List of platforms on which this package is supported."
| type types.list types.platform;
# nix-env won't show this package if `supported` evaluates to false.
supported
| doc "Whether the package is supported on the target platform."
| type types.bool
= elem system supportedPlatforms;
# Nix will refuse to build this package if `enabled` evaluates to false.
enabled
| doc "Whether the package can be built."
| type types.bool
= (free || config.allowUnfree) && supported;
>;
# Global Nixpkgs configuration.
config = <
allowUnfree
| doc "Whether to allow unfree software to be installed."
| type types.bool
= false;
>;
builders.unixPackage = <
extends builders.generic builders.package;
src
| doc "The source tarball of the package."
| type types.path;
configureFlags
| doc "Flags passed to the package's configure script."
| type types.list types.str
= [];
# Create some phases.
phases | label "unpack" = ["unpack"];
phases | label "configure" after "unpack" = ["configure"];
phases | label "build" after "configure" = ["build"];
phases | label "install" after "build" = ["install"];
configurePhase
| doc "Shell code to configure the package."
| type type.lines
= "./configure --prefix=${placeholder out} $configureFlags";
buildPhase
| doc "Shell code to build the package."
| type type.lines
= "make";
installPhase
| doc "Shell code to build the package."
| type type.lines
= "make install";
env = { inherit configurePhase ...; };
>;
# An actual package.
pkgs.hello = <
extends builders.unixPackage;
name = "hello";
version = "1.12";
description = "A program that produces a familiar, friendly greeting";
license = licenses.gpl;
enableGUI
| doc "Enable GTK+ support."
| type types.bool
| default false;
# This uses the functional notation for extending a configuration:
src = builders.fetchurl { url = ...; sha256 = ...; };
buildInputs = if enableGUI then [ gtk ] else [];
# or equivalently:
buildInputs | if enableGUI = gtk;
>;
Since configuration values are computed lazily, a tool like nix-env -qa
only needs to evaluate attributes like name
and enabled
. It
doesn't need to evaluate drv
.
# To override the source of Hello.
pkgs.hello = <
src.url = http://example.org/my-hello.tar.gz;
# Note: this is sugar for:
src = < url = http://example.org/my-hello.tar.gz; >;
# I.e. it adds a module to the configuration 'src'.
>;
# To enable a feature in Hello.
pkgs.hello = <
enableGUI = true;
>;
# To add something to installPhase.
pkgs.hello = <
installPhase | after default = "mv $out/bin/hello $out/bin/my-hello";
>;
# To change a dependency of Hello in Hello *only*.
pkgs.hello = <
targetPackages = < extends pkgs; gtk = < src = ./my-gtk.tar.gz; >; >;
>;
# To replace Hello entirely. FIXME
pkgs.hello | prio 0 = pkgs.my-hello;
pkgs.my-hello = < ... >;
TODO
pkgs.firefox = <
extends builders.unixPackage;
plugins
| doc = "Set of enabled plugins."
| type = subsetOf pkgs.mozillaPlugins;
# this type should implicitly set `scope` to the given attrset
= [];
installPhase = "... generate firefox wrapper script ...";
>;
pkgs.mozillaPlugins.flashplayer = < ... >;
To enable:
pkgs.firefox = <
plugins = [ flashplayer ];
>;
Having explicit package options and attribute annotations allows nix
to show and modify options. E.g.
$ nix query-package nixpkgs.firefox
Available options:
- enablePulseAudio: Enable sound via PulseAudio.
Default value: true
Current value: N/A
- enableOfficialBranding
...
- plugins: Set of enabled plugins.
Possible values:
nixpkgs.flashplayer
nixpkgs.google-talk
...
$ nix install nixpkgs.firefox --with enableOfficialBranding --without enablePulseAudio
$ nix modify-package nixpkgs.firefox --with enablePulseAudio --add plugins nixpkgs.flashplayer
TBD: we probably don't want to show only end-user options like
enablePulseAudio
by default, rather than every option inherited from
builders like buildInputs
or name
. So we need some way to specify
which options should be shown in what context (like the internal
flag in the current NixOS module system).
TODO: how to hook everything up.
At top-level, Nixpkgs should evaluate to a set (or configuration?) of configurations:
{
lib = <
...;
>;
builders = <
derivation = ...;
generic = ...;
package = ...;
unixPackage = ...;
fetchurl = ...;
...;
>;
pkgs = <
glibc = import .../glibc;
hello = import .../hello;
>;
# NixOS modules
modules = <
etc = import modules/.../etc.nix;
...;
top-level = import modules/.../top-level.nix;
...
# The minimal NixOS system.
base = < extends etc top-level initrd kernel ...; >;
# Non-default stuff.
kde = import modules/.../kde.nix;
>;
}
We should no longer include every NixOS module by default. Instead of
using enable
options, you just inherit NixOS modules into your
system configuration, e.g.
myConfig = <
extends modules.base modules.kde modules.nginx ...;
...
>;
In the NixOS Of The Future, most modules shouldn't be top-level configuration modules, since those 1) violate POLA: you don't want the PostgreSQL module to have the ability to change your X11 configuration; and 2) are not functional: you can't easily instantiate a module multiple times, e.g. to have multiple PostgreSQL instances.
Instead most NixOS modules should be implemented as extensions of
smaller configuration types. For example, builders.service
allows
the construction of a service that declares some systemd unit, and
only has access to (say) /var/services/<service-name>
and the
closure of the unit in the Nix store. (This is essentially a
light-weight container.)
builders.service = <
name
| doc "The identifier of the service."
| type str;
systemdUnits
| doc "The systemd units that implement this service."
| type attrsOf /* instances of builders.systemUnit */
= [];
...
>;
So PostgreSQL can be extension of this type:
services.postgresql = builders.service <
# Interface
port
| doc "TCP port on which the server listens."
| type int
= 5432;
# Implementation
systemdUnits =
... a systemd service that runs postgresql, using
/var/services/<service-name> as the data directory ...;
>;
NixOS at top-level has a systemServices
option that allows services
to be hooked into the system:
modules.services = <
systemServices
| doc "Set of isolated system services"
| type attrsOf /* instances of builders.service */;
;
systemd.units =
... the concatenation of the systemdUnits values of all systemServices ..;
>;
A NixOS configuration can then instantiate multiple PostgreSQL instances:
<
systemServices.postgresql-prod = services.postgresql < >;
systemServices.postgresql-test = services.postgresql < port = 12345; >;
>
The idea is to provide an include mechanism as an alternative to
import
. The expression include ./foo.nix
is equivalent to
replacing the expression with the contents of ./foo.nix
. Thus
foo.nix
has access to the lexical scope at the site where it is
included.
This means that we can write all-packages.nix
as a long list of
inputs
:
<
pkgs = <
git = include ./git.nix;
openssl = include ./openssl.nix;
...
>;
>
where the package expressions look like
builders.unixPackage <
name = "git";
version = "2.3.4";
...
buildInputs = [ openssl ];
>
That is, git.nix
is no longer a function that takes builders
as an
argument. Rather it just assumes that builders
exists in the lexical
scope, thus preventing the need for ugly hacks like
callPackage
. This might seem like a radical departure from how we
expressed dependencies in Nixpkgs, but in fact it's already widely in
use, see e.g. perl-packages.nix
or python-packages.nix
where most
dependencies are obtained from the surrounding rec
set. include
simply makes it possible to split such package sets into multiple
files.
Note: The include mechanism must be lazy, i.e., the inclusion
shouldn't be done eagerly while parsing the including file, since we
don't want all-packages.nix
to eagerly all its referenced files.
Note: This is not a token-based include mechanism like C's
#include
. That is, include E
is only valid as an expression
(i.e. the grammar will have a non-terminal expr: INCLUDE expr
), and
the contents of the included file is parsed as an expr
. Thus you
cannot write something like
{ x = 123;
include ./other-attrs.nix
}
TODO: this may not be needed since option values are evaluated in a
dynamic scope anyway. E.g. in the example above openssl
doesn't have
to be provided as a function argument anyway because it doesn't refer
to the lexical scope but to the scope
set by the buildInputs
option. However, builders
does need to come from somewhere, and it
would be annoying if we had to change the whole file into a function
just to pass in builders
. OTOH, if we used the extends
syntax
instead, then maybe this could be avoided: its argument could be
evaluated in a dynamic scope.
Hopefully Nix configurations can lead to less CPU and memory-hungry Nixpkgs/NixOS evaluation. There are some aspects to this:
-
Configuration values should be implemented internally by keeping weak pointers to option values. Thus option values can be garbage collected (and re-evaluated from the saved thunk if necessary). Of course, this is a risky strategy because the thunk might be bigger than the WHNF. So if an option value is "small" (like an integer or small string) it makes sense to overwrite the thunk by a non-weak pointer to the resulting value.
-
Getting rid of pass-thru derivation attributes like
.override
,.overrideDerivation
,.meta
etc. means that after evaluating adrv
option value of a package configuration, we can garbage-collect the entire package configuration (assuming nothing else is keeping it alive). -
Not including every NixOS module by default should save a huge amount of time and memory.
-
Doing the module system in C++ rather than in a slow purely functional language should give a big speedup.
Can we make NixOS module evaluation incremental? E.g. in a NixOps module with multiple machines that share a base configuration:
myBase = <
extends modules.nixos modules.kde ...;
networking.firewall.allowedTCPPorts = [ 1234 ];
>;
machine1 = myBase < networking.hostname = "machine1"; >;
machine2 = myBase < networking.hostname = "machine2"; >;
...
it would be nice if the evaluation of the configurations of machine1
and machine2
only needs to recompute the options that are affected
by the change to networking.hostname
.
The self: super:
style has a self
attribute to refer to the final
set, and super
attribute to refer to the previous set in the
chain. (It should really be named final: previous:
.) In
configuration modules, any reference is to the final configuration, so
no special notation for referring to self
is required. But do we
need an equivalent for super
?
The typical use case is to extend a previous value, e.g.
buildInputs = super.buildInputs ++ [ libfoo ];
However, merge functions remove the need for this: we just concatenate
all definitions of buildInputs
, just as is done in the NixOS module
system, i.e.
buildInputs = [ libfoo ];
If we want to throw away previous definitions, we do:
buildInputs override = [ libfoo ];
(This corresponds with mkForce in the module system.)
The experience with the NixOS module system suggests that this is
sufficient and we don't need a super
keyword.
Historically, Nixpkgs has been full of builder functions that accept
an open set of arguments (...
), e.g. stdenv.mkDerivation
. This is
bad for error checking since a typo in an argument won't be
detected. Thus, configuration modules like builders.package
or
builders.generic
have a closed set of inputs in the sense that they
cannot refer to options that haven't been declared yet. Also,
builders.derivation
doesn't pass through arbitrary arguments to the
builder environment; instead the environment has to be specified
explicitly (by adding values to the env
option).
This should be fine because the use of environment variables really
predates the introduction of string interpolation. Originally, if
(say) buildPhase
needed access to some variable foo
, we would
write
derivation {
inherit foo;
buildPhase = "... $foo ... ";
}
But nowadays we can just write
<
buildPhase = "... ${foo} ... ";
>
TODO
Until there is a way to reflect into expression and tell which inputs it uses, it is useful to specify things twice. Really, if I could hower cursor over package definition in IDE and see it's "inputs" (source inputs, build-time inputs, runtime inputs, build-time inputs inherited from mkDerivation, any other level-2 dependecies) separated into groups, it would be OK to specify things once.