Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Nixpkgs: 2014 sprint report on multiple-outputs

I worked on splitting packages into multiple outputs. Nix does support derivations with multiple outputs for a few years (I estimate), but it is little used. I focused on using it widely in nixpkgs. The main aim is to reduce runtime closure sizes by separating files not needed at runtime.

Most of the general ideas and parts of the current code were done about one and two years ago, mainly by Eelco. I updated the code wrt. stdenv changes since then and started tweaking and fixing the builds from the most deeply depended packages.

The largest savings are currently due to not having headers and documentation in runtime closures. Probably much more could be saved by turning locale data into plugins, but it's unclear how to do it, so that was not attempted (yet).

Code sharing in nix expressions is attempted as much as possible. That is, for typical package it should be enough just to define the list of outputs with standard names, e.g. outputs = [ "dev" "out" "bin" "man" "doc" ]; and the stdenv code ensures passing flags to configure/install phases, and it also moves some kinds of files if present in a wrong output. Most our base packages are built by autotools where this approach already works very well.

The splitting still needs lots of work. The progress done around the sprint can be currently found only in my repo https://github.com/vcunat/nixpkgs/compare/v/modular and it isn't very tidy, as inspecting more packages sometimes uncovered a need for changes in the general stdenv code. I plan to re-work the changes soon into commits that are better described and reviewable.

Before having such features in master, we will also have to solve some issues related to using the split packages. That doesn't seem very realistic for the planned 14.10 release. Up to now all files for a package were on one path. In the nix language one can specify a particular output in style package.bin, and not doing so means the first output, which by convention contains or propagates everything needed to build against that package. That migrates well the package input specifications, but it seems very unsuitable e.g. for systemPackages on NixOS where one often does not e.g. want headers and wants to add man pages instead. Currently, nix-env behaves differently and installs the "out" output by default, which may be missing many files that people were used to until now. Also the names now get like "xz-5.0.5-dev", which plays badly with nix-env, as we've seen, and it seems better to e.g. create names like "xz-dev-5.0.5" (that is nix issue).(Is partially wrong, as nix-env is cleverer about these than I assumed, at least with nix-1.8.)

/cc https://github.com/NixOS/nixpkgs/issues/7117

@wmertens

This comment has been minimized.

Copy link

@wmertens wmertens commented Oct 15, 2014

Unchanged text in readable format: 😁

I worked on splitting packages into multiple outputs. Nix does support derivations with multiple outputs for a few years (I estimate), but it is little used. I focused on using it widely in nixpkgs. The main aim is to reduce runtime closure sizes by separating files not needed at runtime.

Most of the general ideas and parts of the current code were done about one and two years ago, mainly by Eelco. I updated the code wrt. stdenv changes since then and started tweaking and fixing the builds from the most deeply depended packages.

The largest savings are currently due to not having headers and documentation in runtime closures. Probably much more could be saved by turning locale data into plugins, but it's unclear how to do it, so that was not attempted (yet).

Code sharing in nix expressions is attempted as much as possible. That is, for typical package it should be enough just to define the list of outputs with standard names, e.g.

outputs = [ "dev" "out" "bin" "man" "doc" ];

and the stdenv code ensures passing flags to configure/install phases, and it also moves some kinds of files if present in a wrong output. Most our base packages are built by autotools where this approach already works very well.

The splitting still needs lots of work. The progress done around the sprint can be currently found only in my repo https://github.com/vcunat/nixpkgs/compare/v/modular and it isn't very tidy, as inspecting more packages sometimes uncovered a need for changes in the general stdenv code. I plan to re-work the changes soon into commits that are better described and reviewable.

Before having such features in master, we will also have to solve some issues related to using the split packages. That doesn't seem very realistic for the planned 14.10 release. Up to now all files for a package were on one path. In the nix language one can specify a particular output in style package.bin, and not doing so means the first output, which by convention contains or propagates everything needed to build against that package. That migrates well the package input specifications, but it seems very unsuitable e.g. for systemPackages on NixOS where one often does not e.g. want headers and wants to add man pages instead. Currently, nix-env behaves differently and installs the "out" output by default, which may be missing many files that people were used to until now. Also the names now get like xz-5.0.5-dev, which plays badly with nix-env, as we've seen, and it seems better to e.g. create names like xz-dev-5.0.5 (that is nix issue).

@vcunat

This comment has been minimized.

Copy link
Owner Author

@vcunat vcunat commented Oct 15, 2014

Thanks, I didn't check it before posting; just copy-pasted from dynamically wrapped e-mail.

@copumpkin

This comment has been minimized.

Copy link

@copumpkin copumpkin commented Jan 7, 2015

Very exciting! 👍

@copumpkin

This comment has been minimized.

Copy link

@copumpkin copumpkin commented Jan 11, 2015

Is there a way to get different attributes (e.g., propagatedBuildInputs and passthrough stuff) on the separate outputs?

@vcunat

This comment has been minimized.

Copy link
Owner Author

@vcunat vcunat commented Jan 17, 2015

@copumpkin (I probably didn't get github notifications for this):

  • Propagation: our stdenv (for years) propagates whatever is in $output/nix-support/propagated-{,native-}build-inputs. However, our mkDerivation interface currently only uses a single attribute. What I did ATM was that these are propagated from the main development output only, as I haven't yet seen a different use case. When we find different use cases, we can decide what's best (it shouldn't be difficult).
  • Passthru: that's just an evaluation issue (i.e. no rebuilds need to be involved for that). AFAIK we use passthru attributes on the whole package attributes, i.e. just boost.foo and not boost.lib.foo. As above, it's probably pointless to change it before we find the current state causes problems...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment