This is a response to a tweet by @jordwalke asking me about Nix and PackageJsonForCompilers. Twitter is too short form to answer well, so I wrote up some thoughts here. I might turn this into a blog post at some point.
Nix is a package manager, but also a meta-build system and deployment system.
It works based off a global
/nix/store/ folder where everything is stored keyed by a hash of its description and the
description of all its dependencies. This way you can have multiple versions of any package, and not only that different copies of the same version compiled against different versions of its dependencies or with different compile flags. All dependencies must be fully specified and the way the building works enforces this completely. This way completed binaries can be served from a "cache" with no issue. This gives the best of source-based package managers and binary ones.
The way it handles build vs. runtime dependencies is automatic. The "derivation" for a package is itself a thing that gets built into the Nix store which has dependencies which are all the build-time dependencies of the package. This is then used to build the result (with a temporary artifacts directory that is deleted after, but can be kept). The result is then scanned for the md5 hashes in the paths of the build dependencies, anything that shows up is a runtime dependency. These derivations are created from "Nix expressions" which is just a nice small language for writing configs and defining packages.
Does it have all of PJC's advantages? Yes.
Some things takes from the PJC readme:
Yes Nix can do this, and if the underlying builder doesn't support it, it can transparently farm builds out to a machine of the right type while still running commands on your machine.
Nix can use these since the file paths are deterministic, always
Parallelize builds. Recover builds.
Nix knows all dependencies and auto-parallelizes. There's a preference to make it save build artifact folders.
Export your entire project to a Makefile or shell script that can run on any network isolated machine, even if that machine does not have node/npm/opam/crate installed etc.
Nix can't do this but it can export the transitive closure of dependencies of anything, either source or result to another machine. That other machine must have Nix but Nix runs on many OSs and is light. Once you have the closure you don't need the internet to build. Or you can just ship the closure of the build product instead of the source.
pjc is not a build system for any particular compiler. It is a "meta build system", that helps all your individual pacakges' build systems work in harmony together, and then exposes an approachable human interface to that process.
Same for Nix. It already has setups for building and using the package ecosystems of C++, Node, Ruby, Rust, OCaml, Emacs, Python and some more languages.
pjc is not a package manager. It only requires that package sources be located entirely on disk before it begins working. It follows the npm directory structure convention, but this isn't central to pjc. What is central is that pjc tollerate multiple versions of packages existing simultaneously.
Nix is similar. It can fetch packages from the internet, but you have to specify the exact hash of the download result. There are tools like
npm2nix that turn
package.json files into Nix specifications of the correct versions and packages to download and build. Nix requires that you specify all versions exactly, but these tools can use solvers to generate Nix files that specify versions that should work together.
Package managers like yarn could make use of pjc packages' package.json fields in order to perform a more optimal installation (deduping more optimally).
Maybe package managers could do this with the same information from Nix too?
pjc build traverses the dependency graph and automatically runs each package.json's build scripts with a perfectly constructed environment. Each package build will see the right environment variables. Among others, it will see a PATH augmented with binaries built by its dependencies, along with any other variables its depenencies want it to see.
Nix makes sure to isolate things so that they can only use dependencies that are specified properly, so you never mess up. It passes information through environment variables to build scripts.
pjc also automatically prepares an ocamlfind directory structure lib/bin/doc for each depdendency to install itself into. pjc tells you where this is located by setting another special environment variable. (It also generates an ocamlfind.conf).
Nix doesn't do this, but I think you could write a Nix expression that could.
Builds out of source, installs out of source.
Nix can build out of source if you tell it, or use a "binary cache" if you specify one. This is great and symlinking still works.
Allows cleaning up of build artifacts trivially, merely by deleting a single _build directory.
Nix does this by default, but you can override that.
pjc can be invoked with arbitrary shell commands
The way development works with Nix is that in a repo you have a
default.nix file which describes the dependencies and how to build the project. In that folder you can run
nix shell, this transparently install the dependencies if you don't have them, and puts them in the environment. It also defines some commands that you can use to compile the project the same way Nix would if you asked it to build
default.nix. You can now run arbitrary shell commands, do dev work, and build it by running the build tool directly or through Nix.
Things Nix has that PJC doesn't.
- A global cache: I saw this mentioned on Twitter but I don't see how it would work properly based on the spec. How do you plan on separating builds of the same package against different versions of the dependencies and with different compile flags?
- Binary downloads: It's all well and good to always build from source for small projects, but it's really nice when you scale up to be able to install things super quickly, while still being able to customize the build and source if you want.
- Enforcement: Nix tries really hard to make it difficult to screw up. Your things will be deterministic, nearly guaranteed.
- It exists right now: Nix already exists and has thousands of things packaged for it and works with many language ecosystems.
- An OS: NixOS extends Nix to OS configuration and allows you to really easily deploy software packaged with Nix in a deterministic way. Way better than Chef/Puppet.
- CI and remote builds: Nix can build things transparently on remote machines. It also has a CI server (Hydra) which can distribute builds to a cluster, do tests and act as a binary cache. Imagine each unique version/configuration of your code being built only once ever across the entire company on any machine.
I think PJC has a lot of great ideas, but Nix already does a lot of the things that it wants to do, today.
One way to harness this is to implement PJC on Nix, I think the first implementation should do so to leverage the existing ecosystem to the fullest extent and make things easier.
Or, you can go all in and reimagine PJC as a utility to work with Nix, discarding the parts of the spec that are easy to do directly with Nix, and focusing on being an easy workflow with
nix for building packages in development. Nix has a really great story for production builds, but incremental development workflow is where it could use some scripts to make everything tie together nicely.
esydoes have a global cache and takes into account the hash of resolved dependencies in order to compute the hash of the build package.
Our thinking was that the way we cache allows instant downloads simply as a special case of the global cache. If you can just write a tool that pre-floods the local cache with the right artifacts ahead of time, then "building" is instant. It's all about getting the cache keys right.
esycurrently scrubs environment variables and isolates package builds pretty well, but I'm sure there's a ton of edge cases we're missing that nix already handles.