flox lock

flox-lock.md

Constrain build inputs for more reproducible builds

!!! summary This section shows all the ways all the ways your packages' dependencies can be fixed to specific versions.

It may happen at times when a flox expression gets rebuilt (either manuallyT or triggered by flox CIT), the resulting application may exhibit unexpected behaviour (e.g., crash on start, missing features). The reason is that sources and their dependencies, depending the way they have been declared, may not be the same during sub-sequent build jobs.

The solution is to constrain build inputTs:

• indirectly, by locking flox channelsT using flox lock

• directly, by pinning them in the flox build expressionT itself

When to use constraints?

The flox defaultsT use the latest versions of the build inputs whenever the flox build expressionTs are rebuilt, and they are fine in most cases; such as when packaging

TODO Add "flox defaults" (flox buildersT, out of the box staging/stable jobsets, implicit channel dependencies whose packages are pulled into scope - see flox-lib docs, and will link clickup task) to Glossary

TODO Bring examples from Michael's limeytexan user (better yet, clone the ones we needinto flox-examples and link those

• very simple projects (TODO see the ptree wrapper and the corresponding ptree/dfault.nix; the limeytexan flox channel only has staging and stable jobsets, hence no flox lock)

• a stable release of an application external to your GitHub user or organization (perhaps because it is not available in NixpkgsT yet) (TODO xmahjongg/default.nix)

Possibly the only argument against using constraints is that one will have to manually "roll forward" the build input(s) when a new version becomes available.

There are scenarios though when this lack of convenience is dwarfed by the need for a higher degree of determinism, such as when

• developing an application, and the dependencies have to be within certain ranges

• setting up integration testing for a project (see table below TODO create table and link to section)

• the currently packaged version of an application is missing features/behaviour that earlier/later versions possess (and can't use overlays, or they are not preferred)

• maximalizing reproducibility is preferred on principle

Ways to fix build inputs

Pinning individual sources and dependencies

The term "pinning" refers to declaring a source or dependency with the snapshot of the source code at a particular point in time using Nix fetchers.

TODO refer to xmahjongg/default.nix again as example

TODO This is not flox-specific topic, so leaving this section short but including a list would be a nice touch as there are a plethora of blog posts and external tools (niv, nix-wrangle).

Pros & cons

Pros:

• sources and dependencies can be fixed on a per package basis, providing the most granular control over them

• the version of the build inputs are visible in plain sight inside each flox build expression

Cons:

• can be tedious and verbose if there are many dependencies

Locking flox channels

Locking a channel will freeze the versions of every package in it at a point in time, and flox lock lets you manage the locks on your dependent channelsT.

??? info Technically, locking takes a snapshot of all the source code of all branches of all repos (including the floxpkgs repo with all its flox build expressions) in the GitHub organization/user that the channel is based on and whose packages are populated from the build results of that floxpkgs repo.

Why channels need to be locked at all?

When you refer to a package in your Flox expressions, its version will be determined whether there is a lock on the dependent channelT that contains it:

TODO: analogy: cross section of a tree trunk, river bed, etc.; illustration: show simple animation that uses "cut-off" timestamp as a horizontal sliding axis perpendicular to the channels horizontal plane showing columns of packages with discrete version numbers. (Challenge: make it interactive)

TODO2: illustration: represent github repos with towering columns (1 branch = 1 column) at different heights (where height is determined by the number of commits) then a straight line (or a flat plane, if 3D) that cuts through everything a the same height

• If there is no lock, then the package with the latest version will be used from the channel whenever the flox build expression is evaluatedT (i.e., built)*. As a consequence, it is possible that build processes started at various times will refer to different versions of the same build input.

* TODO verify this statement: I think the distinction between "flox expression" and "flox build expression" is important here, because when the latter is evaluated, the referred package's version gets resolved, a derivation is made, a package is built, etc. whereas in the former case, nothing happens: the referred package will just sit in the expression because Nix is lazy, and there is no point in going through the hassle of resolving versions if it is not used in any meaningful way in the first place.

• If the channel is locked, then the package version will be resolved to the latest one that was in effect at the lock's date and time, that will result in a constant value during each build process started after the application of the lock.

TODO Belabor the point by making a table from the points above and by illustrate the cases (animation)

Pros & cons

Pros:

• many sources and dependencies coming from the same channel (or a small number of channels)

• build inputs are fixed on a per channel basis, which can be more convenient if many of them are coming from a small number of channels

Cons:

• it is not apparent what versions of the build inputs are used; one would have to go through the .lock file to see which channels are locked at what time, and would have to compile a list of versions by visiting the referred repos inside the channels' GitHub organization/user at the specific timestamp

• it is possible that some of the required versions are after the the chosen date, but the "cut-off" timestamp can't be moved because then the versions of other build inputs may change too.

Best practices when using locking and pinning together

TODO create table; tend to incoming link from above TODO

--- scratchpad ---

The next sections explain the different kinds of build inputTs, how these affect the build process, and what can be done to constrain them in order to achieve more deterministic builds.

TODO The best way would be to create and link a doc explaining the build process through all flox build pathsT - the main issue is that the prerequisites to understanding these topics are circular.

Build inputs

Build inputTs of a flox build expressionT are

• sourcesT,
• their dependenciesT, and
• controlsT.

Depending on the way they are declared in the expression, build inputs can be

• fixed or
• variable

!!! warning "The efficacy of constraining build inputs depends on the method used (using flox lock or pinning) and the flox build pathsT taken"

Fixed build inputs

The build input is said to be fixed if its contents do not change over time, unless manually updated. Putting it another way, a fixed build input would have the same hash value during sub-sequent build jobs (i.e., whenever the expression is evaluated, producing a derivation).

Variable build inputs

For the majority of the time, the latter are responsible for the differences in sub-sequent build results.

When an application is built from a flox expressionT using the defaults (i.e., flox builders and unlocked channels), there's a chance of it exhibiting unexpected behaviour (e.g., crash on start, missing features), as the versions of its source(s) and dependencies my have changed since the last compilation.

Use cases

It is important to emphasize that going with the flox defaults is completely safe: they have been chosen sensibly to make everyday tasks more convenient but still safe.

One example is using stable packages from the Nixpkgs channel (i.e., nixpkgs.stable.<PKG_NAME>) as [build input] in your [flox expression]; if any of these are updated, your project [will be automatically re-built](TODO:link to flox CI)

Possibly the only argument against using constraints is that one will have to manually "roll forward" the flox expression (i.e., update the references to the [build input]s) when a new version of an application becomes available.

There are scenarios though when this lack of convenience becomes a crucial feature:

• developing a application
• integration testing
• use a version of an application that has features and/or behaviour missing from later versions
• maximal determinism is preferred on principle

Rationale

variable and constant inputs not-idempotent

flox expressions use variable inputs which may not be suitable in more advanced use cases, such as when developing/packaging an application whose correct operation depends on exact versions of its dependencies.

What are variable inputs, and how can they become a problem?

It is important to emphasize that

Depending on when the build process (i.e., Hydra evaluation) took place,

will yield slightly different results each time.

Some flox expressions (TODO: add and link to glossary if term is accepted) may need to be augmented to fit more advanced use cases, such as developing/packaging an application whose correct operation depends on exact versions of its dependencies.

COMMENT: flox expression - a Nix build expression that resides in the floxpkgs directory

flox expressions using flox builders

maintain stable environments (for example, for prduction, development, testing), to release your products, or just for everyday work without unwelcome surprises.

explains how custom jobsets can help to assert more control over building one's projects, and how to create jobsets in a channel with the flox lock CLI command.

??? info "Reminder: channels and jobsets" + A flox channel is a grouping of software available for installation through the web interface and the floxpm CLI, and each channel corresponds to a GitHub organization. (Read more in flox Channels.)

+ [TODO:verify] **Jobsets** in a channel allow flox to build packages against multiple versions of dependencies simultaneously, and a jobset almost always corresponds to a branch of the user's Floxpkgs repo. Each jobset comprises a set of Nix build expressions (or jobs) that will get evaluated automatically when commits are pushed either to your channel's floxpkgs repo or to one of the channel's repos that have the [flox webhooks installed](../tour/tour-4.md#enable-flox-webhooks-github-app). (Read more in the [Glossary](../how-it-works/glossary/#jobset-hydra-jobset).)

Rolling jobsets

Upon creation, each channel gets two initial jobsets: stable and staging, which are rolling jobsets. This means that their input channels are not locked (see below), and when an evaluation is triggered, Hydra will build every package in every jobset in a channel using the latest versions of the channel packages' dependencies. ??? question "Reminder: When will a Hydra evaluation get triggered?" When commits are pushed either to a channel's floxpkgs repo or to one of the channel's repos that have the flox webhooks installed.

[TODO:gif]

This behaviour is not always desirable though. Imagine developing a product, pushing commits frequently, and suddenly your app built by Hydra starts crashing, even though the latest changes couldn't have been the cause. It is more than possible that the some of the packages your project has been relying on have been updated in the meantime, and when your commit triggered an evaluation, these updated dependencies with incompatible changes were pulled in. This would also explain why the project would still run in your dev environment, where developers have tighter controls around dependencies.

Jobsets with locked channels

You can create (and manage) jobsets with similar strict guarantees using flox lock that can freeze the channels of dependent packages. When these jobsets are evaluated, Hydra will pull in the dependencies in the locked channels at a specific date and time; there may be newer version(s) of these packages available, but these will be ignored.

[TODO:gif] (similar to the above figures but now the locked channel pkgs won't change)

To revisit the example above, this time around, when you product is built, Hydra will only pull in dependencies with specific versions, regardless whether those channels have been updated or not.

!!! warning "Locks only work on the channel-level" Locks work by specifing a date and time to apply from which to extract and freeze the contents of the channels that should be locked. Therefore, it may still be a challenge to get versions right if your package has multiple dependencies from the same channel (e.g., from your channel).

Create and manage jobsets

[TODO: do these facts still stand? Form the flox lock proposal:]

The set of channels which the user can decide to lock comprises:

• Nixpkgs
• The focal channel
• Other input channels (channels containing at least one package that's an input to at least one package built in the focal channel) The set of input channels will not include the flox channel itself.

After engaging with the flox lock interface to make these choices, the user must flox commit and flox push to create their new jobset. From that point on, flox CI will evaluate their new jobset (i.e. build the channels packages with the dependencies defined by the jobset) whenever anyone pushes a change to the channel's floxpkgs repo or one of the channel's repos.

Using flox lock

[TODO: The 3.2 User interface seems more restrictive than what Michael sent me last week, and they also differ in syntax; will wait until I can try it out.]

---

scratchpad

https://app.clickup.com/t/18c0ukj

TODO: On the 2nd image there is a remard starting with "This is possible because (?)" which is probably untrue/misleading. Couldn't express my hunch clearly, and now that I think about it, it may even be stupid.