Skip to content

Instantly share code, notes, and snippets.

@fzakaria
Created September 20, 2026 17:58
Show Gist options
  • Select an option

  • Save fzakaria/e7914976f9c947be22baa0703908bf28 to your computer and use it in GitHub Desktop.

Select an option

Save fzakaria/e7914976f9c947be22baa0703908bf28 to your computer and use it in GitHub Desktop.
Dynamic derivation whose chain closes into a cycle crashes the scheduler: `Assertion '!awake.empty()' failed` in `Worker::run`

Title

Dynamic derivation whose chain closes into a cycle crashes the scheduler: Assertion '!awake.empty()' failed in Worker::run

Body

Describe the bug

A derivation can emit a derivation whose inputDrvs carries a dynamicOutputs entry. If that entry names the derivation that is currently producing it, the build graph has a cycle. Nix does not report a cycle. It runs out of runnable goals and trips an assertion in the scheduler:

terminating due to unexpected unrecoverable internal error:
Assertion '!awake.empty()' failed in void nix::Worker::run(const nix::Goals&) at ../build/worker.cc:363
Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:

std::terminate() called without exception
Stack trace:
 0# 0x000000000011DFEF in nix
 1# 0x00000000000C533A in .../gcc-15.3.0-lib/lib/libstdc++.so.6
 2# std::unexpected() in .../gcc-15.3.0-lib/lib/libstdc++.so.6
 3# 0x000000000010203F in .../nix-util-2.35.2/lib/libnixutil.so.2.35.2
 4# __wrap___assert_fail in .../nix-store-2.35.2/lib/libnixstore.so.2.35.2

The process aborts and dumps core. It is not a hang; it fails within a second of the second builder finishing.

Reaching the cycle does not require trying to. I hit it while writing a chain of dynamic derivations that generates its successor at build time. Each generation has to be a distinct derivation or the chain closes on itself — in my case the generations were distinguished only by a depth counter in the environment, and dropping that counter turns a well-founded chain into this crash. I would expect a cycle error, or at worst a scheduler deadlock message, rather than an assertion failure.

Steps to reproduce

Two files, stock nixpkgs, no pinned store paths.

cycle.nix
let
  pkgs = import <nixpkgs> { };
  # Must be the same Nix you are running the build with.
  nixPkg = pkgs.nixVersions.latest;
  root = derivation {
    name = "root.drv";
    system = "x86_64-linux";
    builder = "${pkgs.bash}/bin/bash";
    args = [ "-e" ./cycle.sh ];
    BASHPKG = "${pkgs.bash}";
    COREUTILS = "${pkgs.coreutils}";
    NIXBIN = "${nixPkg}";
    STEP = "${./cycle.sh}";
    PATH = "${pkgs.coreutils}/bin";
    requiredSystemFeatures = [ "recursive-nix" ];
    outputHashMode = "text";
    outputHashAlgo = "sha256";
    __contentAddressed = true;
  };
in
builtins.outputOf root.outPath "out"
cycle.sh
set -eu
export PATH="$COREUTILS/bin:$NIXBIN/bin"
export NIX_CONFIG='experimental-features = nix-command ca-derivations dynamic-derivations'

PH=$(nix eval --raw --expr 'builtins.placeholder "out"')
bashB=$(basename "$BASHPKG"); coreB=$(basename "$COREUTILS")
nixB=$(basename "$NIXBIN");   stepB=$(basename "$STEP")

# A generator that is byte-identical at every level: nothing here varies, so
# `nix derivation add` hands back the same store path every time.
cat > next.json <<EOF
{
  "version": 4,
  "name": "gen.drv",
  "system": "x86_64-linux",
  "builder": "$BASHPKG/bin/bash",
  "args": ["-e", "$STEP"],
  "env": {
    "out": "$PH",
    "BASHPKG": "$BASHPKG", "COREUTILS": "$COREUTILS",
    "NIXBIN": "$NIXBIN", "STEP": "$STEP",
    "PATH": "$COREUTILS/bin:$NIXBIN/bin",
    "requiredSystemFeatures": "recursive-nix"
  },
  "inputs": { "drvs": {}, "srcs": ["$bashB", "$coreB", "$nixB", "$stepB"] },
  "outputs": { "out": { "method": "text", "hashAlgo": "sha256" } }
}
EOF
nextDrv=$(nix derivation add < next.json)
nextB=$(basename "$nextDrv")

# ...and a wrapper that depends on that generator's dynamic output. At the
# second level the generator being described is the one currently running.
inner=$(nix eval --impure --raw --expr \
  "builtins.outputOf (builtins.outputOf (builtins.storePath \"$nextDrv\") \"out\") \"out\"")

cat > wrap.json <<EOF
{
  "version": 4,
  "name": "wrap",
  "system": "x86_64-linux",
  "builder": "$BASHPKG/bin/bash",
  "args": ["-c", "cp \$inner \$out"],
  "env": { "out": "$PH", "inner": "$inner", "PATH": "$COREUTILS/bin" },
  "inputs": {
    "drvs": {
      "$nextB": {
        "outputs": [],
        "dynamicOutputs": { "out": { "outputs": ["out"], "dynamicOutputs": {} } }
      }
    },
    "srcs": ["$bashB", "$coreB"]
  },
  "outputs": { "out": { "method": "nar", "hashAlgo": "sha256" } }
}
EOF
cp "$(nix derivation add < wrap.json)" "$out"

Then:

$ export NIX_CONFIG='experimental-features = nix-command ca-derivations dynamic-derivations recursive-nix'
$ nix build --store /tmp/cycle-repro -f ./cycle.nix --no-link -L \
    --option system-features 'recursive-nix kvm big-parallel nixos-test benchmark'

A chroot store is used only because ca-derivations has to be enabled for the store doing the building, and my daemon does not have it.

Output:

warning: Ignoring dynamic derivation /nix/store/rk26gx60645chq8fk83ls918jjhxbkwx-root.drv.drv^out while querying missing paths; not yet implemented
building '/nix/store/m618pw5a05zga4gzn0hd06abgppb68py-root.drv.drv'...
building '/nix/store/k8higxq4wyashgk3fpm1vwl9w7chdpz3-gen.drv.drv'...

terminating due to unexpected unrecoverable internal error: Assertion '!awake.empty()' failed in void nix::Worker::run(const nix::Goals&) at ../build/worker.cc:363

What the two builders do:

  1. root.drv.drv runs cycle.sh, which instantiates gen.drv.drv and a wrapper depending on gen.drv.drv^out^out, and emits the wrapper.
  2. gen.drv.drv runs the same cycle.sh. Because the script is deterministic, nix derivation add returns the same store path it just produced at step 1, so the wrapper it emits depends on the dynamic output of the derivation currently being built.
  3. The scheduler has a goal waiting on itself, nothing is awake, assertion.

Expected behavior

An error naming the cycle, the way Nix reports cycles in other places, or a scheduler deadlock message. Not an assertion failure and a core dump.

I do not know whether cycle detection is decidable here in general — the successor is produced by an arbitrary builder — but the specific case where a derivation depends on a dynamic output of a derivation already on the goal stack looks detectable, and either way the assertion should become a real error.

nix --version output

Reproduces on both:

  • nix (Nix) 2.35.2 (the assertion is at worker.cc:363)
  • nix (Nix) 2.36.0pre20260829 built from master (worker.cc:362)

Additional context

experimental-features = nix-command ca-derivations dynamic-derivations recursive-nix, system-features including recursive-nix, x86_64-linux, NixOS.

Context for why anyone would generate a self-identical derivation: I was building a chain where each level emits its successor at build time, so the successors differ only by whatever the builder varies. Get that wrong and you get this, with no hint of what went wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment