Lisanna's eclectic nix troubleshooting guide for suicidal power users

help.md

Lisanna's eclectic nix troubleshooting guide for suicidal power users

tar fails with exit code 1

unpacking 'https://example.com/nixexprs.tar.xz'...
tar: Error opening archive: Failed to open '/nix/store/411sszfqfya5ad2wllmqcr15z5d30575-nixexprs.tar.xz'
error: program 'tar' failed with exit code 1

Discussion:

• NixOS/nix#2085
• NixOS/nix#2521

Hydra isn't scheduling my builds!

• Make sure you have https://github.com/NixOS/hydra/pull/597/commits/73ca325d1c0f7914640a63764c9a6d448fde5bd0 applied
• You (perhaps inadvertantly) asked Hydra to build something for a system / requiring a system feature that none of your builders identify as being able to handle.
  • Sanity-check the "system" being used in your evaluation, and the requiredSystemFeatures of your custom derivations
  • For x86_64-linux systems, also make sure they identify as being able to build i686-linux derivations.
  • Hydra treats local builds very differently from how Nix does. You need an explicit localhost line that looks something like this: localhost builtin,x86_64-linux,i686-linux - 70 1 kvm,local,big-parallel
  • Try building it with Nix from your hydra instance, and stop it after it starts the first build. This will narrow it down to exactly what derivation Hydra is stuck on and help you debug.

linux-related evaluation errors using NixOps on darwin

Don't use NixOps on darwin. It's a fool's errand. NixOps has too much of its own Nix code for me to trust it to not accidentally trip over itself during evaluation. Setup a NixOS virtualbox instance, map your NixOps code directory as a shared folder into the box, and do NixOps deploys from linux.

The Nix installer can't install

Create /etc/nix/nix.conf with just this:

build-users-group =

Proxies aren't working!

Set these environment variables:

• http_proxy=http://proxy.example.com:port
• https_proxy=http://proxy.example.com:port
• NIX_CURL_FLAGS="-x http://proxy.example.com:port"

Or, on NixOS, configure the networking.proxy option.

If that's still not working, you may have screwed yourself by getting the daemon installed and running (multi-user mode) without first making sure it is exposed to those environment variables. For nixos-rebuild/darwin-rebuild, fix your configuration, set the above environment variables manually, and run the rebuild command with NIX_REMOTE= set (e.g., NIX_REMOTE= nixos-rebuild switch).

Any horrifying / meaningless evaluation error deep in nixpkgs

• Make judicious use of builtins.trace (and friends in lib) to narrow down what parts of your code are getting touched by the evaluator (before the error is thrown) and which parts are not getting reached.
• Pay close attention to which functions in nixpkgs are part of the --show-trace. Compare it with your code to see what parts of your code might be causing those functions to get invoked.
• Check your map functions! You probably forgot to specify the expression that it should actually be mapping over, since you got so caught up in writing the function to be mapped itself.
• Add asserts everywhere to enforce that things are the type you expect (e.g., that you're not getting a partially applied function when you're expecting an attrset).

Nix isn't using my substituter!

• Did you actually add it to your substituters list in your nix.conf? Are you signing your packages, and do you have the substituter's key added to your nix.conf?
• Run with -v verbose flags (the more flags, the more verbose). With enough verbosity Nix will eventually tell you why it decided to not use your substituter.

Random / crazy issues with unpacking tarballs (e.g. nixexprs.tar.xz)

Delete ~/.cache.

Hydra evaluation hangs

• systemctl restart hydra-evaluator
• Stop using import-from-derivations in the Nix code that Hydra evaluates. In my experience, the Hydra evaluator hates IFDs.

How do I pin nixpkgs for my project/channel?

Either get all your users to agree on a single consistent nixpkgs URI, or make nixpkgs a submodule of your project. Don't IFD it.

The remote builder has gotten stuck / isn't building / isn't copying.

On the remote builder, killall nix-store. There are probably some stuck nix-store processes.

A derivation's build isn't ending even though the top-level script has exited

You have children or orphan processes still alive with stdio file descriptors that Nix is monitoring. Nix will not end a build as long as there are still processes running that can produce stdout/stderr output. Make sure your builder cleans up its children, or otherwise spawn your children with their stdout/stderr redirected to /dev/null.

Refusing to copy corrupted path to binary cache / hash mismatch importing path

This error is seen if the actual hash of a store path doesn't match what the nix database says it should be, and you are trying to export or import some path from or to a nix store or binary cache.

This corruption can be caused by if anything has performed a write to the path in /nix/store. If your /nix/store directory is bind-mounted read-only, then this can also be caused by NixOS/nix#2535.

There are several ways to fix this:

• Modify the derivation slightly so that it is forced to be rebuilt, allowing you to abandon / ignore the corrupted path.
• Delete the derivation and all of its referrers with nix-store --delete. This can be tedious though, if the path or its referrers have GC roots, since you will also have to delete all of those.
• Use nix-surgeon: https://gist.github.com/ledettwy/b06f767f9aa30f5d68861799fa169ce9