Build CockroachDB on NixOS (Cockroach Build Secrets Exposed)

Build CockroachDB on NixOS.md

Build CockroachDB on NixOS

(entirely serious, this is how you do it, last tested January 12, 2023)

Summary: you can use nixpkgs's bazel_5 to build and run CockroachDB's bazelbuilder Docker container, which will use the bazel version it really wants to use for building cockroach. You need to patchelf a few things along the way, and then patchelf the resulting cockroach binary.

---

1. Add this to your NixOS configuration:

{ pkgs, ... }:

{
  virtualisation.docker.enable = true;

  users.users = {
    YOURUSER = {
      extraGroups = [ "docker" ];
    };
  };

  environment.systemPackages = with pkgs; [
    bazel_5
    patchelf
  ];
}

where YOURUSER is your username, and nixos-rebuild.

Note that if you're working in a tmux and started the tmux server before you put yourself in the docker group with that configuration, you might need to e.g.

setfacl -m u:YOURUSER:rw /var/run/docker.sock

(as root)

2. Save this script to ~/bin/fix-cockroach; it will be used later:

#!/usr/bin/env bash

set -eu -o pipefail

interpreter=$(patchelf --print-interpreter /bin/sh)
libstdc=$(nix-build '<nixpkgs>' --no-out-link -A stdenv.cc.cc.lib)/lib
ncurses=$(nix-build '<nixpkgs>' --no-out-link -A ncurses6\.out)/lib

for i in ~/bin/cockroach; do
	patchelf --set-interpreter "$interpreter" "$i"
	patchelf --set-rpath "$libstdc:$ncurses" "$i"
done

3. Enjoy patchelf pain

git clone https://github.com/cockroachdb/cockroach
cd cockroach

# Remove .bazelversion so that it doesn't demand a bazel 5.1.0
# which would not work on NixOS anyway due to its bundled JRE
rm -f .bazelversion

# Have it try to build the bazelbuilder container
./dev builder -- ls

# After it fails with an `bin/go’: No such file or directory` error, patch `go` and try again.
for i in ~/.cache/bazel/*/*/external/go_sdk/bin/go; do patchelf --set-interpreter $(patchelf --print-interpreter /bin/sh) "$i"; done
./dev builder -- ls

# After it fails with a `bin/dev-versions/dev.*: No such file or directory` error, patch dev.* and try again.
for i in bin/dev-versions/dev.*; do patchelf --set-interpreter $(patchelf --print-interpreter /bin/sh) "$i"; done
./dev builder -- ls

# Restore .bazelversion and build cockroach
git checkout HEAD -- .bazelversion
./dev builder -- bazel build pkg/cmd/cockroach --config=with_ui

# Retrieve cockroach from the container
./dev builder -- cat _bazel/bin/pkg/cmd/cockroach/cockroach_/cockroach > ~/bin/cockroach && chmod +x ~/bin/cockroach

# Patch the binary to make it work on NixOS
~/bin/fix-cockroach