Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Okay, I've got a need to build Firefox from source, and I'd like to do that on a remote machine, and then copy build result back to my laptop. With Nix, using bastion host. I'll note details of my successful adventure.

Setup & Sources of knowledge

Here's the list of resources I've used actively:

Here's my setup:

  • local machine - ThinkPad 8cores 16Gb RAM, NixOS
  • remote machine - some 48-core 512Gb RAM, Ubuntu, with Nix installed in multi-user way
  • jump host - a bastion host, Ubuntu, ssh to remote machine works only from jump host

I wanted to build nixpkgs.firefox-esr-52, it is last one to support NPAPI plugins (I need icedtea for fucking remains of Java-applet in my bank). It stopped being built by Nixpkgs Hydra because it is "insecure". And previously built firefox stopped working because of some GLIBC version mismatch....

First step - simplify SSH

Let's make ssh remote_host seamless. Append to ~/.ssh/ssh_config:

Host remote_host
  ProxyCommand ssh -W remote_host:1221 ubuntu@jump_host
  User ubuntu

If ssh remote_host now works, remember, it is not enough! This should work for root too, but you can't edit /root/.ssh/ssh_config. It just doesn't work. You have to edit /etc/ssh/ssh_config or set extra config through NixOS options:

  programs.ssh.extraConfig = ''
    Host remote_host
      ProxyCommand ssh -i /root/.ssh/my_key -W remote_host:1221 ubuntu@jump_host
      IdentityFile /root/.ssh/my_key
      User ubuntu
  '';

You should play with various SSH config params until ssh remote_host works as root. For example, note that you should have your private key copied to root home, with root owner.

NOTE: Ideally you should connect to remote host as root, but I didn't have such an opportunity. This will make a bit pain later.

Configure signing

Signing should prevent you to download malware build artifacts. Even when we use SSH private key to reach our builder. But oh well, let's do this.

Remote host:

  • nix-store --generate-binary-cache-key builder-name cache-priv-key.pem cache-pub-key.pem
  • echo "\nsecret-key-files = $PWD/cache-priv-key.pem" >> /etc/nix/nix.conf
  • sudo $(which nix) sign-paths --all -k cache-priv-key.pem

(if you ask why $(which nix) - it is because by default sudo on Ubuntu limits PATH envvar.)

Local host:

  nix.binaryCachePublicKeys = [
    "builder-name:dauKsezR2tY+HsLpeH7hzx8LcCTdPvUPBoJ/3Y="   # this one is cache-pub-key.pem content
  ];
  nix.trustedBinaryCaches = [
    "ssh-ng://remote_host"
    "ssh://remote_host"
  ];

VERY IMPORTANT: don't set nix.binaryCaches here. nix.trustedBinaryCaches sets optional builders, nix.binaryCaches sets default builders.

Controlling build process

So, when you do nix-build '<nixpkgs>' -A firefox-esr-52 you download everything from cache, except firefox itself. So we want to build only firefox remotely, and everything else should be downloaded directly.

The build is like this:

  • first we generate .drv for firefox package
  • then we copy this .drv to remote builder
  • then we SSH and build .drv on a remote builder
  • then we copy result back
  • from now on, our local nix commands will know about package and won't try to build it from source.

This is described in NixOS/nix#1639 (comment). I'll give direct commands on what was done.

The firefox expression

It is actually an overlay:

self: super: let
in {
  ff_java = (import <nixpkgs> {
    config = {
      allowUnfree = true;
      permittedInsecurePackages = [
        "firefox-52.9.0esr"
        "firefox-esr-unwrapped-52.9.0esr"
      ];
      firefox = {
        ffmpegSupport = false;
        icedtea = true;
      };
    };
  }).firefox-esr-52;
}

in /home/danbst/.config/nixpkgs/overlays/firefox_java.nix

Instantiate

$ nix-instantiate '<nixpkgs>' -A ff_java
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/ynpj76a9nfdrx82zqirfvvrldw2gf3fn-firefox-52.9.0esr.drv

Copy .drv

$ nix-copy-closure --to remote_host -s /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv

The -s param is important both as optimization and a fix for a problem. It says "do downloads where possible on a remote side". If you don't set it, it will copy your local .drv-s to remote host. But then you can have a problem - not all your .drv-s are trusted on remote builder! -s soemhow fixes this problem.

Build

$ ssh remote_host
remote_host $ nix build /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
.... nice Nix2.0 build output ...
.... unfortunately, Nix2.0 doesn't show the path to package built, so we have to run old nix-build. It will be no-op later ... 
remote_host $ nix-build /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
/nix/store/9x839fn56fyc6ar3wpgp6x9h5zn91m1j-firefox-52.9.0esr

But because we've built it not as root, we have to sign our build result. Easiest is to do again

$ sudo $(which nix) sign-paths --all -k cache-priv-key.pem

Copy back

$ nix-copy-closure --from remote_host -s /nix/store/9x839fn56fyc6ar3wpgp6x9h5zn91m1j-firefox-52.9.0esr

And that's it! I have now firefox-52 on my laptop. Last thing -- pin it in as garbage root:

$ nix-env -f '<nixpkgs>' -iA ff_java -p /nix/var/nix/profiles/per-user/danbst/firefox-java

Failed approach

I've tried also distributed builder config:

  nix.distributedBuilds = true;
  nix.buildMachines = [
    {
      hostName = "remote_host";
      system = "x86_64-linux";
      maxJobs = 32;
    }
  ];

Well, I was able to launch it

$ nix build -vvvvvvvvvv --option extra-substituters ssh-ng://remote_host --option max-jobs 0 /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv

however it started downloading Firefox sources to my laptop. I've canceled it before it started sending Firefox sources from my latop to remote host 🤦

@balsoft
Copy link

balsoft commented Jan 16, 2020

If you're still interested in this, you can add --builders-use-substitutes to your nix build so that the sources are downloaded straight to the remote host.

@knedlsepp
Copy link

knedlsepp commented Sep 27, 2021

How would nix-build --store ssh-ng://remote_host -A ff_java compare to that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment