Patching the Dynamic Linker/Loader of a Compiled Binary using patchelf #nix

patching_the_dynamic_linker_loader_of_a_compiled_binary_using_patchelf.md

Patching the Dynamic Linker/Loader of a Compiled Binary using patchelf

Given a foreign binary on a Nix system, you have to patch its dynamic linker/loader.

First you need to find out what linker/loader to use.

You can do this by going into your nix repl and loading your desired nixpkgs.

Then you run:

"${pkgs.glibc}/lib64/ld-linux-x86-64.so.2"

Which will return you a path that looks like this:

/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib64/ld-linux-x86-64.so.2

You can then use:

patchelf --set-interpreter '/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib64/ld-linux-x86-64.so.2' ./binarytobepatched

Or you can run it directly:

/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib64/ld-linux-x86-64.so.2 ./binarytobepatched

Note that if you are doing this inside a Nix expression, you should be instead using:

$(cat $NIX_CC/nix-support/dynamic-linker)

---

Note that if you use ldd on NixOS on an unpatched binary it will probably show:

        /lib64/ld-linux-x86-64.so.2 => /nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib64/ld-linux-x86-64.so.2 (0x00007f04fcc29000)

That ld-linux-x86-64.so.2 path is derived from the Glibc that ldd itself was built with.

Note after patching the interpreter, upon running the binary, you may still fail to load some shared library.

For example, the official released NodeJS (https://nodejs.org/en/download/) are partially statically compiled binaries with some additional shared libraries still required. In particular libstdc++.so.6.

So after patching the ./bin/node, and running it, you'll still get something like:

./node: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

This is also revealed by:

$ patchelf --print-needed ./node 
libdl.so.2
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

And by:

$ ldd ./node 
	linux-vdso.so.1 (0x00007ffdf4d7e000)
	libdl.so.2 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libdl.so.2 (0x00007fbd78151000)
	libstdc++.so.6 => not found
	libm.so.6 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libm.so.6 (0x00007fbd78078000)
	libgcc_s.so.1 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libgcc_s.so.1 (0x00007fbd7805e000)
	libpthread.so.0 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libpthread.so.0 (0x00007fbd78059000)
	libc.so.6 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libc.so.6 (0x00007fbd77e59000)
	/nix/store/lxpdbaazqd2s79jx6lngr8nak2rjdaq1-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 (0x00007fbd78158000)

At this point the rpath must be set. However the interpreter and the rpath should probably be set together.

Some more notes.

If RUNPATH is set, RPATH is ignored
RPATH is deprecated and should be avoided
RUNPATH is preferred because it can be overridden by LD_LIBRARY_PATH

See: https://stackoverflow.com/questions/7967848/use-rpath-but-not-runpath

The gcc compile can set the rpath too using -R --enable-new-dtags.

Take note of $NIX_LDFLAGS and $NIX_CFLAGS_COMPILE when gcc is being used.

Read: https://nixos.wiki/wiki/C.

There is a program called ld-linux-x86-64.so.2. This is the "dynamic linker/loader". It is the interpreter of compiled binary programs. https://linux.die.net/man/8/ld-linux.so

# finds the 32 bit dynamic linker and loader
ls /nix/store/*glibc*/*/ld-linux.so.2
# finds the 64 bit dynamic linker and loader
ls /nix/store/*glibc*/*/ld-linux-x86-64.so.2

When you are in a nix-shell or a nix build environment, it's easier to do this:

cat $NIX_CC/nix-support/dynamic-linker

This program is not usually put on the PATH, but you running it directly will give you options that allow you to specialise the execution of ELF executables on Linux operating systems.

According to the FHS, the designated locations for the dynamic linker and loader are:

# for 32 bit
/lib/ld-linux.so.2
# for 64 bit
/lib64/ld-linux-x86-64.so.2

This is why for portable Linux binaries, you generally will want the interpreter path set to one of those paths depending on if you are producing 32 bit binary or 64 bit binary. In fact this is the case for the nodejs executables when downloaded directly off https://nodejs.org.

However not all distros follow this process. NixOS is one of them that does not have anything in /lib nor /lib64.

On NixOS, binaries are expected to have their interpreter path set directly to one of the paths within the /nix/store. Furthermore, binaries are expected to have their DT_RUNPATH also set to relevant library paths to the /nix/store.

This is why so-called portable binaries do not run on NixOS normally, give a weird error message when you execute them:

bash: ./result: No such file or directory

Portable binaries will generally not have the DT_RUNPATH set, because they would expect the operating system to have the correctly configured search paths either via LD_LIBRARY_PATH or the ld.so.cache or finally compiled "system search path" that is embedded into the dynamic linker & loader.

When you use ldd on a binary executable, ldd knows how to find the ld-linux.so so when it shows up, that won't affect the actual execution of the program.

Ok so the easiest way to run a non-NixOS executable in this case is to do these things:

$(cat $NIX_CC/nix-support/dynamic-linker) --library-path '<setsomepath>:<setsomepath>' ./binary

  postFixup = ''
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath "${lib.makeLibraryPath [ stdenv.cc.cc.lib glibc ]}" \
      $out/bin/binary
  '';

In our case, our postFixup would then do the opposite, rather than setting the interpreter and path to NixOS, it would be:

postFixup = ''
  patchelf --remove-rpath ./prebuilds/linux-x64/node.napi.node
'';