Skip to content

Instantly share code, notes, and snippets.

@CMCDragonkai

CMCDragonkai/building_a_nix_package_c&c++.md

Last active April 8, 2024 16:38
Show Gist options
  • Star 20 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save CMCDragonkai/41593d6d20a5f7c01b2e67a221aa0330 to your computer and use it in GitHub Desktop.
Save CMCDragonkai/41593d6d20a5f7c01b2e67a221aa0330 to your computer and use it in GitHub Desktop.
Download ZIP
Building a Nix Package (The C&C++ Version)
Raw
building_a_nix_package_c&c++.md

Building a Nix Package (The C&C++ Version)

Nix can be used to build any kind of package. But here I'm just going to focus on the simple C&C++ case.

Firstly we have to know that the final built packages will located inside /nix/store. Which is globally readable directory of all build inputs and build outputs of the Nix system. The emphasis is on readable, not writable, that is /nix/store is meant to be modified by the user or programs except for the Nix system utilities. This centralises the management of packages, and keeps our packages and package configuration consistent.

So what exactly are we trying to build. Our goal is to build a directory that will be located in /nix/store/*-package-version/, where * is the hash of the package. Preferably a version is also available, but some C&C++ packages don't have versions, so in that case, there's only /nix/store/*-package/.

What will be inside this directory? It follows the GNU Coding Standards described here: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html

Here's an imaginary package (with the /nix/store/ omitted):

*-package-version/bin/                               # executables
*-package-version/lib/                               # shared libraries
*-package-version/include/                           # header files
*-package-version/etc/                               # configuration files
*-package-version/share/                             # architecture independent data files
*-package-version/share/man                          # man pages
*-package-version/share/man/man1                     # man for general commands
*-package-version/share/man/man6                     # man for games and screensavers
*-package-version/share/man/man8                     # man for system administration commands and daemons
*-package-version/share/info                         # info pages
*-package-version/share/doc/package-version/         # arbitrary documentation
*-package-version/share/doc/package-version/pdf/     # pdf documentation
*-package-version/share/doc/package-version/ps/      # ps docuumentation
*-package-version/share/doc/package-version/html/    # html documentation
*-package-version/share/doc/package-version/html/en/ # language specific html documentation

Notice we don't really have /var or /run or /com inside the /nix/store, any programs requiring these, will be patched or configured to use locations outside of /nix/store.

When such a package will is installed, the relevant directories and files will be symlinked to either ~/.nix-profile/, or in /run/current-system/sw/, allowing you access the man pages, the shared libraries, their configuration and other documentation.

The package we are going to try to build is the TraceFileGen and TraceFileSim from the GarCoSim project.

For these 2 packages, we shall create them in nixpkgs/pkgs/development/tools/analysis/garcosim/tracefilegen and nixpkgs/pkgs/development/tools/analysis/garcosim/tracefilesim. We will also modify the nixpkgs/pkgs/top-level/all-packages.nix to add the top-level aliases for the 2 packages that we are creating.

The key tools that help us create these 2 packages are:

  • nix-repl from the nix-repl package
  • nix-build
  • nix-prefetch-git from the nix-prefetch-scripts package

Before we even get to developing our package build and deployment scripts, we need to first content address our dependencies. We'll use nix-prefetch-git for this, as the 2 packages are exported via github.com. Remember to use a revision SHA256 so as to make our package deterministic. Even if you want a release tag, it's still better to get the content address associated to the release tag, as the tag is mutable (can be changed willy-nilly by the package author), but the content address isn't. We can however use a corresponding release tag as a mutable semantic label, designed for human readable aliases.

> nix-prefetch-git --url "https://github.com/GarCoSim/TraceFileGen.git" --rev "4acf75b142683cc475c6b1c841a221db0753b404"
Initialized empty Git repository in /tmp/git-checkout-tmp-rmmqIKdj/git-export/.git/
remote: Counting objects: 161, done.
remote: Compressing objects: 100% (111/111), done.
remote: Total 161 (delta 73), reused 111 (delta 46), pack-reused 0
Receiving objects: 100% (161/161), 163.40 KiB | 101.00 KiB/s, done.
Resolving deltas: 100% (73/73), done.
From https://github.com/GarCoSim/TraceFileGen
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
git revision is 4acf75b142683cc475c6b1c841a221db0753b404
git human-readable version is -- none --
Commit date is 2015-11-13 11:13:13 -0400
removing `.git'...
hash is 69b056298cf570debd3718b2e2cb7e63ad9465919c8190cf38043791ce61d0d6
path is /nix/store/wda0pgx1m01aza4d1mhh35yp6di97bcl-git-export
69b056298cf570debd3718b2e2cb7e63ad9465919c8190cf38043791ce61d0d6

Now we know the package hash that we shall use for content addressing the upstream package. Repeat for TraceFileSim.

We can now develop the package build and deploy scripts.

For TraceFileGen, we actually have 2 files, this is because its build phase is non-standard.

Here's our default.nix a Nix expression that will be deriving the package:

{ stdenv, fetchgit, cmake }:

stdenv.mkDerivation rec {

  name = "tracefilegen";

  src = fetchgit {
    url = "https://github.com/GarCoSim/TraceFileGen.git";
    rev = "4acf75b142683cc475c6b1c841a221db0753b404";
    sha256 = "69b056298cf570debd3718b2e2cb7e63ad9465919c8190cf38043791ce61d0d6";
  };

  buildInputs = [ cmake ];

  builder = ./builder.sh;
  
  meta = with stdenv.lib; {
    description = "Automatically generate all types of basic memory management operations and write into trace files";
    homepage = "https://github.com/GarCoSim"; 
    maintainers = [ maintainers.cmcdragonkai ];
    license = licenses.gpl2;
    platforms = platforms.linux;
  };

}

And here's the builder.sh:

source "$stdenv"/setup

cp --recursive "$src" ./

chmod --recursive u=rwx ./"$(basename "$src")"

cd ./"$(basename "$src")"

cmake ./ 

make

mkdir --parents "$out"/bin
cp ./TraceFileGen "$out"/bin

mkdir --parents "$out"/share/doc/"$name"/html
cp --recursive ./Documentation/html/* "$out/share/doc/$name/html/"

While for TraceFileSim, it's very standardised build phase, but its install phase is not standardised, so we just have to override the installPhase:

{ stdenv, fetchgit }:

stdenv.mkDerivation {

  name = "tracefilesim";
  
  src = fetchgit {
    url = "https://github.com/GarCoSim/TraceFileSim.git";
    rev = "368aa6b1d6560e7ecbd16fca47000c8f528f3da2";
    sha256 = "22dfb60d1680ce6c98d60d12c0d0950073f02359605fcdef625e3049bca07809";
  };
  
  installPhase = ''
    mkdir --parents "$out/bin"
    cp ./traceFileSim "$out/bin"
  '';

  meta = with stdenv.lib; {
    description = "Ease the analysis of existing memory management techniques, as well as the prototyping of new memory management techniques.";
    homepage = "https://github.com/GarCoSim";
    maintainers = [ maintainers.cmcdragonkai ];
    licenses = licenses.gpl2;
    platforms = platforms.linux;
  };

}

Before we modify nixpkgs/pkgs/top-level/all-packages.nix, we need to test if our package works locally, this can easily be done using nix-build.

nix-build --keep-failed --expr 'with import <nixpkgs> {}; callPackage ./default.nix {}'

When it fails, because it will probably do so before you get your build scripts correct, you can find the build directory inside /tmp. The state of the directory will be left at exactly when the build was considered to be failed. This can help you iteratively build the build and deploy scripts.

Repeat for both packages.

When a build succeeds, it will place a ./result symlink that goes to the built package in /nix/store. Here you can use it to test the binaries and documentation. It will not be "installed" on your system, so once you remove the ./result symlink, the build artifacts can be garbage collected by Nix.

Once that is done, we can now modify the all-packages.nix to include:

...

  tracefilegen = callPackage ../development/tools/analysis/garcosim/tracefilegen { };

  tracefilesim = callPackage ../development/tools/analysis/garcosim/tracefilesim { };

...

It appears that there isn't much of an order to all-packages.nix, so I just placed it near other packages that had trace in their name.

Note that it is possible to imperatively compile something. Just do it at the user level.

nix-env -iA nixos.gcc
nix-env -iA nixos.gnumake
nix-env -iA nixos.cmake

You can now, run make and cmake in user space.

Nix's mkDerivation is quite advanced and can deal with most standardised configure && make && make install packages. It even detects cmake packages and appropriately configures them. But this time, the 2 packages did not respect the usual configuration parameters of cmake that allows one to specify the output directory. Instead a manual copy of the binaries and documentation was required.

Note that the src attribute can be specified to a local directory. However be aware that ./. is valid for the surrounding directory, but not ./ nor ..

Once you have submitted a PR, or are looking at somebody else's PR for a package or change to NixOS or nixpkgs.

It's easy to check whether the commit has been applied to your branches if you're using pinned OS nixpkgs.

You just do git branch --contains <commit-id> 2>/dev/null. You just need to use the PR commit hash for this.

The command will list all the branches which contain that commit.

Some gnome packages require wrapGAppsHook to be put into their nativeBuildInputs especially when dealing with Gnome settings error.

You should be putting pkgconfig in the nativeBuildInputs instead of buildInputs. Cause it's generally only required at build time, and not runtime.

Even statically linked libraries are still considered buildInputs since the code that gets generated is still running at runtime.

Sometimes deps you need isn't available.

So use things like nix-shell to find things. Use :l <nixpkgs> to load global nixpkgs. Use pkgs = import (fetchTarball ....tar.gz) {} to load other kinds of nixpgks. I think you can also do :l ~/Projects/nixpkgs to nixpkgs as directories as well.

Finally remember once you have checked out your own branch, you can do nix-env -f ~/Projects/nixpkgs -i gpredict to install a package that only exists on your dev branch.

shellHook = ''
  echo 'Entering ShelfTrend API Environment'
  . ./.env
  set -v

  alias mysql="\mysql --socket='./.mysql/mysql.sock' "$SHELFTREND_DB_DATABASE""
  alias mysqladmin="\mysqladmin --socket='./.mysql/mysql.sock'"
  alias mysqld="\mysqld \
    --datadir="$(pwd)/.mysql" \
    --socket="$(pwd)/.mysql/mysql.sock" \
    --bind-address="$SHELFTREND_DB_HOST" \
    --port="$SHELFTREND_DB_PORT""

  alias flyway="\flyway \
    -user="$SHELFTREND_DB_USERNAME" \
    -password="$SHELFTREND_DB_PASSWORD" \
    -url="jdbc:mysql://$SHELFTREND_DB_HOST:$SHELFTREND_DB_PORT/$SHELFTREND_DB_DATABASE" \
    -locations=filesystem:$(pwd)/migrations";

  set +v
'';
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment