CMCDragonkai

Atomic Directory Commits

Sometimes you need to "commit" a directory to target location atomically. That is you don't want to have a partial directory where only a subset of files is stored.

Being able to atomically commit a directory relies on 2 conditions:

1. Having an atomic rename operation where the source and target is on the same filesystem device.
2. Being able to create a temporary directory descriptor similar to a temporary file descriptor.

CMCDragonkai

Nix CA Certificate Handling

Some applications requires contacting HTTPS endpoints. In those cases you need to supply the CA certificates.

Most Nix applications won't package in the CA certificates, this is because they can make use of the OS provided CA certificate store.

The NixOS location for this is at: /etc/ssl/certs.

The OpenSSL library in Nixpkgs is compiled to use that path if there is no environment variables such as SSL_CERT_FILE.

CMCDragonkai

# sometimes you need a container with an embedded `/tmp`
# the `/tmp` will be useful for temporary operations
# note that the `/tmp` may not be very fast depending on the container fs backend
# it will be a true fs-backed `/tmp` since it will be removed along with the container
# with no extra volumes to manage!
# in some cases it would be better to use tmpfs, but that cannot be specified along with the container
# note that nix uses qemu to do this
with import <nixpkgs> {};
{
  docker = dockerTools.buildImage {

CMCDragonkai

#!/usr/bin/env sh

docker load --input "$(nix-build --no-out-link -E 'with import <nixpkgs> {}; dockerTools.buildImage { name = "bash"; contents = [ bash coreutils ]; }')"
docker run -it --rm bash /bin/bash

# inside the container you can explore things
# like look at what a docker container built with nix looks like
# or even inspect inside volumes and test mounting with tmp or whatever

CMCDragonkai

#!/usr/bin/env python3

import argparse


def main():

    # in python you can call functions with positional or keyword arguments
    # you can also do something similar with argparse using the below style
    # note however if you use the type constraint, you'll have to supply custom functions

CMCDragonkai

Using File Descriptors for Temporary Files and Temporary Directories

You can easily use file descriptors to create temporary files:

tmpfile=$(mktemp /tmp/abc-script.XXXXXX)
exec 3<>"$tmpfile"
echo 'Hello World' >"$tmpfile"
rm "$tmpfile"
cat <&3

CMCDragonkai

#!/usr/bin/env python

# if you are handling an exception, you can raise another exception in 4 ways:

# this reraises the same exception, in this case
# it just means you couldn't handle the exception
try:
    raise Exception
except Exception as e:
    raise e

CMCDragonkai

Universal State

Universal addresses

Examples:

• URIs
• Multiaddresses https://github.com/ipfs/ipfs/issues/227
• Magnet Links
• Content Addressing

CMCDragonkai

#!/usr/bin/env python3

import dask.array as da

# a dask array is a grid of individual numpy arrays
# this way you compute on out-of-memory numpy arrays

# this means dask exposes IO functions for dask arrays
# that is intended to work on collections of numpy arrays

CMCDragonkai

Count the number of files

And we want do it fast!

# this ignores the directory itself, counting ONLY the files that exist in that directory
find $DIR -mindepth 1 -maxdepth 1 -printf '\0' | wc -c

Want it even faster? Here are the options: