Instantly share code, notes, and snippets.

View atomic_directory_commits.md

Atomic Directory Commits

Sometimes you need to "commit" a directory to target location atomically. That is you don't want to have a partial directory where only a subset of files is stored.

Being able to atomically commit a directory relies on 2 conditions:

  1. Having an atomic rename operation where the source and target is on the same filesystem device.
  2. Being able to create a temporary directory descriptor similar to a temporary file descriptor.
View nix_ca_certificate_handling.md

Nix CA Certificate Handling

Some applications requires contacting HTTPS endpoints. In those cases you need to supply the CA certificates.

Most Nix applications won't package in the CA certificates, this is because they can make use of the OS provided CA certificate store.

The NixOS location for this is at: /etc/ssl/certs.

The OpenSSL library in Nixpkgs is compiled to use that path if there is no environment variables such as SSL_CERT_FILE.

View docker_containers_with_tmp.nix
# sometimes you need a container with an embedded `/tmp`
# the `/tmp` will be useful for temporary operations
# note that the `/tmp` may not be very fast depending on the container fs backend
# it will be a true fs-backed `/tmp` since it will be removed along with the container
# with no extra volumes to manage!
# in some cases it would be better to use tmpfs, but that cannot be specified along with the container
# note that nix uses qemu to do this
with import <nixpkgs> {};
{
docker = dockerTools.buildImage {
View shell_container.sh
#!/usr/bin/env sh
docker load --input "$(nix-build --no-out-link -E 'with import <nixpkgs> {}; dockerTools.buildImage { name = "bash"; contents = [ bash coreutils ]; }')"
docker run -it --rm bash /bin/bash
# inside the container you can explore things
# like look at what a docker container built with nix looks like
# or even inspect inside volumes and test mounting with tmp or whatever
View argparse_positional_keyword.py
#!/usr/bin/env python3
import argparse
def main():
# in python you can call functions with positional or keyword arguments
# you can also do something similar with argparse using the below style
# note however if you use the type constraint, you'll have to supply custom functions
View temp_files_directories.md

Using File Descriptors for Temporary Files and Temporary Directories

You can easily use file descriptors to create temporary files:

tmpfile=$(mktemp /tmp/abc-script.XXXXXX)
exec 3<>"$tmpfile"
echo 'Hello World' >"$tmpfile"
rm "$tmpfile"
cat <&3
View raising_exceptions.py
#!/usr/bin/env python
# if you are handling an exception, you can raise another exception in 4 ways:
# this reraises the same exception, in this case
# it just means you couldn't handle the exception
try:
raise Exception
except Exception as e:
raise e
View universal_state.md
View dask_storing_and_loading.py
#!/usr/bin/env python3
import dask.array as da
# a dask array is a grid of individual numpy arrays
# this way you compute on out-of-memory numpy arrays
# this means dask exposes IO functions for dask arrays
# that is intended to work on collections of numpy arrays
View count_number_of_files.md

Count the number of files

And we want do it fast!

# this ignores the directory itself, counting ONLY the files that exist in that directory
find $DIR -mindepth 1 -maxdepth 1 -printf '\0' | wc -c

Want it even faster? Here are the options: