Writing (and re-writing) a requestor container start script

requestor_start.md

What are we making?

This little write-up describes a container image for a requestor on the Golem network, including a start-up script that cleanly handles the different ways we might want to spin up the container. I drew a lot of inspiration from a couple excellent repos that cut out a lot of the initial legwork. They did things slightly differently to my approach so I would recommend spending some time with them as well.

The Containerfile

I use podman for my containerization needs, and have created the following Containerfile for our requestor container.

FROM condaforge/mambaforge

RUN apt update && apt install screen jq -y

VOLUME /yagna

COPY benchmarks /mnt/benchmarks
COPY config /mnt/config
COPY resources /mnt/resources
COPY results /mnt/results
COPY workflow /mnt/workflow

# Install yagna requestor
ARG YAG_VER=v0.10.1
WORKDIR /requestor
RUN wget "https://github.com/golemfactory/yagna/releases/download/${YAG_VER}/golem-requestor-linux-${YAG_VER}.tar.gz"
RUN tar -xzf "golem-requestor-linux-${YAG_VER}.tar.gz"
RUN mv golem-requestor-linux-${YAG_VER}/* /usr/bin/
RUN rm -rf /requestor/*

# Update base env, since that's where we get dropped into by default
# This installs GATK, snakemake and yapapi
RUN mamba env update --name base --file /mnt/workflow/envs/env.yml

COPY start.sh /requestor/start.sh
ENTRYPOINT ["/bin/bash", "/requestor/start.sh"]
CMD [ "-m", "interactive" ]

We'll focus on the installing yagna sections as well as the ENTRYPOINT and RUN. The mamba env stuff was a fiddle and is interesting on it's own, but that's more related to the app itself that I'm building and is therefore out of scope.

The first challenge was installing the yagna requestor itself. There is a convenience script available (which I believe is the recommended method) but I opted for fetching the release archive directly. This was mainly because there's an interactive acknowledgment step with the auto-installer that didn't play nice with the docker build process. There are other, more succinct, ways of dealing with this issue as outlined in the aforementioned repos, but this felt more explicit to me.

Using ENTRYPOINT in conjunction with RUN provides a nice mix of flexibility and rigidity where appropriate on startup. A good write-up summarizing their differences is available here. The gist of it being that ENTRYPOINT will specify a script that always runs on startup and CMD can be used to pass sensible defaults that can be easily overridden.

A last point to note, keep in mind the order of your layers while you're developing. When sensible, keep what you're currently working on at the bottom of your Containerfile. This is to avoid having to rebuild previous layers whenever you make a change. Conretely, I used to have the COPY start.sh .. directive before installing yagna and my mamba env. This meant I was constantly rebuilding those layers every thime I wanted to test a change to my script - not good. Moving it down to the bottom made my change/test cycles much shorter.

The O.G. script

The objectives for the start script are:

• run yagna in the background, while allowing it's process to be easily inspected
• detect app-key and/or funding, and fetching/creating those things when they're not already present (this will detect a yagna vol mounted via -v /local/path/to/yagna/dir:/yagna:z in the docker/podman run cmd)
• set YAGNA_APPKEY var so it's available when the actual requestor app is run
• accept commandline args to allow for some flexibility in execution

Here's the original start.sh:

#!/bin/bash
while getopts m: flag
do
    case "${flag}" in
        m) mode=${OPTARG};;
    esac
done

# Start the yagna daemon and put it in the background
echo "Starting yagna daemon in screen session.."
screen -d -m -S yagna_daemon yagna --datadir /yagna service run
sleep 2

get_appkey () {
    echo $(yagna --datadir /yagna app-key list --json | jq -r .values[0][1])
}

if [ $(get_appkey) = "null" ]
then
    echo "No appkey found with --datadir /yagna, creating requestor.."
    yagna --datadir /yagna app-key create requestor
    sleep 5
    echo "Funding.."
    yagna --datadir /yagna payment fund
    sleep 10
else
    echo "Found existing appkey with --datadir /yagna"
fi

echo "Exporting app key and initializing yagna as sender.."
export YAGNA_APPKEY=$(get_appkey)
yagna --datadir /yagna payment init --sender # init sender

echo "Running script in ${mode} mode.."
case "${mode}" in
    interactive) /bin/bash;;
    automatic) snakemake -c1 -s=/mnt/workflow/rules/Snakefile;;
    *) echo "Please select automatic or interactive mode";;
esac

I'll be honest, it took me an embarassing amount of time (4-5 hrs) to put this together. I struggled with the funding steps due to not having enough time between commands for things to "settle". I also had a really hard time with the get_appkey function - a $ here, a set of quotes there. Look, bash is great, but beyond a 1-liner with a few pipes and re-directs it gets pretty arcane pretty quickly. I've worked on pretty massive bash scripts with people far more experienced than I am and the general consensus was "don't touch it unless you absolutely have to".

Why the re-write?

The above script works fine but I know down the line it will grow. As it grows it will become more and more brittle, all the while compounding the original tech debt as it gets to be a bigger and bigger task re-writing it. So immediately after struggling all that time and getting something working (can't give up early and let bash win) I decided to re-write it all in Python... eventually. I originally planned to re-write it immediately but subprocess has it's own particularities and I'm on a bit of a schedule. This will work for now.

Honorable mention: I considered re-writing this in Nushell since it's also a full blown programming language and others have espoused how well it's integrated and how easy it is. However, considering how young it still is and it would be another dependency, I'm opting against it.. for now.

I'm sure some bash veterans would groan, but I don't use vi either so keep your evangelizing to yourself, I strayed from the light long ago. I picked Python for a number of reasons:

• It's easier to write! (for me, and I'm writing it, so that matters)
• It will be more maintainable as it inevitably scales (compounding tech debt)
• It's a lot easier to test

Here's another excellent article going into all the finer points.

I'll be sure to post a write-up once I get around to re-writing the start script, complete with pros/cons and challenges along the way. For now, this can be my first real piece of tech-debt, some debt is healthy right? Right??