dispatchsrht.md

builds.sr.ht: a Different Kind of CI Service

If you've been around the hacker-news-sphere during the last year or so you may have heard of Sourcehut (also known as sr.ht), a new no frills open source alternative to GitHub and GitLab. Sourcehut's offering is split into a growing list separate services, including git and mercurial hosting, ticket tracking, and even wiki hosting.

But one of Sourcehut's strongest offerings, and perhaps its most interesting, is its build system, https://builds.sr.ht/. Despite its spartan appearance, sourcehut's CI offering has many features that put it a step above competing CI services, such as GitLab's or Travis'.

For instance, in addition to the usual Linux distro images running on x86 architecture, Sourcehut supports a large number of operating system and architecture combos, ranging all the way from Linux on x86 all the way to FreeBSD running on PowerPC. This eclectic selection has made it the service of choice for many open source projects that wish to build on many operating systems and architectures.

Let's step aside for a moment and take a look at how dispatching a build to SourceHut looks.

Dispatching manually

The fastest way to get started with builds.sr.ht, is to dispatch a build manually. After we've made an account, we can simply click on the Submit Manifest button on the main builds.sr.ht page and start typing in our build manifest into the textbox.

SourceHut's manifest format shouldn't be too hard to understand if you're familiar with similar services. Here's an example manifest, used to run ledgeroni's test suite.

image: debian/testing
packages:
  - build-essential
  - libssl-dev 
  - zlib1g-dev
  - libbz2-dev
  - libreadline-dev 
  - libsqlite3-dev
  - wget
  - llvm
  - libncurses5-dev
  - libncursesw5-dev
  - xz-utils 
  - tk-dev
  - libffi-dev
  - liblzma-dev
sources:
  - https://github.com/rafacastillol/ledgeroni.git
tasks:
  - setup: |
      curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash
      export PATH="/home/build/.pyenv/bin:$PATH"
      eval "$(pyenv init -)"
      eval "$(pyenv virtualenv-init -)"
      pyenv install 3.7.5
      pyenv local 3.7.5
      cd ledgeroni
      pip install pytest
      pip install .
  - test: |
      export PATH="/home/build/.pyenv/bin:$PATH"
      eval "$(pyenv init -)"
      eval "$(pyenv virtualenv-init -)"
      cd ledgeroni
      pyenv local 3.7.5
      pytest

Manifest files are written in yaml format. We specify what image to use for the virtual machine, we specify some dependencies that should be installed, business as usual. We have to specify where to get our sources from, since our build isn't tied to a source control repo, and finally include a list of tasks to be run sequentially. These are simple shell scripts, nothing fancy. We can now submit our build and watch our tests run peacefully.

I mean, that's an ideal world, but the build actually fails! Looking at the logs, we can trace the cause down to forgetting to include curl in our dependency list. Anyone following fromm home can click the edit and resubmit button, make the edit, and watch it go!

Before that we can use another of SourceHut's features to make sure that was the issue. If you run the failing manifest, you'll see a message at the top:

When a build fails, SourceHut will actually keep the virtual machine around for a while and allow us to ssh into it to investigate the failure in search of a possible fix. Copying and running the ssh command provided, we can immediately connect to the VM, install Curl, and re-run our failing commands.

Being able to ssh into the machine when build fails is the sort of feature that sets SourceHut apart from the competition, and makes it a first class experience in terms of developer ergonomics.

Automatically dispatching builds

If you've used CI before, you're probably used to builds triggering when you upload new changes to a repo. You may have noticed these features are suspiciously missing from builds.sr.ht. If you host your code on SourceHut, builds happen automatically if you include a build.yml in your code repository, but that doesn't mean that you're left stranded if you happen to use a different code hosting service.

dispatch.sr.ht is the SourceHut service responsible for this kind of automation. It currently supports integration with GitHub and GitLab, but it's possible other services will be added in the future.

Setting up dispatch.sr.ht to work with GitHub is a simple process, similar to the way you'd set up CI on a similar service such as Travis. At dispatch.sr.ht, click the configure new task button. After that, on the next page pick the GitHub Commits option. You'll have to authorize your GitHub account to interact with SourceHut. After this simply select the repo you want to set up. Once you've done this, SourceHut will run any build.yml file checked into your repo every time you commit to GitHub.

Using dispatch.sr.ht, transitioning into the SourceHut ecosystem can be gradual and relatively painless. One should keep in mind that the service is still in alpha status, so there's still many rough edges (patches are welcome!). Still, it's definitely a name to keep in mind if you want to experiment with different ways of developing software beyond the standard GitHub model.