Do you rely on test workflows for upholding a high level of code quality? Have you ever been frustrated at the fact that every time you want to make changes to your github workflow .yml you have to commit changes which may lead to failed builds, again and again? have you ever been frustrated that this leads to "commit pollution"? I have. And this is why I decided to use act, which is a superb tool that spawns a docker instance to run the workflow locally. It is user friendly, but still requires some configuration to get it up and running, especially if you want fancy things like authenticating and connecting to external services.
The objective of this short tutorial is to run a Github test workflow on our local machine to speed up development iterations. This is of particular importance when you don't want to pollute Git history by committing a lot of code whose purpose is to get the test workflow to pass. In our case, this is especially important, as our tests are data-intensive and incur large bandwidth overhead on our Azure instance, which translates to higher cost.
We are going to use act, which is a tool that spawns a docker instance to run the workflow locally.
NOTE: If you find this tool useful, please consider supporting the developer here.
For this to work we will need to do the following:
- Install Docker Engine. Please make sure that the Docker daemon is running and functional prior to proceeding with this tutorial.
- Create an
.actrcconfig file, which tellsactwhich base image to use. We can use a pre-built base image, or we can build our own base image. - Create a
Dockerfileand build it. - Run the docker image, passing some environment variables needed for the workflow to execute.
- (Optional) Create a
act.vaultfile, which stores our authentication credentials for connecting to our external service.
Let's start! 🚀
At this point your local directory structure should look something like this:
https://gist.github.com/a0b8806d8d4983c1d97745a54dab1960
The .actrc config file should be in your top-level directory.
Contents of .actrc:
https://gist.github.com/f10b5292cfbe2ee2e5402aa1ec3e9989
This specifies which ubuntu image the Dockerfile should use. For more info on available docker images for act have a look here.
A typical workflow .yml file (i.e. tests.yml) may look like this:
https://gist.github.com/db6a82e04428aa5b74791125e81bd3ef
NOTE: In our case we are using Azure Blob Storage to store the data and DVC to version it.
The line
https://gist.github.com/24a92a936a08af1a3294010ee1998041
specifies upon which action (push, pull_request) the workflow should run. The $ACTION variable will be passed to the container during runtime.
At this point your local directory structure should look something like this:
https://gist.github.com/8ffa09f07949c4b50d7bbc2439498e72
Let us create a Docker-in-Docker Dockerfile:
Contents of Dockerfile:
https://gist.github.com/064357706ea987d814df0201da3b2282
At this point your local directory structure should look something like this:
https://gist.github.com/6b1c34cb86ce5eb2e3ce2bf8fb10bae1
Let's build the Dockerfile via
https://gist.github.com/a1b143504495afe7b915bab8e2621eff
NOTE: If you haven't enabled rootless mode, you may have to use
sudo.
Now you can run docker images (or sudo docker images) and see the newly built image.
Now we can run our image and look at the logs.
https://gist.github.com/76365f944ff603696113b6184ba0ad19
Hopefully, this should now run your workflow, if you don't require any kind of authentication to access your external service.
At this point your local directory structure should look something like this:
https://gist.github.com/73434f1dde9ba13d170ef4c9235e5472
To observe the logs, run
https://gist.github.com/f4b669299ee19660258e856e9700e332
Sometimes we are connecting to external services (i.e. Azure Blob Storage)in order to fetch some data or do other things. To understand how to set up an Azure AD application and service principal, have a look at this tutorial. In our case, we have registered our Github workflow as an app on Azure, and have obtained an Azure secret credential which is passed to the workflow using a Github environment variable. It happens to be called secrets.AZURE_CREDENTIALS. On Github, this can be set via repository settings menu, available to the administrator.
Once you have set up your app on Azure and obtained your secret key, then you can also use this key locally. We can employ the --secret-file $PATH_TO_SECRET flag to tell act to look inside a file where we have stored our secret credential, i.e. act.vault. We have to be careful how we store our secret key inside this file, especially if it is a JSON file (check out this for more details).
Contents of act.vault, which in this case is formatted in yaml:
https://gist.github.com/d855b1a09d951b703939ed353696815c
(...make sure there are no newlines in your JSON!)
We have to put our secret file act.vault inside a directory secret/. Our directory structure should now look something like this:
https://gist.github.com/757d9be35ee8f05d5d91e6d35a185bf2
As of the time of writing this, the Ubuntu 20.04 image kindly provided by @catthehacker does not come with the Azure CLI preinstalled, so we will have to use this image as a base and install az on top of it. The Dockerfile for our new base image will look like this:
https://gist.github.com/0f4e7b0c6987504d9f9e5b7cdaf0c46e
We have to first build the act image:
act image:
https://gist.github.com/a5cf349a32e9a3d3eed46b63b6c62410
Once we decide which act image to use (pre-built or our own), we also have to change the contents of our .actrc to use the new act image in our dind container:
Contents of .actrc:
https://gist.github.com/468b699a822a3fade4c3f25d13393f91
I have provided a prebuilt image in my Docker hub repo. If you want to use that instead, you can replace the above with -P ubuntu-latest=orphefs/orphefs:act-ubuntu-20.04 in your .actrc.
Now, let's include the new argument inside Dockerfile:
Contents of Dockerfile:
https://gist.github.com/8304cc7f0b1784247655f20ebf3a232f
Now,let's build the dind image again:
https://gist.github.com/f380c1490fa61c4b863c5fad5eab405d
Now we can run the dind container using
https://gist.github.com/260561f261b1557cf92a5ec3a01fef11
Hopefully the above runs smoothly and updates the ci-logs/run.log file, so we can view the output on stdout via
https://gist.github.com/676e6a7a6496d37d898ada73d8b39d6c
which should look something like this:
https://gist.github.com/05806fe51fdd90800fe2e5179e8bcc3c
Happy workflowing 👍
You can find the template files used for this tutorial in this repo.
This tutorial was inspired by:
- https://gauravgahlot.in/blog/run-github-actions-locally-docker-nektos-act/
- Conversations with the
actdevelopers on Gitter - Coffee