Did you ever run into some issue where a job would behave slightly different in you CI environment than on your local machine? Did you ever wish you could run just a few commands in a shell on your build machine?
These are, of course rhetorical questions. And if you're using Github Actions to run your CI jobs, you'll have noticed that this use case is not supported at all. Can't create a nice walled garden if anyone could just run a CI job, after all. There are some workarounds (e.g. https://github.com/nektos/act), but since they're not officially supported they can be a bit unstable. Also, even they usually don't reproduce the exact environment found on github's servers.
Anyways, here's a cool technique to investigate your CI failures interactively. It's creating a reverse shell from the build machine, with strict TLS certificate pinning to prevent any random internet person to just look around your build.
First, run this on any server connected to the internet (or at least connected to the build machine):
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl s_server -quiet -key key.pem -cert cert.pem -port 2222
The cert.pem created from the first command here should be made available to the CI job.
In your workflow definition, add the following step:
- name: Do regular CI stuff
[...]
- name: Spawn Reverse Shell on Failure
if: failure()
run: |
sudo apt-get -qqy install openssl
mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -strict -verify 1 -verify_return_error -CAfile cert.pem -connect $SERVER_IP:2222 > /tmp/s; rm /tmp/s
(In case you're wondering, yes, -strict -verify still has openssl ignoring the validity of
the remote certificate)
Of course, this technique is not unique to github, it can be used on any CI runner with network connectivity
to the target host. Which these days is almost always the case. If the target port is changed from 2222 to 443,
it will even go through almost any firewalls.