Did you ever run into some issue where a job would behave slightly different in you CI environment than on your local machine? Did you ever wish you could run just a few commands in a shell on your build machine?
These are, of course rhetorical questions. And if you're using Github Actions to run your CI jobs, you'll have noticed that this use case is not supported at all. There are some workarounds (e.g. https://github.com/nektos/act), but since they're not officially supported they can be a bit unstable. Also, even they usually don't reproduce the exact environment found on github's servers.
Anyways, here's a cool technique to investigate your CI failures interactively. It's creating a reverse shell from the build machine, with strict TLS certificate pinning to prevent any random internet person to just look around your build.
First, run this on any server connected to the internet (or at least connected to the build machine):
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl s_server -quiet -key key.pem -cert cert.pem -port 2222
The cert.pem created from the first command here should be made available to the CI job.
In your workflow definition, add the following step:
- name: Do regular CI stuff
[...]
- name: Spawn Reverse Shell on Failure
if: failure()
run: |
sudo apt-get -qqy install openssl
mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -no-CApath -strict -verify 1 -verify_return_error -CAfile cert.pem -connect $SERVER_IP:2222 > /tmp/s; rm /tmp/s
(credit to int0x33 for the original inspiration)
Of course, this technique is not unique to github, it can be used on any CI runner with network connectivity
to the target host. Which these days is almost always the case. If the target port is changed from 2222 to 443,
it will even go through almost any firewalls.
The above solution works great, but after using it a few times it the limitations become more and more noticable: There is no auto-completion, no line editing and, worst of all, hitting CTRL-C immediately and irrevocably terminates the connection.
To remedy all of this, we can set up a ssh reverse shell. The basic idea is to start up a local ssh server on the CI machine, and then create a reverse ssh tunnel so remote users can connect to the local server.
First, generate a new SSH key pair:
ssh-keygen -t rsa -f rshell.id_rsa
The public and secret key generated by this should be made available to the CI job. We're using the keys in both directions below, i.e. both to log in to the middle-man machine and to log in one the CI machine.
We also need a middle-man host, which needs to be accessible from the internet, or at least from the CI
machine. On this host, we create an rshell user that can be used for establishing the reverse tunnel.
It is set up to be able to login only with the private key generated above. To prevent this user from
doing anything else, its shell is set to /bin/true:
adduser --disabled-password --shell /bin/true rshell
mkdir -p /home/rshell/.ssh
cp ${PUBLIC_KEY} /home/rshell/.ssh/authorized_keys
chown -R rshell:rshell /home/rshell/.ssh
The final step is to set up the local ssh server and to create the tunnel: (it should also be possible to use sshd as the ssh server, but it has to be started as root and one needs to be careful with the configuration to stop it from interfering with the system.)
- name: Spawn Deluxe Reverse Shell on Failure
if: failure()
env:
MIDDLEMAN: ${{ secrets.RSHELL_REMOTE_HOST }}
run: |
sudo apt-get -qqy install dropbear-bin openssh-client
mkdir -p ~/.ssh
echo "${{ secrets.RSHELL_USER_PUBLIC_KEY }}" >> ~/.ssh/authorized_keys
echo "${{ secrets.RSHELL_USER_SECRET_KEY }}" >> ~/.ssh/rshell.id_rsa
echo "${{ secrets.RSHELL_REMOTE_HOSTKEY }}" >> ~/.ssh/known_hosts
chmod 0400 ~/.ssh/rshell.id_rsa ~/.ssh/authorized_keys
chmod go-w ~
dropbearkey -t rsa -f ~/.ssh/dropbear.rsa
dropbearkey -t ecdsa -f ~/.ssh/dropbear.ecdsa
dropbear -E -R -w -g -a -p 2222 -P ./dropbear.pid -r ~/.ssh/dropbear.rsa -r ~/.ssh/dropbear.ecdsa
ssh -N -i ~/.ssh/rshell.id_rsa -R 2222:127.0.0.1:2222 ${MIDDLEMAN}
Now, it is possible to log in from the middle-man machine. (if this is a special-purpose machine, one might consider setting GatewayPorts yes in the local sshd config to allow
$ ssh -i ${PRIVATE_KEY_FILE} -p 2222 runner@127.0.0.1
Voila. The connection will persist until the CI job is shut down, and even supports multiple users investigating the CI job in parallel. As above, this can also be obfuscated by using port 443 for environments with a restrictive firewall.
This also works on Mac, except that the apt-get in the first line has to be replaced by
brew install dropbear openssh