Skip to content

Instantly share code, notes, and snippets.

Created February 24, 2020 17:59
Show Gist options
  • Save lava/62a6697abcaf6e5709939ae29e7bd4be to your computer and use it in GitHub Desktop.
Save lava/62a6697abcaf6e5709939ae29e7bd4be to your computer and use it in GitHub Desktop.
CI Debugging with a TLS reverse shell

Interactive Debugging on Github Actions

Did you ever run into some issue where a job would behave slightly different in you CI environment than on your local machine? Did you ever wish you could run just a few commands in a shell on your build machine?

These are, of course rhetorical questions. And if you're using Github Actions to run your CI jobs, you'll have noticed that this use case is not supported at all. Can't create a nice walled garden if anyone could just run a CI job, after all. There are some workarounds (e.g., but since they're not officially supported they can be a bit unstable. Also, even they usually don't reproduce the exact environment found on github's servers.

Anyways, here's a cool technique to investigate your CI failures interactively. It's creating a reverse shell from the build machine, with strict TLS certificate pinning to prevent any random internet person to just look around your build.

First, run this on any server connected to the internet (or at least connected to the build machine):

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl s_server -quiet -key key.pem -cert cert.pem -port 2222

The cert.pem created from the first command here should be made available to the CI job.

In your workflow definition, add the following step:

      - name: Do regular CI stuff

      - name: Spawn Reverse Shell on Failure
        if: failure()
        run: |
          sudo apt-get -qqy install openssl
          mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -strict -verify 1 -verify_return_error -CAfile cert.pem -connect $SERVER_IP:2222 > /tmp/s; rm /tmp/s

(In case you're wondering, yes, -strict -verify still has openssl ignoring the validity of the remote certificate)

Of course, this technique is not unique to github, it can be used on any CI runner with network connectivity to the target host. Which these days is almost always the case. If the target port is changed from 2222 to 443, it will even go through almost any firewalls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment