This will work on any machine that can freely connect to outside ports, but can't listen for incoming connections.
In particular, Azure DevOps CI agents have no "Rebuild with SSH" option (like CircleCI does), so this technique can be handy for debugging CI issues.
- You must be able to run arbitrary commands on the remote host, ideally including installing an SSH server.
- You need a machine on the internet that's able to open a listening port. I used my Linode. You could use an AWS free tier
t2.micro, or open a port to your local machine. Anything works as long as it runs SSH and can receive packets from the target machine. We'll call this machine the 'bounce server'.
So, the target machine is behind some sort of firewall or NAT meaning you can't connect to it directly—perhaps it even has no publicly-accessible IP. You'd like to ssh into the target machine, but you can't even open a TCP connection to it.
Enter port forwarding! Here's what we're going to do:
- First, we'll cause the target machine to connect to our bounce server.
- The target machine will then ask the bounce server to relay packets on a particular port from the bounce server back to the target machine.
- We'll connect to that port on our bounce server using ssh, which will then be forwarded to the target machine.
- Presto! We have ssh on the target machine and can proceed to debug.
The key feature we'll use here is the -R option to ssh, aka "remote port forward". This option allows us to instruct the ssh server on the machine we're sshing to to open a port and listen for traffic, then forward that traffic to the local machine. The syntax for the -R flag is: ssh -R port:host:hostport. host:hostport is the address to connect to from the local machine (the one executing the ssh command), and port is the port to listen on on the remote machine. So if I were on host host1, and I wanted to forward traffic on port 8000 from host2 to port 9000 host1, I would run:
host1$ ssh -R 8000:localhost:9000 host2
In our case, we'd like the bounce server to relay ssh traffic to the CI agent machine on port 22, so we'll run something like:
ci-agent$ ssh -R 2023:localhost:22 bounce.server
And then connecting to port 2023 on the bounce server will be just like connecting to port 22 on ci-agent!
So to do this, you'll need to give the CI machine access to log in to your server. That might be a little scary, and if so you can lock things down a little by specifying restrictions in your ~/.ssh/authorized_keys file on the bounce server:
restrict,port-forwarding,command="false" ssh-rsa YoURKey1+/23125 you@email.address
I recommend generating a new key with ssh-keygen and adding the public key to your authorized_keys with the above restrictions. That key will be allowed to log in, but will only be permitted to use the port forwarding features that sshd provides, and not run commands or request X11 forwarding and so on. You can then put the specially-generated private key on the CI server and even if someone mean gets their hands on it, all they'll be able to do is forward ports. (You can restrict things even further using the permitopen option instead of the blanket port-forwarding option.) I'll refer to the private key of this keypair as BOUNCE_PRIVATE_KEY below, and it will be copied to the target machine.
Assuming your target machine is running sshd, you'll want to generate a keypair that will let you log in, which will end up going in the ~/.ssh/authorized_keys of the target server. I'll refer to the public key of this keypair as TARGET_PUBLIC_KEY below, and the private key as TARGET_PRIVATE_KEY.
All that remains now is to string these commands together. Here's the Azure DevOps pipeline step I created that allowed me to log in to the CI agent and debug my issue:
- script: |
echo "$BOUNCE_PRIVATE_KEY" > port_forward_key
chmod 0600 port_forward_key
mkdir -p ~/.ssh && chmod 0700 ~/.ssh
echo "$TARGET_PUBLIC_KEY" >> ~/.ssh/authorized_keys
ssh -o BatchMode=yes -o StrictHostKeyChecking=no -N -i port_forward_key -R2023:localhost:22 bounce@bounce.server &
whoami
sleep 600
(You might find it useful to encode the BOUNCE_PRIVATE_KEY data as base64 and pass it through base64 -d to decode it if you're pasting the data into a tool that doesn't support newlines.)
Once this series of commands has run, you should be able to log in to the target server with:
ssh -i ./target_private_key -p 2023 user@bounce.server
The ssh connection will get tunneled from port 2023 on bounce.server over to port 22 on the target machine, and with any luck, you'll be greeted by a bash prompt!