This is a brief guide on how to configure an SSH reverse tunnel that automatically establishes on boot and will continuously attempt to re-connect when it fails.
It is very useful if you are deploying a device somewhere without a public IP, e.g. behind a NAT, and need to be able to SSH into it from the wider internet.
Let's refer to the NAT'ed device as the client. This guide assumes that the client is able to create outgoing SSH connections to at least destination port 443.
You will need root access to a server with a static IP on the internet which runs an openssh server.
On my-server.example.com add the following to /etc/ssh/sshd_config, changing tunnel-user to whichever username you want to use (this will be a new user, not an exising user) and changing the PermitOpen line:
ClientAliveInterval 10
Match User tunnel-user
AllowTcpForwarding yes
X11Forwarding no
PermitTunnel no
GatewayPorts yes # Allow users to bind tunnels on non-local interfaces
AllowAgentForwarding no
PermitOpen localhost:2222 my-server.example.com:2222 # Change this line!
ForceCommand echo 'This account is restricted for ssh reverse tunnel use'
If you want to receive the incoming SSH connections from the client on port 443 (in case the client is on a restricted/firewalled network) then assuming this server is not already using port 443 for something else (like HTTPS) in the same file find the line Port 22 towards the top and add Port 443 to it so it looks like this:
Port 22
Port 443
If Port 22 is commented out make sure you remove the #.
Now restart the ssh server:
/etc/init.d/ssh restart
Now, stil on the server, add a user with the same username:
adduser \
--disabled-password \
--shell /bin/false \
--gecos "user for reverse ssh tunnel" \
tunnel-user
Switching to the client, first check that the user that will be opening the ssh tunnel has a public key at ~/.ssh/id_rsa.pub. If not, become that user and create it:
sudo -i -u tunnel-user
ssh-keygen -t rsa
Hit enter thrice to accept the defaults.
Now open the public key file:
less ~/.ssh/id_rsa.pub
and copy the contents into your copy-paste buffer.
Back on the server, create the .ssh directory and authorized keys file. Set permissions and open the file in an editor. Then paste the public key into the file and close and save like so:
cd /home/tunnel-user
mkdir .ssh
touch .ssh/authorized_keys
chmod 700 .ssh
chmod 600 .ssh/authorized_keys
chown -R tunnel-user.tunnel-user .ssh
nano .ssh/authorized_keys
# paste in the public key, hit ctrl+x then y, then enter to save and exit
Back on the client test that you can open a connection. Do not skip this step as it will ask you to verify the public key fingerprint of the server on first connect. Everything else will fail if this is not done:
ssh -N tunnel-user@my-server.example.com
If it asks for a password something went wrong. If it just sits there forever, apparently doing nothing, then everything is working as expected. Hit ctrl-c once you're satisfied.
Now try to create a tunnel:
ssh tunnel-user@my-server.example.com -N -R my-server.example.com:2222:localhost:22
while that is running, from e.g. your laptop try to connect to the client computer via the reverse tunnel:
ssh -p 2222 someuser@my-server.example.com
Where someuser is the user on the client you're connecting as.
If this works you can now set up autossh to make the client auto-establish the tunnel on boot and auto-re-establish this tunnel every time it fails.
If your client is a regular desktop linux distro that uses openssh then this section is for you.
On the client install autossh:
sudo apt install autossh
Now create the file /usr/local/bin/autossh_reverse_tunnel with the following contents:
#!/bin/bash
REMOTE_HOST=my-server.example.com
REMOTE_PORT=443 # Where REMOTE_HOST has an sshd listening on its public IP and localhost
REVERSE_PORT=2222 # The port where REMOTE_HOST should listen and reverse forward
LOCAL_PORT=22 # The port on the client where the openssh server is listening
REMOTE_USER=tunnel-user # The user on REMOTE_HOST which is allowed to tunnel
while :
do
echo "(re)starting autossh"
autossh -M 0 -N -q -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval=60" -o "ServerAliveCountMax=3" -p $REMOTE_PORT -l $REMOTE_USER $REMOTE_HOST -R ${REMOTE_HOST}:${REVERSE_PORT}:localhost:${LOCAL_PORT}
sleep 10
done
Edit the variables at the top to your liking. If you're using the standard port 22 rather than port 443 on the server then set REMOTE_PORT to 22. Remember that REVERSE_PORT must be higher than 1024 unless you are logging in as root. See the end of this guide for a more in-depth explanation of this script.
Make the script executable:
chmod 755 /usr/local/bin/autossh_reverse_tunnel
Try it:
/usr/local/bin/autossh_reverse_tunnel
Again you should be able to ssh into my-server.example.com on port 2222.
Kill the script, then again on the client create /etc/systemd/system/autossh.service with the contents:
[Unit]
Description=Keeps a reverse tunnel to 'my-server.example.com' open
[Service]
User=somelocaluser
ExecStart=/usr/local/bin/autossh_reverse_tunnel
[Install]
WantedBy=multi-user.target
Where you should change somelocaluser to the user on the client that you want to run the autossh command as.
To make it auto-start on boot:
sudo systemctl enable autossh
sudo systemctl daemon-reload
Start it now:
sudo systemctl start autossh
That's it! It may take a few minutes for the tunnel to re-establish if the client connection drops out, especially if the client gets a new internet IP since the old tunnel then has to first time out followed by the client re-establishing a new tunnel. You can sudo tail -f /var/log/auth.log on the server to watch the client connection attempts which is useful for debugging.
First, autossh is being started in a loop because I have observed the autossh process dying. I don't know if this is a bug or I'm just not using the right options but there's no reason to take chances.
The -M 0 argument to autossh disables keepalive using the old-school echo service on a separate port, which could be problematic through firewalls and requires extra configuration on most servers.
The options ServerAliveInterval=60 and ServerAliveCountMax=3 enable an alternate keepalive strategy over the ssh connection. These options basically say "If no data is received from server in 60 seconds, send a keepalive request. If nothing has been received back from the server after sending three keepalive requests and waiting 60 seconds after each, consider the connection dead."
The option ExitOnForwardFailure=yes causes ssh to exit if the SSH connection was established but creating the tunnel did not succeed. Without this the ssh connection can easily end up hanging permanently and autossh will not save you from this fate.
These options are documented in man ssh_config.
The argument -R ${REMOTE_HOST}:${REVERSE_PORT}:localhost:${REMOTE_PORT} creates the reverse tunnel. It says: After successfully ssh'ing into the server, on the server open a tunnel listening on port REVERSE_PORT on the public IP associated with REMOTE_HOST and forward connections on this port to localhost on port REMOTE_PORT.
Note that if you use a hostname as REMOTE_HOST then this assumes that the server resolves its own hostname to the same public IP as a public DNS server on the internet. E.g. if my-server.example.com resolves to 1.2.3.4 from the internet but on the server itself resolves to 127.0.0.1 then this will not work and in the -R command you should replace ${REMOTE_HOST} with 1.2.3.4. The -R option is explained in man ssh.
Create the script /usr/bin/reverse_ssh_tunnel with the following contents, editing the variables for your needs:
#!/bin/sh
REMOTE_HOST=my-server.example.com
REMOTE_USER=tunnel-user
REMOTE_PORT=443 # Where REMOTE_HOST has an sshd listening on its public IP and localhost
REVERSE_PORT=2222 # The port where REMOTE_HOST should listen and reverse forward
LOCAL_PORT=22 # The port on the client where the openssh server is listening
KEEPALIVE=5
# openwrt /etc/init.d scripts don't set these
# which causes dropbear ssh to look for known_hosts in /.ssh/known_hosts
# instead of /root/.ssh/known_hosts
USER='root'
LOGNAME='root'
HOME='/root'
export USER
export LOGNAME
export HOME
while :
do
echo "(re)connecting reverse ssh tunnel to $REMOTE_HOST"
ssh ${REMOTE_USER}@$REMOTE_HOST -K $KEEPALIVE -N -p $REMOTE_PORT -R ${REMOTE_HOST}:${REVERSE_PORT}:localhost:$LOCAL_PORT
sleep 10
done
Then chmod 755 /usr/bin/reverse_ssh_tunnel
Now create the script /usr/bin/start_reverse_ssh_tunnel:
#!/bin/sh
/usr/bin/reverse_ssh_tunnel &
Then chmod 755 /usr/bin/start_reverse_ssh_tunnel.
This script exists only because the initd stuff doesn't seem to like starting a progress in the background. I'm sure there's a better way so please leave a comment if you know how!
This next sub-section is for systems that just use generic init.d scripts without procd. See the OpenWRT subsection below for a solution that works with procd.
Now create /etc/init.d/reverse_ssh_tunnel:
#! /bin/sh
### BEGIN INIT INFO
# Provides: reverse-ssh-tunnel
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Should-Start:
# Should-Stop:
# Short-Description: Start reverse ssh tunnel
### END INIT INFO
case "$1" in
start)
echo "Starting reverse ssh tunnel"
/usr/bin/start_reverse_ssh_tunnel &
;;
stop)
echo "Stopping reverse ssh tunnel NOT IMPLEMENTED"
;;
reload|force-reload)
echo "Reloading reverse ssh tunnel NOT IMPLEMENTED"
;;
restart)
echo "Restarting reverse ssh tunnel NOT IMPLEMENTED"
;;
*)
echo "Usage: /etc/init.d/reverse_ssh_tunnel {start}"
exit 1
esac
exit 0
Then chmod 755 /etc/init.d/reverse_ssh_tunnel.
Test that it's working.
To stop the tunnel first run /etc/init.d/reverse_ssh_tunnel stop and then use ps | grep ssh to find any remaining reverse ssh related process and kill it with kill <pid>.
cd /etc/rc.d
ln -s ../init.d/etc/init.d/reverse_ssh_tunnel S96reverse_ssh_tunnel
Now the reverse ssh tunnel should auto-start on boot.
Now create /etc/init.d/reverse_ssh_tunnel:
#!/bin/sh /etc/rc.common
START=96
STOP=01
USE_PROCD=1
NAME=reverse_ssh_tunnel
start_service() {
procd_open_instance
procd_set_param command /bin/sh "/usr/bin/start_reverse_ssh_tunnel"
procd_set_param stdout 1
procd_set_param stderr 1
procd_close_instance
}
Then chmod 755 /etc/init.d/reverse_ssh_tunnel.
Test that it's working. You can view the script's stdout and stderr by starting logread -f before running /etc/init.d/reverse_ssh_tunnel start.
To stop the tunnel first run /etc/init.d/reverse_ssh_tunnel stop and then use ps | grep ssh to find any remaining reverse ssh related process and kill it with kill <pid>.
cd /etc/rc.d
ln -s ../init.d/etc/init.d/reverse_ssh_tunnel S96reverse_ssh_tunnel
Now the reverse ssh tunnel should auto-start on boot.