Skip to content

Instantly share code, notes, and snippets.

@NiklasGollenstede

NiklasGollenstede/!autossh-service.md

Last active November 26, 2023 03:57
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save NiklasGollenstede/a9e7d3b5520fe05f2027009c2c8211a0 to your computer and use it in GitHub Desktop.
Save NiklasGollenstede/a9e7d3b5520fe05f2027009c2c8211a0 to your computer and use it in GitHub Desktop.
Download ZIP
Automatic, persistent, and secure SSH port forwarding between hosts
Raw
!autossh-service.md

This is a comprehensive and automated guide to set up automatic, persistent, and secure SSH port forwarding between hosts.

Please read the content for more information.

Raw
content.md

AutoSSH server port forwarding

This helps to connect a server that is sitting behind a rather restrictive firewall, and/or a NAT without port forwarding, to the Internet via SSH port forwarding. For that, the restricted server is set up to basically run autossh root@proxy -R ... as a service to connect to a gateway. The gateway should be publicly accessible at the desired port numbers.

Prerequirements

  • The device behind the firewall, referred to as server.
    • The server must have ssh (client) installed and use systemd.
    • Unless autossh is already installed, the server should have apt-get with autossh available in its packet sources. If that is not the case, see "AutoSSH stub" below.
  • A device that is accessible from the Internet, referred to as gateway, where the ports will be tunneled to (make sure to add firewall exceptions here if required).
    • The gateway must run sshd, for all security settings to apply it should be openssh-server >= v8.2.
  • Root SSH access from the local terminal to both the server and the gateway (see GATEWAY_ROOT_SSH and SERVER_ROOT_SSH config vars).

Steps performed

If the prerequirements are met and the configuration (see below) is made, all necessary steps to set up the tunnel will be performed automatically:

  • if the user $SERVER_USERNAME does not exist on the server
    • create its account there (without login)
  • if the key $SERVER_KEYFILE does not exist on the server
    • generate it as unencrypted ed25519 keypair on the server
    • if user $GATEWAY_USERNAME does not exist on the gateway
      • create its account there (with SSH login)
      • restrict the users SSH access to port forwards, via config added to gateway:/etc/ssh/sshd_config(.d/)
    • safe the generated public key in gateway:~/.ssh/authorized_keys
  • test the SSH connection with the new accounts and keys, establish trust on first contact
  • install AutoSSH (unless it's already on the $PATH)
  • write systemd service config with all parameters
  • enable and start the service

Notes / Warnings

  • This explicitly bypasses firewalls, make sure any exposed services are configured securely on the server side, e.g. when exposing port 22 / the SSH server, require keypair based authentication and set strong or disable passwords.
  • Using this to bypass the technical restrictions of a NAT is probably fine, using it to tunnel into a secure intranet likely isn't. Check against any local IT security guidelines and use at your own responsibility!

Configuration

Edit the configuration variables below and paste/run them in a local shell that has SSH set up to log in as root to both the server and the gateway (where it doesn't say TODO, the value can likely remain unchanged):

# . <(cat << "#EOF" # copy from after the first #
# DNS name or public IP of the gateway
GATEWAY_HOSTNAME=TODO
# Port number at which gateway has its own SSH server running (which can not be any of the forwarded ports)
GATEWAY_SSH_PORT=22
# Name of the user on the gateway that receives the SSH connection and opens the ports to forward
GATEWAY_USERNAME=TODO # e.g. autossh-from-<server>
# Name of the server. Only used in this configuration, see below
SERVER_HOSTNAME=TODO
# Name of the user on the server that starts the SSH connection
SERVER_USERNAME=TODO # e.g. autossh-to-<gateway>
# Private key file on the server to log in as $GATEWAY_USERNAME@$GATEWAY_HOSTNAME
SERVER_KEYFILE=/home/$SERVER_USERNAME/.ssh/autossh_ed25519
# Comment to add to the new key, if one is generated.
NEW_KEY_ARGS="-t ed25519 -C 'autossh from $SERVER_USERNAME@$SERVER_HOSTNAME to $GATEWAY_USERNAME@$GATEWAY_HOSTNAME'"
# Service names and ports to be forwarded, as mapping of name to SSH `-R`, `-L` or `-D` option.
# For later administration, it is recommended to only do one forward per service.
# `-R` opens a LISTEN port on the gateway and forwards incoming connections to the server, format `-R '<interface to listen on on gateway>:<port on gateway>:<address to connect to on server>:<port to connect to>'`, where the first element is optional and should likely be empty or `*` for all interfaces on the gateway, and the address to connect to should likely be `localhost`, i.e. the server.
# `-L` does the same, only swapping "gateway" and "server".
declare -A SERVICES=( # TODO: these are just examples, replace them!
    [autossh-expose-ssh]="-R '*:8022:localhost:22'"
    [autossh-expose-http]="-R '*:8080:localhost:80'"
    [autossh-get-dns]="-R 'localhost:53:localhost:53'" # TCP only, though
    [autossh-socks5-to-$GATEWAY_HOSTNAME]="-D localhost:1234"
)
# SSH commands to log in to the respective devices. Either one could also be "bash -c" to target the local host.
GATEWAY_ROOT_SSH="ssh -p $GATEWAY_SSH_PORT root@$GATEWAY_HOSTNAME -q -T"
SERVER_ROOT_SSH="ssh root@$SERVER_HOSTNAME -q -T"
# The server initiates the SSH connection, but when clients connect after idling, only the gateway would detect if it was interrupted. So the server must make sure the connection stays up. The gateway must also ping to know when to free ports it is listening on, so that new SSH connections can open them again.
SSH_ALIVE_INTERVAL=10 # Time between pings in seconds.
SSH_ALIVE_COUNT_MAX=3 # Number of consecutive pings that may be not replied to, before disconnecting.
#  `GatewayPorts`: Can be set to `no` if no `-R` forwards are used or their bind addresses on the gateway are `localhost`, otherwise it has to be `clientspecified` to allow other binds.
SSHD_GATEWAY_PORTS=clientspecified # TODO: adjust to the `SERVICES` specified
# `PermitListen`(available from OpenSSH 7.7): Restricts what `-R` can listen on. `none`, `any`, or a space separated list of `[host:]port` with `*` wildcards allowed in `host` and `port`.
SSHD_PERMIT_LISTEN='*:8022 *:8080 localhost:53' # TODO: adjust to the `SERVICES` specified
# `PermitOpen`: Restricts what `-L` and `-D` can connect to. `any` or a (non-empty) space separated white list of `host:port` (but no black list).
SSHD_PERMIT_OPEN=any # TODO: adjust to the `SERVICES` specified
# Note that the `sshd_config` will only be appended to, and only if $GATEWAY_USERNAME does not yet exist.
#EOF
);

Setup and installation

After setting the variables above, run this:

# { (. <(cat << "#EOF" # copy from after the first #
#!/usr/bin/env bash
set -eu

# autossh options:
#     -M 0: let ssh itself do heartbeats
#     -oServerAlive*: ssh heartbeats, ping every Interval sec, disconnect if more than CountMax pings await reply
#     -oExitOnForwardFailure=yes: exit (and retry) if forwarding fails
#     -N -q: no shell, quiet
GENERIC_OPTIONS="-M 0 -oServerAliveInterval=$SSH_ALIVE_INTERVAL -oServerAliveCountMax=$SSH_ALIVE_COUNT_MAX -oExitOnForwardFailure=yes -N -q"
# AUTOSSH_GATETIME=0 below makes AutoSSH not exit if the first connection fails

## generates a script for remote execution that captures a list of named global variables
function with {
    cmd=$(cat); # read actual script from stdin (must be done first)
    echo 'true;'; # make sure that ssh uses all of the following as its [command]
    for name in $@; do # for each variable name as argument
        # capture the value and "bind" it to the script
        #printf "%s=%q\n" $name "${!name}";
        declare -p $name;
    done;
    printf '\n%s' "${cmd}"; # paste the script itself
}

# make sure SERVER_USERNAME exists
$SERVER_ROOT_SSH "$(with SERVER_USERNAME << "#EOS"
    if ! id -u $SERVER_USERNAME ; then
        echo "creating user $SERVER_USERNAME on server"
        adduser --system $SERVER_USERNAME --gecos '' --group --disabled-login
    fi
#EOS
)"

keyExists=0; $SERVER_ROOT_SSH "test -e '$SERVER_KEYFILE'" || keyExists=$?

if [ $keyExists -ne 0 ] ; then

    echo "generating keypair $SERVER_KEYFILE"
    $SERVER_ROOT_SSH "$(with SERVER_USERNAME SERVER_KEYFILE NEW_KEY_ARGS << "#EOS"
        su $SERVER_USERNAME --shell=/bin/sh --command="ssh-keygen -N '' -f '$SERVER_KEYFILE' -q $NEW_KEY_ARGS"
        # -t: type, -N: new_passphrase, -f: output_keyfile, -C comment, -q: quiet
#EOS
    )"
    publicKey=$($SERVER_ROOT_SSH "cat '$SERVER_KEYFILE.pub'")
    echo "public key is: $publicKey"

    $GATEWAY_ROOT_SSH "$(with GATEWAY_USERNAME SSHD_GATEWAY_PORTS SSHD_PERMIT_LISTEN SSHD_PERMIT_OPEN SSH_ALIVE_INTERVAL SSH_ALIVE_COUNT_MAX publicKey << "#EOS"
        set -eu

        configFile=/etc/ssh/sshd_config
        if sshd -V 2>&1 | grep -qP '^OpenSSH_(8[.]([2-9]|\d\d+)|9[.]|\d\d+[.])'; then
            configFile="${configFile}.d/user-${GATEWAY_USERNAME}.conf"
        fi

        # add user and its sshd_config
        if ! id -u $GATEWAY_USERNAME ; then
            echo adding configuration for $GATEWAY_USERNAME in $configFile
            printf '%s' "
            Match User $GATEWAY_USERNAME
                GatewayPorts $SSHD_GATEWAY_PORTS
                PermitListen $SSHD_PERMIT_LISTEN
                PermitOpen $SSHD_PERMIT_OPEN
                AllowTcpForwarding yes

                AllowAgentForwarding no
                X11Forwarding no
                PermitTunnel no
                ForceCommand /bin/echo 'This account can only be used for port forwards'; /bin/false
                ClientAliveInterval $(( SSH_ALIVE_INTERVAL * 12 / 10 + 1 ))
                ClientAliveCountMax $SSH_ALIVE_COUNT_MAX
            " | sed -r 's/^ {12}//' >> $configFile
            # see https://askubuntu.com/questions/48129/how-to-create-a-restricted-ssh-user-for-port-forwarding
            systemctl reload ssh
            echo "creating user $GATEWAY_USERNAME on gateway"
            # this user needs to be available for login, but it should not have a shell
            adduser --system $GATEWAY_USERNAME --gecos '' --group --disabled-password --shell /bin/false
        fi

        # add public key for user
        mkdir -p /home/$GATEWAY_USERNAME/.ssh/ && chown $GATEWAY_USERNAME:$GATEWAY_USERNAME /home/$GATEWAY_USERNAME/.ssh && chmod 700 /home/$GATEWAY_USERNAME/.ssh
        touch /home/$GATEWAY_USERNAME/.ssh/authorized_keys && chown $GATEWAY_USERNAME:$GATEWAY_USERNAME /home/$GATEWAY_USERNAME/.ssh/authorized_keys && chmod 600 /home/$GATEWAY_USERNAME/.ssh/authorized_keys
        echo "adding public key to /home/$GATEWAY_USERNAME/.ssh/authorized_keys"
        printf '%s\n' "$publicKey" >> /home/$GATEWAY_USERNAME/.ssh/authorized_keys

#EOS
    )"
    echo "keysetup done"
else
    echo "key $SERVER_KEYFILE exists, assuming it is set up on the gateway"
fi

# all the rest happens on the server:
$SERVER_ROOT_SSH "$(with GENERIC_OPTIONS GATEWAY_HOSTNAME GATEWAY_SSH_PORT GATEWAY_USERNAME SERVER_HOSTNAME SERVER_USERNAME SERVER_KEYFILE NEW_KEY_ARGS SERVICES SSH_ALIVE_INTERVAL SSH_ALIVE_COUNT_MAX SSHD_GATEWAY_PORTS SSHD_PERMIT_LISTEN SSHD_PERMIT_OPEN << "#EOS"
set -eu

# test the SSH connection and add the gateway to /home/$SERVER_USERNAME/.ssh/known_hosts
test=(su $SERVER_USERNAME --shell=/bin/sh --command="ssh -oStrictHostKeyChecking=no -p $GATEWAY_SSH_PORT -l $GATEWAY_USERNAME -i '$SERVER_KEYFILE' $GATEWAY_HOSTNAME -q false")
printf "testing connection with: %q %q %q %q\n" "${test[@]}"
code=0; "${test[@]}" || code=$?
if [[ $code == 1 ]]; then # all connection (or authentication) failures should result in 255
    echo 'test succeeded'
else
    echo "connection failed with $code"; exit 1
fi

# install AutoSSH
if ! which autossh ; then apt-get install autossh -y; fi
autosshPath=$(which autossh)


for serviceName in "${!SERVICES[@]}"; do
    portForwards="${SERVICES[$serviceName]}"
    #echo $serviceName="${SERVICES[$serviceName]}"; continue;

    # write systemd service config
    cat << EOC | sed -r 's/^ {8}//' > /etc/systemd/system/$serviceName.service
        [Unit]
        Description="autoSSH $GATEWAY_HOSTNAME $portForwards"
        After=network.target

        [Service]
        User=$SERVER_USERNAME
        Environment=AUTOSSH_GATETIME=0
        ExecStart='$autosshPath' $GENERIC_OPTIONS -p $GATEWAY_SSH_PORT -l $GATEWAY_USERNAME -i '$SERVER_KEYFILE' $GATEWAY_HOSTNAME $portForwards
        Restart=always

        [Install]
        WantedBy=multi-user.target

        ## Configuration used (for future reference):
        # $(declare -p GATEWAY_HOSTNAME)
        # $(declare -p GATEWAY_SSH_PORT)
        # $(declare -p GATEWAY_USERNAME)
        # $(declare -p SERVER_HOSTNAME)
        # $(declare -p SERVER_USERNAME)
        # $(declare -p SERVER_KEYFILE)
        # $(declare -p NEW_KEY_ARGS)
        # declare -A SERVICES='([$serviceName]="$portForwards")'
        # $(declare -p SSH_ALIVE_INTERVAL)
        # $(declare -p SSH_ALIVE_COUNT_MAX)
        # $(declare -p SSHD_GATEWAY_PORTS)
        # $(declare -p SSHD_PERMIT_LISTEN)
        # $(declare -p SSHD_PERMIT_OPEN)
EOC
    # 'After' could be set to 'network-online.target' as well, but 'network.target' works, and 'network-online.target' seems to have a number of disadvantages.
    # Could probably also use -t option (background) instead of 'Environment=AUTOSSH_GATETIME=0'.

    # enable and start service
    systemctl enable $serviceName.service
    systemctl daemon-reload
    systemctl start $serviceName

done
#EOS
)"

echo "AutoSSH service forwarding ${SERVICES[@]} to $GATEWAY_HOSTNAME"
#EOF
)); }

AutoSSH stub

This is optional and not recommended. If AutoSSH is available in the servers package sources, simply install that and skip this.

If AutoSSH is unavailable, the script below can be executed as root on the server to install a very basic replacement of AutoSSH:

# bash -c $(cat << "#EOF"
#!/usr/bin/env bash

mkdir -p /usr/local/bin/

cat << '#EOS' | sed -r 's/^ {4}//' > /usr/local/bin/autossh
    #!/usr/bin/env bash

    ##
    # Very simple AutoSSH stub. -M 0 is assumed and implied.
    ##

    echo "AutoSSH stub" # announce to the user or log that this is not actually AutoSSH

    cleanup () {
        #kill -s SIGINT $!
        kill $!
        exit 0
    }
    trap cleanup SIGINT SIGTERM

    params=() # move "$@", but strip "-M" and its value
    while [[ $# -gt 0 ]]; do
        if [[ "$1" == "-M" ]]; then
            shift; shift # ignore and skip argument
        else
            params+=("$1"); shift # move to $params
        fi
    done

    ssh=$(which ssh)

    echo > $ssh "${params[@]}" &

    while [ 1 ]; do
        $ssh "${params[@]}" &; wait $!
        sleep 1 &; wait $!
        echo reconnecting
    done
#EOS

chmod +x /usr/local/bin/autossh

#EOF
)); }

Troubleshooting

  • Read the command output, esp. look for error messages and "keysetup done" vs "key XXX exists" (to see whether the user account on the gateway was created).
  • Check systemctl status $serviceName on the server for status and logs.
  • Search lsof -i for the host SSH connection, from ssh $SERVER_USERNAME and a random port to $GATEWAY_HOSTNAME: $GATEWAY_SSH_PORT, to sshd $GATEWAY_USERNAME` with the same TCP addresses (but the server's may be NATed).
  • For -R forwards, search lsof -i on the gateway for sshd $GATEWAY_USERNAME *:<port_num> (LISTEN) (or whichever remote interface was specified instead of *).
    • If it shows as localhost:<port> (LISTEN) instead, check that GatewayPorts clientspecified is set in the servers sshd_config for the user.
  • Also check lsof -i on the server that -L and -D local forwards show as local servers. i.e. LISTEN.
  • Make sure the gateway has firewall exceptions for any exposed ports, e.g. ufw allow <port_num>/tcp (if UFW is used).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment