Skip to content

Instantly share code, notes, and snippets.

@drmalex07
Last active March 15, 2024 15:56
Show Gist options
  • Save drmalex07/c0f9304deea566842490 to your computer and use it in GitHub Desktop.
Save drmalex07/c0f9304deea566842490 to your computer and use it in GitHub Desktop.
Setup a secure (SSH) tunnel as a systemd service. #systemd #ssh #ssh-tunnel #ssh-forward

README

Create a template service file at /etc/systemd/system/secure-tunnel@.service. The template parameter will correspond to the name of target host:

[Unit]
Description=Setup a secure tunnel to %I
After=network.target

[Service]
Environment="LOCAL_ADDR=localhost"
EnvironmentFile=/etc/default/secure-tunnel@%i
ExecStart=/usr/bin/ssh -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -L ${LOCAL_ADDR}:${LOCAL_PORT}:localhost:${REMOTE_PORT} ${TARGET}

# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

We need a configuration file (inside /etc/default) for each target host we will be creating tunnels for. For example, let's assume we want to tunnel to a host named jupiter (probably aliased in /etc/hosts). Create the file at /etc/default/secure-tunnel@jupiter:

TARGET=jupiter
LOCAL_ADDR=0.0.0.0
LOCAL_PORT=20022
REMOTE_PORT=22

Note that for the above to work we need to have allready setup a password-less SSH login to target (e.g. by giving access to a non-protected private key).

Now we can start the service instance:

systemctl start secure-tunnel@jupiter.service
systemctl status secure-tunnel@jupiter.service

Or enable it, so it get's started at boot time:

systemctl enable secure-tunnel@jupiter.service
@sanludhi
Copy link

I was struggling... thanks for saving so much of time ... it worked perfectly fine

@sanludhi
Copy link

How to forward multiple ports

@htfy96
Copy link

htfy96 commented Sep 21, 2018

Maybe it's better to set it as a user service

@dhruv
Copy link

dhruv commented Nov 7, 2018

Thank you for this!

@sunnyszy
Copy link

I also think it's better to exec in user mode. But my machine systemctl has bug when running --user, so I add user permission in service file

@linuxmalaysia
Copy link

@maykel535
Copy link

I used the -R instead -L because did not works...

@siliconhippy
Copy link

Can this be modified to set up reverse SSH for multiple hosts behind firewalls, with as much automation as possible?

E.g., the port #s on the middleman machine represent each individual host, with the port #s left as variable in the host script. This variable port# determined at client- middleman login time and displayed, from within a pool of port #s.

The keygen should be done on middleman machine, and the public keys copied to target host machines apriori.

@liloew
Copy link

liloew commented May 16, 2019

Helpful a lot, thanks.

@stiv-yakovenko
Copy link

Thank you friend, I was missing: -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes

@Blaimi
Copy link

Blaimi commented Jan 20, 2020

I needed to have a ssh-connection to some jump-host to connect to a mysql-database so I've changed it a little bit to have more options and use the native syntax of the config file to configure the service.

You can configure whatever ssh can, even multiple port forwardings, LocalForwardings, RemoteForwardings, etc.

[Unit]
Description=Setup a secure tunnel to %I
After=network.target

[Service]
ExecStart=/usr/bin/ssh -F /etc/default/secure-tunnel.config -NT %i

# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

contents of /etc/default/secure-tunnel.config

Host example
   HostName example.com
   User some-user
   IdentityFile /etc/some_folder/id_rsa.some-user.examle.com
   LocalForward 127.0.0.1:4306 dbserver.example.com:3306
   ServerAliveInterval 60
   ExitOnForwardFailure yes

to enable the service, you can use the name given in the Host-line (systemctl enable --now secure-tunnel@example)

@lsaavedr
Copy link

lsaavedr commented Feb 3, 2020

@Blaimi thanks!

@dixon1e
Copy link

dixon1e commented Mar 17, 2020

Thank you, this is extremely helpful and a great time saver.

@antortjim
Copy link

Yet another thank you from a thankful beginner sysadmin!
I arrived here after spending a while trying to get a script running port forwarding with ssh via a systemd service like this:

ssh -fNL dest_port:127.0.01:source_port SERVER

The script can be executed interactively (./script.sh) but it won't work when run in systemd. I confirm the solution is your answer!

@fcjbispo
Copy link

fcjbispo commented Apr 1, 2020

Hi, i am using the version that uses the /etc/default/secure-tunnel.config file and able to get connection when i run the command from shell. But when i try to start it from systemctl (on a Debian 9 system), i receive a failed response as follow:

secure-tunnel@ackt0.service - Setup a secure tunnel to ackt0
Loaded: loaded (/etc/systemd/system/secure-tunnel@.service; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2020-04-01 14:51:07 UTC; 1s ago
Process: 10744 ExecStart=/usr/bin/ssh -F /etc/default/secure-tunnel.config -NT ackt0 (code=exited, status=255)
Main PID: 10744 (code=exited, status=255)
Apr 01 14:51:07 pfmw-traveller1 systemd[1]: secure-tunnel@ackt0.service: Unit entered failed state.
Apr 01 14:51:07 pfmw-traveller1 systemd[1]: secure-tunnel@ackt0.service: Failed with result 'exit-code'.

Does someone can help me to figure out what this status 255 means? Thanks.

@antortjim
Copy link

journalctl -ru secure-tunnel@ackt0.service
you will see more details into what went wrong, as it shows logs including Exceptions like the one you are getting. It goes from most recent to oldest logs as you scroll down

@fcjbispo
Copy link

fcjbispo commented Apr 1, 2020

thanks!

@fcjbispo
Copy link

fcjbispo commented Apr 1, 2020

Hi guys,

To avoid any error due to host key checking on startup of the service, add the -o StrictHostKeyChecking=no on ssh command line or equivalent on secure-tunnel.config. I did put it on the ssh call like this:

[Service]
ExecStart=/usr/bin/ssh -o StrictHostKeyChecking=no -F /etc/default/secure-tunnel.config -NT %i

@gushauptfleisch
Copy link

@drmalex07
SUPERB
Your detail has saved me a lot of digging - thank you!

@RestOp
Copy link

RestOp commented Aug 12, 2020

Hello,

i have at least the same example


[Unit]
Description=Tunnel for HA
After=network.target

[Service]


ExecStart=/usr/bin/ssh -i "/home/pi/.ssh/id_rsa" -p 22 -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -R 31281:localhost:8123 login@server.com

RestartSec=5s
Restart=always

[Install]
WantedBy=multi-user.target

But it do not start. i get
image

It also do not shows any logs with command
image

PLease advice me what have i've done wrong?

Thanks a lot!

@MartianRover
Copy link

So, I followed your example, got everything working as far as I can tell.
Some issues I had as others mentoned above but it was just because I didn't have my ssh keys in the root users .ssh directory.
So I have my RPi4 (which is on my home network) connected to my DigitalOcean droplet and I have my laptop connected to the same droplet but from my wireless hotspot.
The service is running on both machines.... but now what.

This is probably a stupid question but how do I actually interact with my RPi4 from my laptop while on a different network?
Can I access my RPi4's web server from my laptop that way? If so, how?
How do I access it via the CLI? Can I access the desktop with VNC using this tunnel?

Thanks for any help or pointers. I'm not sure what to ask google to get the answers I'm looking for.

@deepu-abraham
Copy link

Thank you

@dazz100
Copy link

dazz100 commented Jul 22, 2021

Hi
I think
RestartSec=5
Needs to be greater than ServerAliveInterval or and ClientAliveInterval, especially for tunnels.
To prevent the possibility of trying to open a failed SSH connection at one end (AA), while the other end (BB) still thinks the connection is alive.
The effect is that the failed end (AA) tries to open a new connection, which is blocked by the half dead connection at the other end (BB).
I think 'RestartSec' should be long enough that both ends know their connection is broken.

Worst case scenario.
ServerAliveInterval = 10
ClientAliveInterval = 3
End AA sends a packet that is received by BB but the connection is blocked before a reply is received by AA.
AA detects a failed linkand will close the connection in 30sec.
BB received the packet from AA and thinks the connection is still open.
After 10sec, BB sends a packet, with no reply.
BB now detects a failed link and will close the connection in 30sec. This is 10sec later than AA.
In that 10s period, AA would try to open a new connection that would fail because it is blocked by BB.

Solution for tunnels
Set:
ExitOnForwardFailure yes
RestartSec >> ServerAliveInterval
Same applies to ClientAliveInterval

where: >> means "much greater than"

This will ensure that both ends have killed their ssh connections before trying to re-open them.
The TCP protocol should flag a sent packet missing an ACK but I have experienced problems with blocked SSH links.

@dazz100
Copy link

dazz100 commented Jul 22, 2021

Hi, i am using the version that uses the /etc/default/secure-tunnel.config file and able to get connection when i run the command from shell. But when i try to start it from systemctl (on a Debian 9 system), i receive a failed response as follow:

secure-tunnel@ackt0.service - Setup a secure tunnel to ackt0
Loaded: loaded (/etc/systemd/system/secure-tunnel@.service; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2020-04-01 14:51:07 UTC; 1s ago
Process: 10744 ExecStart=/usr/bin/ssh -F /etc/default/secure-tunnel.config -NT ackt0 (code=exited, status=255)
Main PID: 10744 (code=exited, status=255)
Apr 01 14:51:07 pfmw-traveller1 systemd[1]: secure-tunnel@ackt0.service: Unit entered failed state.
Apr 01 14:51:07 pfmw-traveller1 systemd[1]: secure-tunnel@ackt0.service: Failed with result 'exit-code'.

Does someone can help me to figure out what this status 255 means? Thanks.

Hi
Try specifying a username or/and the path to the keys in the ssh command. It is likely that the service ssh command is being run as root and looking for the keys in the /etc/ssh/ dir. You are likely to be getting authentication errors.

@wpyoga
Copy link

wpyoga commented Aug 12, 2021

ExecStart=/usr/bin/ssh -o StrictHostKeyChecking=no -F /etc/default/secure-tunnel.config -NT %i

@fcjbispo instead of passing StrictHostKeyChecking=no, I would recommend that you first run the ssh command, accept the host key, and then start the service

Passing StrictHostKeyChecking=no opens you up to man-in-the-middle attacks.

@Cimplex
Copy link

Cimplex commented Sep 1, 2021

Thank you

@askrabal
Copy link

askrabal commented Jan 7, 2022

Thank for this, I've used adaptations of this a few times now. The latest was a dynamic proxy so that apt and web traffic could all be tunneled though a jump host.

@suxiaojin
Copy link

Hello,

i have at least the same example


[Unit]
Description=Tunnel for HA
After=network.target

[Service]


ExecStart=/usr/bin/ssh -i "/home/pi/.ssh/id_rsa" -p 22 -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -R 31281:localhost:8123 login@server.com

RestartSec=5s
Restart=always

[Install]
WantedBy=multi-user.target

But it do not start. i get image

It also do not shows any logs with command image

PLease advice me what have i've done wrong?

Thanks a lot!

Hi, did you solve this problem ?

@RestOp
Copy link

RestOp commented Mar 18, 2022

@ranomier
Copy link

ranomier commented Sep 2, 2022

I also think it's better to exec in user mode. But my machine systemctl has bug when running --user, so I add user permission in service file

@sunnyszy I forked aswell and added a littlebit of stuff:

https://gist.github.com/ranomier/2e6ae91fa26cc0c0d6557238a8f60764

@dbelkovsky
Copy link

Greetings colleagues. Tell me what I need to add in the config. to make my unit work.
I'm setting up ssh sock-proxy.
In the terminal, the command looks like this:
ssh -D 0.0.0.0:8080 user@remoteserver -i /home/user/.ssh/id_rsa

@MGLiMlsg
Copy link

What is the alternative to this on Windows? Or am I missing something

@stiv-yakovenko
Copy link

What is the alternative to this on Windows? Or am I missing something

Install cygwin and you get it on Windows.

@Robin479
Copy link

Robin479 commented Aug 3, 2023

Instead of setting up the service template to a fixed set of environment variables and to use distinct environments, why not setup the service with minimal default options like timeouts and disabling password prompt, and to use distinct ssh_config.d profiles instead, e.g. /etc/systemd/system/ssh-client@.service:

[Unit]
Description=Auto-SSH to %i
After=network.target

[Service]
ExecStart=/usr/bin/ssh -NT -o PasswordAuthentication=no -o TCPKeepAlive=yes -o ServerAliveInterval=60 %i
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

...and then define your ssh_config in /etc/ssh/ssh_config.d/auto-ssh (name doesn't actually matter):

Host myalias
    User myuser
    HostName hostname.example.org
    ProxyJump other@jumphost.example.org
    …

...and then activate it:
$ systemctl start ssh-client@myalias.service

...or enable autostart, respectively:
$ systemctl enable ssh-client@myalias.service

@Blaimi
Copy link

Blaimi commented Aug 3, 2023

I'm using this concept since a while ;)

I use this especially to open SOCKS proxies (DynamicForward) to access private kubernetes-API-endpoints via bastion-hosts configured as --user services.

https://gist.github.com/drmalex07/c0f9304deea566842490?permalink_comment_id=3145418#gistcomment-3145418

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment