Accessing a subnet that is behind a WireGuard client using a site-to-site setup

WireGuard-site-to-site.md

WireGuard Site-to-Site

Accessing a subnet that is behind a WireGuard client using a site-to-site setup

Problem Summary

We want to access a local subnet remotely, but it is behind a NAT firewall and we can't setup port forwarding. Outgoing connections work, but all incoming connections get DROPPED by the ISP's routing policy.

Solution Summary

We'll create a site-to-site connection with WireGuard allowing us to access the local subnet on a remote device (smartphone, in this example) by connecting through a cloud server in the middle.

Working Example

First let's define our three hosts. They all have WireGuard installed.

A the Linux machine on the local subnet, behind the NAT/firewall
B the Linux cloud server (VPS, like an Amazon EC2 instance)
C a third WireGuard client; a smartphone in this example

Host 'A'

The Host A's /etc/wireguard/wg0-client.conf:

[Interface]
Address = 10.200.200.5/24                  
PrivateKey = <HOST 'A' PRIVATE-KEY>
ListenPort = 27836                         # optional; will be randomly assigned otherwise
DNS = 1.1.1.1                              # or your own DNS server if you're running one    

[Peer]
PublicKey = <PUBLIC KEY OF HOST 'B'>
Endpoint = host-b-fqdn.tld:51820
AllowedIPs = 0.0.0.0/0, ::/0

PersistentKeepalive = 25                   # to keep connections alive across NAT

Here's what we need to add to Host A's iptables rules, expressed as the commands you would use to ADD them:

# iptables -A FORWARD -i wg0-client -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Finally, we need to make sure IP forwarding is enabled in Host A's kernel:

$ sysctl net.ipv4.ip_forward=1

Host 'B'

Host B's /etc/wireguard/wg0.conf:

[Interface]
Address = 10.200.200.1/24
PrivateKey = <HOST 'B' PRIVATE KEY>
ListenPort = 51820

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE


# This is the peer that is on the private subnet that we want to access.
#
# Notice the AllowedIPs... without this part, WireGuard will drop the 
# packets destined for the HOST 'A' subnet.  AllowedIPs is acting like
# a routing table and ACL here.

[Peer]
PublicKey = <HOST 'A' PUBLIC KEY>
AllowedIPs = 10.200.200.5/32, 100.10.202.0/24

# The smartphone
[Peer]
PublicKey = <HOST 'C' PUBLIC KEY>
AllowedIPs = 10.200.200.3/32

# An additional peer...
[Peer]
PublicKey = <Additional peer pubkey>
AllowedIPs = 10.200.200.4/32

Like we did with Host A, IP forwarding must also be enabled on Host B:

$ sysctl net.ipv4.ip_forward=1

Host C

Host C's configuration file:

[Interface]
PrivateKey = <HOST 'C' PRIVATE KEY>
Address = 10.200.200.3/24
DNS = 1.1.1.1


[Peer]
PublicKey = <HOST 'B' PUBLIC KEY>
AllowedIPs = 0.0.0.0/0
Endpoint = host-b-fqdn.tld:51820
PersistentKeepalive = 25

You're finished.
Make sure WireGuard is running on both HOSTS A and B, and then on the smartphone (HOST C), after connecting to HOST B with WireGuard you should be able to ping 10.200.200.5.

Hi insdavm,
First of al thanks for your help! i have the feeling that i'm very close to get this working but I can't figure out what i'm missing...

1. Are the two RPi's on the same local subnet?
Yes they are. Although the final goal is to have the RPi's in different local subnets, for practical reasons the testing is now in the same subnet. I assume this does not make a difference, or does it? Both RPi's reach out independent to the EC2 node, so I would think it does not matter if they are in the same or different local subnet. Or is there something I overlook?

2. What IP are you trying to connect to (i.e., tunnel IP, local subnet IP)?
I try to connect via the tunnel IP's. The local IP's are 192.168.68 and 192.168.109. When using the local addresses, ping and ssh does work in both direcions.
The tunnel IP's are 100.64.0.101 and 100.64.0.102 (the EC2 node acts as wg server and has IP address 100.64.0.1)
Ping works between all 3 nodes.
ssh only from 100.64.0.1 to both 100.64.0.101 and 100.64.0.102
ssh does not work from 100.64.0.101 to 100.64.0.102
As mentioned in my first message, running an http server on 100.64.0.101 and chatting to it with telnet on 100.64.0.102 (and the other way around) does work...

3. Can you confirm SSH is listening on 0.0.0.0:22 on both RPi's? (sudo ss -tunlp)
this is confirmed:
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=564,fd=3))

Here how the configuration of the nodes look:
pi@rp1: ~ sudo wg show
interface: wg0
public key: vL2byDvX281fra3Xe9JSTu3CjHypL1UzHNfLPDCkI2A=
private key: (hidden)
listening port: 51505
fwmark: 0xca6c

peer: NuHDg0RykQ6hdevLxBaSlCuTuSD1QlxKiG2qmzUZywM=
endpoint: "external-ip-address-of-EC2-node":42840
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 18 seconds ago
transfer: 3.20 KiB received, 6.59 KiB sent
persistent keepalive: every 25 seconds

pi@rp2:~ $ sudo wg show
interface: wg0
public key: N//AIetJgc6W6AmU24sJeuewB2ZKuJbFX8VqzcbkYjA=
private key: (hidden)
listening port: 40374
fwmark: 0xca6c

peer: NuHDg0RykQ6hdevLxBaSlCuTuSD1QlxKiG2qmzUZywM=
endpoint:"external-ip-address-of-EC2-node":42840
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 35 seconds ago
transfer: 34.13 MiB received, 66.13 MiB sent
persistent keepalive: every 25 seconds

ubuntu@my_ec2:~$ sudo wg show
interface: wg0
public key: NuHDg0RykQ6hdevLxBaSlCuTuSD1QlxKiG2qmzUZywM=
private key: (hidden)
listening port: 42840

peer: qd+nyR96UNNQXuWqoYhMJ3QBSwhwyBAnaKLOzzNj2xM=
endpoint: "my-external-ISP-ip-address":48311
allowed ips: 100.64.0.102/32
latest handshake: 1 minute, 21 seconds ago
transfer: 360.70 KiB received, 103.08 KiB sent
persistent keepalive: every 25 seconds

peer: vL2byDvX281fra3Xe9JSTu3CjHypL1UzHNfLPDCkI2A=
endpoint: "my-external-ISP-ip-address":48221
allowed ips: 100.64.0.101/32
latest handshake: 1 minute, 22 seconds ago
transfer: 313.04 KiB received, 128.68 KiB sent
persistent keepalive: every 25 seconds

Problem solved!
because the connection seemed to be ok (ping and simple wevbserver did work) somehow i had the feeling that it had to do something with the MTU size. So I started with using ping with specific sizes and noticed that the default MTU size that wireguard uses (1500) could not be pinged:
ping 100.64.0.102 -s 1500 did not work.
Trying something smaller did work:
ping 100.64.0.102 -s 1400
So next i set "MTU = 1400" in the interface section of the wg config:

[Interface]
PrivateKey = xxxxxxx
MTU = 1400

And now ssh works :-)

It also appears that it is sufficient to have the MTU setting on the "initiating" side. But because all nodes can be initating an ssh session,
So I set the value to all client nodes.

It would be good to know how to handle this issue properly. For example, ssh between the EC2 node and the RPi's did already work without the MTU setting. When is a specific MTU setting required? Is this dependent on the kind of peers nodes?
Anyway I am happy to have a solution for my vpn with RPi's :-)

Great one! In my particular case i had to manually add a route on the host outside of the network pointing to the internal lan via the router connected to the wg server internal ip. Also on the local pc a route to the wireguard network via the local ip of the router that acts as an access server. Else, it wouldn't work.
Thanks!

Can you explain why

# iptables -A FORWARD -i wg0-client -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

and

$ sysctl net.ipv4.ip_forward=1

on host A? It should work without these, all 3 devices are in the same subnet.

I am confused, at first I assumed this is a guide about routing between client subnets, but in fact all three devices here are in the same subnet (10.200.200.0/24).

Can anybody elaborate, please?

Can you explain why

# iptables -A FORWARD -i wg0-client -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

and

$ sysctl net.ipv4.ip_forward=1

on host A? It should work without these, all 3 devices are in the same subnet.

They're not on the same subnet. Host A is acting as a "server" so we can access the private subnet that's behind it, so we have to use IP forwarding and NAT to translate those packets from the tunnel to the local (to host A), private subnet.

I am confused, at first I assumed this is a guide about routing between client subnets, but in fact all three devices here are in the same subnet (10.200.200.0/24).

They're not; that's the tunnel subnet.

Can you explain why

# iptables -A FORWARD -i wg0-client -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

and

$ sysctl net.ipv4.ip_forward=1

on host A? It should work without these, all 3 devices are in the same subnet.

They're not on the same subnet. Host A is acting as a "server" so we can access the private subnet that's behind it, so we have to use IP forwarding and NAT to translate those packets from the tunnel to the local (to host A), private subnet.

I am confused, at first I assumed this is a guide about routing between client subnets, but in fact all three devices here are in the same subnet (10.200.200.0/24).

They're not; that's the tunnel subnet.

Sorry, I was indeed confused. You are right, thanks!
Although, for even more representative demo you could rather ping something in 100.10.202.0/24 (A's local network) rather than A's IP on tunnel interface.
Cheers

Does all the WireGuard traffic go through host B or is it direct A-C after establishing the connection? I would like to achieve a direct A-C connection (something like ZeroTier) in order to avoid VPS extra traffic fees.

Does all the WireGuard traffic go through host B or is it direct A-C after establishing the connection? I would like to achieve a direct A-C connection (something like ZeroTier) in order to avoid VPS extra traffic fees.

I ended up using https://tailscale.com

Hi everybody,
I have a little problem with this config... please help me! Hope it may help others.
The question is that i want to have a site to site connection using different subnet for vpn tunnel (10.0.0.0/24) and for my home subnet (172.16.1.0/24).
With the following configs, my 'A' node does not allow me to access to other IPs behind it... is it normal? What should I check/add? Forward is done on both nodes.

this is my A node config:

[Interface]
Address = 10.0.0.186/32
PrivateKey = ...
DNS = 1.1.1.1

# as for the B node, I added here iptables route... is it ok?
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enol -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o enol -j MASQUERADE

[Peer]
PublicKey = ...
# here, I specify only the wireguard lan IPs, because I dont want that my A node send all traffic out via wg0
AllowedIPs = 10.0.0.0/24
PersistentKeepalive = 10
Endpoint = B-node-public-IP:51820

this is the result of ip r on node A:

default via 172.16.1.1 dev eno1
10.0.0.0/24 dev wg0 scope link
172.16.1.0/24 dev eno1 proto kernel scope link src 172.16.1.186

this is my B node config:

[interface]
Address = 10.0.0.1/32
ListenPort = 51820
PrivateKey = ...

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = ...
AllowedIPs = 10.0.0.186/32, 172.16.1.0/24
#PersistentKeepalive = 25

this is my A node config:

[Interface]
Address = 10.0.0.186/32

# as for the B node, I added here iptables route... is it ok?

this is my B node config:

[interface]
Address = 10.0.0.1/32
ListenPort = 51820

At least your Address config is only a /32. That means both devices are NOT on the same subnet. I would change that to /24. If you look closely on the example config above, it is quite the same there.

Thanks for the fast answer!
I will try but I dont understand why it should work... the problem is that I cannot reach the 172.16.1.0/24 subnet behind the node A. If I ping or access to server via 10.0.0.186 it works. Better, it works also if I access to the lan IP address of the node A server, 172.16.1.186, via tunnel, from host C: but if i traceroute another subnet ip 172.16.1.x, it stop after reaching the node A... There is something on node A misconfigured I think, dont know what :(

it doesnt work...

Thanks, very useful and clear enough! Is it possible for Host A to be a virtual router (like OPNsense or VyOS) ?
Allowing me to access a whole virtual LAN ?

Absolutely. My 'host A' is an OPNsense box and this config works like a charm. Make sure that you have the correct firewall rules on your OPNsense box

@euterklaas I have been trying to do this for a long time. My opnsense client connects and can be accessed on the wireguard network, but does not allow other wireguard clients opnsense lan access. Are you able to post some more info or contact me about your setup ?

@micktech

Hey there! I'm happy to help.
I have a vserver which basically acts as my gateway to the internet. My opnsense firewall at home connects to the vserver via wireguard and the clients on my local network use this connection to reach the internet. Additionally, there are some external clients, like my mobile phone or my parent's notebook which use the vserver as their gateway. The external clients are also able to reach the clients on my home network.
The wireguard configs are the following:

Node A ("main" node, vserver)

Address = 10.0.0.2/24
PrivateKey = [...]
ListenPort = 51820

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE


# This is the peer that is on the private subnet that we want to access.
#
# Notice the AllowedIPs... without this part, WireGuard will drop the
# packets destined for the HOST 'A' subnet.  AllowedIPs is acting like
# a routing table and ACL here.
# Home
[Peer]
PublicKey = [...]
AllowedIPs = 10.0.0.1/32, 192.168.100.0/24, 192.168.99.0/24

[Peer]
PublicKey = [...]
AllowedIPs = 10.0.0.3/32

[Peer]
PublicKey = [...]
AllowedIPs = 10.0.0.4/32

[Peer]
PublicKey = [...]
AllowedIPs = 10.0.0.5/32

Node B (opnsense firewall @home)

Address = 10.0.0.1/24
DNS = 192.168.100.1,192.168.100.254
ListenPort = 51820
PrivateKey = [...]
[Peer]
PublicKey = [...]
AllowedIPs = 0.0.0.0/0
Endpoint = nodeA:51820
PersistentKeepalive = 25

Node C (external client aka mobile phone)

Address = 10.0.0.3/24
DNS = 192.168.100.1, 192.168.100.254
PrivateKey = [...]

[Peer]
AllowedIPs = 0.0.0.0/0
Endpoint = nodeA:51820
PersistentKeepalive = 25
PublicKey = [...]

I hope this helps...

Thanks @euterklaas I appreciate your help,
I have exactly the same configuration except my opnsense wireguard client wont let 0.0.0.0/0 in the AllowedIPs of the endpoint section. If i add it, wireguard crashes and doesn't connect at all. I'm thinking it might be a bug. When I just leave AllowedIPs = (Wireguard network subnet) it connects and allows me to access just the wireguard IP from a mobile phone and not the internal network.

Sounds like I'll be doing some more debugging. Are you running the latest opnsense ? Thank you for getting back to me.

You're welcome. I'm running the current version, OPNsense 21.1.2-amd64.
Did you use the GUI to configure the opnsense client? Because when I first configured it, I had some difficulties using the GUI. So I created the config file manually via SSH. Just put the file wg0.conf under /usr/local/etc/wireguard/.
For the basic setup beforehand (setting up the forwarding and the wireguard interface) I used this guide.

Can someone help me understand one thing..
My wg subnet is 10.0.25.0.
My home subnet is 192.168.25.0.
Host A is 192.168.25.81, and I'm giving it 10.0.25.81 on the wg net.
Host B is 10.0.25.1
Host C is 10.0.25.50.

Let's say Host D now connecting and should be able to ping Host C at 10.0.25.50 and Host A at 192.168.25.81, but should it also be able to ping 192.168.25.100 (not a client running wg)?

Second question, with the above configuration, all of Host A's outgoing data is hitting the VPS, which is unnecessary. I believe it's because of AllowedIPs = 0.0.0.0/0, ::/0, but I'm not sure what to change it to. Help?

Hi everyone, I have a similar situation to all of the above but with some exceptions. Basically I'm trying to get to a web site hosted on a Arduino (MicroA) connected to a Rasperry Pi Access Point (RpiAP) which is WireGuard connected to a Rasberry Pi WireGuard VPN Server (RpiVPN) to my home network. The RpiAP is behind a Public WiFi router. All of this is working and I can VNC to the RpiVPN and then VNC to from there to the RpiAP and then browse the MicroA.

So understanding all this, I would like to setup a port on my home firewall (aaa.aaa.aaa.aaa:xxx) which if accessed (the easy part), will forward to the RpiVPN (192.168.1.22)(wireguard 10.6.0.1) which will then route to the RpiAP (10.6.0.2) subnet(10.10.10.1) which with then route to the MicroA (10.10.10.180).

Allowing me to get to the MicroA from anywhere Public.

This All seems doable but I lack the skill to setup all the tables (ipTables and Wireguard).

Can anyone offer some help.

Thanks.

Lifesaver. Working as of this date.

Thanks so much for the gist. I've been using this to connect to my local home automation setup for a while without issue via Phone > VPS > VM > LAN. I'm now looking to use this to connect to a different LAN, hoping to keep using the same VPS to save myself an extra $5/mo and I was hoping you could help me understand the iptables bit of the config on Host B.

My current wg network runs 10.200.200.0/24. I've managed to set up another instance listening on a different port in the 10.300.300.0/24 range, but I'm not sure what (if anything) needs to change about the forwarding rules for my public host to forward the traffic correctly.

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE

Will cloning the above work since anything that needs to be forwarded would already be on the 300 network, or will it break things because Host B also has an interface to the 200 network? I'm not sure how %i gets interpreted here. Normally I'd just try it and see but I don't want to bork the remote connection and lock myself out.

This is not so great for security. You wanted to connect Host A and Host C, presumably both under your physical control, but your solution opens both of these hosts to a third one, Host B, which is a cloud server, controlled by a third party. This setup allows anyone who can gain physical access to Host B (for instance, all the admins and hackers of Amazon EC2), to be able to connect to your Host A and Host C and decrypt all traffic between them.

I would prefer a solution which only uses Host B as a relay that cannot decrypt the traffic. AFAIK this could be achieved either by using 2 instances of WireGuard (which is a bit silly); or some kind of traffic forwarder like socat, a simple unencrypted tunnel or similar.

I'm curious if this could work using just 1 instance of WireGuard and proper forwarding. Host A would need to have Host C's public key, Host C would need to have Host A's public key, but I'm not sure what AllowedIPs and Endpoints they would have.

The point is not to create a one-sizs-fits-all solution, but to demonstrate how a private subnet that sits behind a NAT can be accessed when port forwarding is not an option. You are assuming Host B is a rented cloud server. But it can be anything you want, and if your concern is Host B's security, then just make Host B something you own and control. Whether or not you have sole physical access to it is up to you and how you design your system. For me, the intent is to encrypt the connection between A and C, and work around the inability to port-forward through the NAT router. The hardening of my cloud instance is another matter and outside the scope of this example.

The point is not to create a one-sizs-fits-all solution, but to demonstrate how a private subnet that sits behind a NAT can be accessed when port forwarding is not an option.

If given a solution that works, and a solution that works and is both faster (because Host B doesn't have to decrypt and re-encrypt all traffic between A and C) and more secure, people should generally opt for the latter.

You can demonstrate the exact same thing (how a private subnet that sits behind a NAT can be accessed when port forwarding is not an option) without introducing a potential security issue.

You are assuming Host B is a rented cloud server.

Well, it was you who originally assumed it.

if your concern is Host B's security, then just make Host B something you own and control

Not quite, my concern is the security of traffic going from A to C and from C to A. The whole purpose of this set up was to connect host A to host C, not to enable host B to spy on their traffic. So it should also be your concern.

Even if Host B was something I owned and controlled, there is still no need for it to be able to read or re-encrypt traffic between A and C.

For me, the intent is to encrypt the connection between A and C, and work around the inability to port-forward through the NAT router.

Exactly my point. You wanted to encrypt the connection between A and C, but instead you encrypted connection between A and B and between B and C - therefore, you didn't achieve your intent.

This doesn't just work around the inability to port-forward through the NAT router, it introduces a security problem along the way.

The hardening of my cloud instance is another matter and outside the scope of this example.

I didn't talk about hardening of your cloud instance, I talked about properly securing the connection between A and C.

Cool man. If you don't like it, don't use it. Figure it out and post your Gist here if you think you have a better solution.

I don't yet have a straightforward solution. It appears you either need two instances of WireGuard, some other tunnel (like GRE or IPIP or sth), or a more full-fledged option like ZeroTier, Nebula or Tailscale.

Hi everybody,
I'm getting crazy, it's two days I'm making changes and reading things, I almost wanna cry.
I've used these configs several time, but now I cannot replicate it on a debian machine.
If In the debian node A config file I added into the allowedIPs the subnet 172.16.1.0/24 to reach it, when I start the interface I obtain:
RTNETLINK answers: File exists
I have seen the ip route table and there is a route to this subnet:
172.16.1.0/24 dev enp2s0 proto kernel scope link src 172.16.1.127
But how can remote? and moreover, Should I? I tried some commands around but it seems that if I del it, the network stopped to work and I have to restart the machine.

This is the full file:

[Interface]
PrivateKey = 
Address = 10.0.10.2/32
ListenPort = 55610

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o enp2s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp2s0 -j MASQUERADE

# VPS
[Peer]
PublicKey = 
PresharedKey = 
AllowedIPs = 10.0.10.0/24,172.16.1.0/24
PersistentKeepalive = 10
Endpoint = 1.2.3.4:55610

Hi guys!
I am trying to accomplish almost the same, except that Instead of accessing the home network, as per this solution, I want to "exit" the mobile wg-client's internet traffic through "home" device A!
The mobile unit C must be using the A internet connection and have, of course, the external IP accordingly.
Any ideas?

Hi,

thank you for these instructions. I am using this setup to get along with my IPV6 dual stack lite internet connection. So Host A is a debian server in my home network, Host B a Cloud Server and C is any Wireguard client. My only problem is that the upload and download performance (data from and to host A and the home network) is quite poor. The internet connection should make 400 MBit upload as well as the Cloud Server, but the real performance is much, much below (about 18 MBit).

Any idea/suggestion to improve that (and to measure the improvement)?

Thank you!

Just in addition: Using an iperf3 client and server between the cloud server and a local server in my home network results in about 380 Mbits/sec. I am still wondering why the speed to other Wireguard clients is so low.

Took me a while to figure out I need to add

iptables -A FORWARD -i eth0 -j ACCEPT

on the client in order to access resources in the client network from the server....