Skip to content

Instantly share code, notes, and snippets.

@zOrg1331
Last active April 22, 2024 11:37
Show Gist options
  • Star 51 You must be signed in to star a gist
  • Fork 10 You must be signed in to fork a gist
  • Save zOrg1331/a2a7ffb3cfe3b3b821d45d6af00cb8f6 to your computer and use it in GitHub Desktop.
Save zOrg1331/a2a7ffb3cfe3b3b821d45d6af00cb8f6 to your computer and use it in GitHub Desktop.
wireguard, wireguard layer 2, wireguard over TCP

Intro

This note describes how to connect two networks/devices/VMs over public network using Wireguard with Layer 2 support (ARP, IPv6 link-local, etc).

This can also be achieved using SSH and its "tap" tunnel, however, it does not provide the same level of latency and bandwidth as full-blown VPN such as Wireguard.

In addition, this note describes how to tunnel Wireguard over TCP connection. This may be of use if you encounter firewall in-between so, for instance, you can use TCP port 443 only.

Objective

Let's draw a network diagram of what we would like to have in as a result.

 Host on private LAN                                         Host on the Internet
 +---------------------------------+                       +-------------------------------+
 |   brtun bridge                  |                       |                  brtun bridge |
 | +-------------+                 |                       |                 +--------+    |
 | | ethX gretun |<->wg0<->udp2raw | <-Internet, TCP 443-> | udp2raw<->wg0<->| gretun |    |
 | +-------------+                 |                       |                 +--------+    |
 +---------------------------------+                       +-------------------------------+
 
 brtun: 192.168.0.200/24                                    brtun: 192.168.0.50/24
 wg0: 12.12.12.2/24                                         wg0: 12.12.12.1/24
 gretun: 12.12.12.2->12.12.12.1                             gretun: 12.12.12.1->12.12.12.2
 

Setting things up

Distinguishing between server and client here is based on the answer to the question: who can connect to Wireguard listener socket.

Server

Generating a key:

wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivs
ip a a 12.12.12.1/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubc) allowed-ips 0.0.0.0/0

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.1 remote 12.12.12.2
ip l s gretun up

Configuring a bridge which includes GRETAP interface. This step is in fact optional, but it allows you some flexibility: you can assign an IP address to this bridge interface and/or you can add additional interfaces to this bridge.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's assign an IP address from the target network to this bridge:

sudo ip a a 192.168.0.51/24 dev brtun

If we want Wireguard over TCP, launch UDP2RAW listener which will do its magic. You can get UDP2RAW from this Git repo: https://github.com/wangyu-/udp2raw-tunnel/releases/

./udp2raw_amd64 -s -l YOUR_PUBLIC_IP:443 -r 127.0.0.1:51820 -a --fix-gro

Client

Client is very similar to the server described above.

Generating a key:

wg genkey | tee wgkeyprivc | wg pubkey > wgkeypubc

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivc
ip a a 12.12.12.2/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key, endpoint address and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint YOUR_PUBLIC_IP:51820 persistent-keepalive 15

In case you want Wireguard over TCP, set endpoint address to a localhost interface:

wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint 127.0.0.1:7777 persistent-keepalive 15

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.2 remote 12.12.12.1
ip l s gretun up

Configuring a bridge which includes GRETAP interface.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's add a physical interface belonging to a LAN network to the bridge and start DHCP client:

ip l set ethX master brtun
dhcpcd ethX

If we want Wireguard over TCP, launch UDP2RAW listener on the localhost interface which connects to the corresponding public listener.

./udp2raw_amd64 -c -l 127.0.0.1:7777 -r YOUR_PUBLIC_IP:443 -a --fix-gro

Results

That's it. Now you will have Layer 2 connectivity between two separated machines over Wireguard VPN (optionally, over TCP). Enjoy your ARP and IPv6 link-local experience.

References

@mrbluecoat
Copy link

I think cat wgkeypubc is supposed to be cat wgkeypubs

@legale
Copy link

legale commented Nov 15, 2022

@zOrg1331 помоги разоюраться. Неделю ковыряюсь. Никак не заставлю туннель работать.

Есть сервер для туннеля (debian11) клиент точка доступа openwrt 21.02.

На сервере делаю так:

export ADDR=$(ip addr show dev ens192 | grep -Eo 'inet [0-9./]+' | cut -d ' ' -f 2)
export MAC=$(cat /sys/class/net/ens192/address)
export GW=$(ip ro show dev ens192 | grep -Eo 'via [0-9./]+' | cut -d ' ' -f 2)

ip link add br0 type bridge
ip link set ens192 master br0
ip link set dev br0 address $MAC
ip addr add $ADDR dev br0
ip link set br0 up
ip ro del default dev ens192
ip route add default via $GW dev br0
ip addr flush dev ens192
ip link add gre1 type gretap local 10.66.66.1 remote 10.66.66.2
ip link set gre1 master br0

10.66.66.1 это работающий туннель wg.

На клиенте тоже делаю бридж и к нему добавляю gre интерфейс.
вот что показывает ip link show

43: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1368 qdisc noqueue state UP mode DEFAULT group default qlen 1000  link/ether d6:9f:22:36:47:f6 brd ff:ff:ff:ff:ff:ff
45: gre4t-gre@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1368 qdisc fq_codel master br-lan state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 9a:cd:bc:67:dd:56 brd ff:ff:ff:ff:ff:ff

ip link show master br-lan

44: wlan0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP mode DEFAULT group default qlen 4096
    link/ether 44:d1:fa:bc:0e:60 brd ff:ff:ff:ff:ff:ff
45: gre4t-gre@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1368 qdisc fq_codel master br-lan state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 9a:cd:bc:67:dd:56 brd ff:ff:ff:ff:ff:ff

В итоге по dhcp на клиенте на br-lan интерфейс получаю ip адрес из сети на другом конце туннеля. Пинг работает. А вот wget просто повисает в раздумьях, а потом вышибает по таймауту.

Что я пропустил? Куда копать?

@AndrewL733
Copy link

Hello, do you by any chance have instructions that use Ubuntu’s netplan instead of “ip” commands? Or do you know anybody who has created such instructions?

@zOrg1331
Copy link
Author

Hello, do you by any chance have instructions that use Ubuntu’s netplan instead of “ip” commands? Or do you know anybody who has created such instructions?

Sorry, I never used netplan and even do not have any contacts with a person who might do that on such advanced level.

Probably you could try to use an alternative approach like custom atartup servoce which does this network configuration for you? I doubt network configuration systems like netplan are suitable to offer support for non-standart setups.

@AndrewL733
Copy link

@zOrg1331 Thanks for your reply. Can you explain whether the configuration you have described would create a site-to-site bridge? The answer seems to be “no" but maybe I’m missing something. My ultimate goal is to create a site-to-site Wireguard VPN with layer 2 support, such that all computers at Site A can communicate with all computers at Site B, and so that all broadcast traffic goes over the VPN. I am currently able to achieve this with OpenVPN TAP and also with ZeroTier (about 1.3x the performance as OpenVPN). It would seem very useful to be able to use Wireguard for the same purpose.

@zOrg1331
Copy link
Author

@zOrg1331 Thanks for your reply. Can you explain whether the configuration you have described would create a site-to-site bridge? The answer seems to be “no" but maybe I’m missing something. My ultimate goal is to create a site-to-site Wireguard VPN with layer 2 support, such that all computers at Site A can communicate with all computers at Site B, and so that all broadcast traffic goes over the VPN. I am currently able to achieve this with OpenVPN TAP and also with ZeroTier (about 1.3x the performance as OpenVPN). It would seem very useful to be able to use Wireguard for the same purpose.

Well, as for me, the whole purpose of this exercise was to link two networks on L2 level by using Wireguard.

Side note: I used it for a quite non-standard purpose. During a penetration test I had to establish a tunnel towards a corporate network via a corporate firewall and have L2 communications at the same time. By following the described method I had transparent (for userspace tools) L2 communications between a Wireguard client (remote VM) and the remote network. So the only difference between site-to-site VPN was that I did not have network with multiple hosts on one side.

Once you have GRE L2 tunnel working over Wireguard, the rest is up to you and your setup.

@AndrewL733
Copy link

Thanks! I will be setting this up later in the week and I will let you know how (and if) it works. I did find instructions for setting up wireguard and gretap tunnels with netplan, but I think I’ll just try using “ip” first. I just tried it and it seems to work okay on top of a running netplan setup. I think I’ll just disable the running Zerotier configuration before starting to run wireguard on both sides (don’t want to end up with a loop in my network).

@SandeepAthiyarath
Copy link

Thanks for this article.
I was looking for a fastest option to bridge two locations with same LAN network.
Wireguard plus GRETAP and bridging seems to be the best option for this. I followed this article and with the help, I am able to
connect two hosts to reach each other with LAN IP addresses . I.e with this method we can connect two systems with a bridge over Wireguard and GRETAP. But when try to reach other systems than the Wireguard host, it seems not working.

On a boarder way I am able to see that Wireguard is doing some mischief with 'Allowed IPS' option.
Being both side same LAN I am forced to use 'Allowed IPS' same network. Say for example if you
are planning to extend 192.168.5.0/24 with this solution, we need to add Allowed IPs = 192.168.5.0/24 on both side.
in addition to the WG interface IPs. This seems creating problem as other systems in the same network can't reach
Wireguard system LAN ip address as Wireguard forwarding that packets to peer system. So once Wireguard is up,
the Wireguard systems are not ping-able from the LAN network on each side, because Wireguard seems to forward that
packets to the peer host instead of responding to ping.

Wondering any workaround for this problem ?

@zOrg1331
Copy link
Author

zOrg1331 commented Sep 3, 2023

Hi @SandeepAthiyarath,

With AllowedIPs and routing it is all about careful and accurate planning, not to break the stick you are sitting on. :) So yes, with some configurations you indeed can block you own traffic. I would suggest to exclude VPN server's IP address from internal networks. Or even come up with private segment just for VPN segment, depends. So you can have IP address to connect to outside of AllowedIPs range.
BTW, you can have many networks inside AllowedIPs option.

Another scenario (but may be weird) is to have a masquerading of some sort which will hide the network segment behind itself. In this case Wireguard will receive all packets from a single address.

Nevertheless, I am just producing some thoughts. :) I never tried to build production/office networks with such configuration. However, there are even paid products offering such, so it should be possible.

@SandeepAthiyarath
Copy link

Hmm. Thank you. This is valuable inputs.
In any case we are not going to use duplicate IP s on both side.
I was expecting a magic solution. Now I understand it wont work like that :)
Still I never came across a commercial solution to bridge LANs. If you have any reference I am eager to look at it.
Thank you for your valuable article and comment @zOrg1331.

@SandeepAthiyarath
Copy link

SandeepAthiyarath commented Sep 4, 2023

HI @zOrg1331
I am clueless here after connecting with other systems in the LAN. I managed to use Allowed IPs with /32 and expecting it is working. As mentioned in my previous comment I am able to communicate with LAN IPs between the Wireguard hosts.
Problem starts after I add LAN side interface to bridge. This looks weird. My LAN side IP is 192.168.17.1 on enp2s0 interface. I have a
system with 192.168.17.5 connected with a switch to enp2s0 of Wireguard host.
The moment I add enp2s0 to the bridge these systems stops communicating each other. No Ping. No arp info. If I remove enp2s0 from the
bridge with nomaster option, then they again start communicating each other. I am clueless here.

root@srv1 fcoos]# bridge l
3: enp2s0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 master brtun state forwarding priority 32 cost 4
12: gretun@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1382 master brtun state forwarding priority 32 cost 100
If I run
root@srv1 fcoos]# ip l set enp2s0 nomaster
[root@srv1 fcoos]# ping 192.168.17.5
PING 192.168.17.5 (192.168.17.5) 56(84) bytes of data.
64 bytes from 192.168.17.5: icmp_seq=1 ttl=64 time=0.166 ms

But If I add again to the bridge
oot@srv1 fcoos]# ip l set enp2s0 master brtun up
[root@srv1 fcoos]# ping 192.168.17.5
PING 192.168.17.5 (192.168.17.5) 56(84) bytes of data.
From 192.168.17.1 icmp_seq=1 Destination Host Unreachable

I was expecting the reverse of this. Thinking I am doing something wrong here. But what is that ? Please note that I need to bridge two LANS not two hosts. I have bridged LAN with OpenVPN tap. But performance is not as expected. I am hopeful with wire-guard. But :). Need help

@AndrewL733
Copy link

AndrewL733 commented Sep 4, 2023

Hello @zOrg1331, @SandeepAthiyarath

@zOrg1331 I asked you some questions a couple of weeks ago and I'm now reporting back to you. @SandeepAthiyarath I hope my findings can be useful for you.

I currently have a Wireguard + GRETAP system working successfully and I'm going to document here what I did. My objective was to connect 2 sites -- one on each side of the Atlantic ocean -- as if they were on a single LAN. A key requirement for me was that broadcast traffic be seen on both sides. There was some trial and error to get this to work, but now that it's working, I'm getting at least 4x the performance I used to get with OpenVPN TAP, and significantly better than what I get with Zerotier. I'm getting 200 Mbps in both directions between the two sites (my Internet speed in North America is the limiting factor). Here is a step-by-step explanation of what I did.

BACKGROUND

In my network, both sides are on the same subnet -- 192.168.15.0/24. Each side has its own DHCP server. However, the addresses given out by Side A's router (192.168.15.1) go from .100-.169 and by Side B's router (192.168.15.2) from .170-.254. Using ebtables, I block DCHP requests from going over the VPN, so each side only sees its local DHCP server (I explain this at the bottom). I have a number of static IP addresses in use on either side but they are all in the 10-99 range.

I have dedicated Wireguard gateway servers on each side. The physical hardware for each server is a $160 Beelink Intel Quad-core with 1 x 1 Gb Ethernet port. The servers are running Ubuntu 22.04 LTS and they are using the default "Netplan" system for configuring everything related to networking. I realize most distributions don't use Netplan, but I find Netplan very sensible and easy to understand because it puts all network configuration in one single file (although it's a little fussy when it comes to proper indenting).

One other thing. I use UDP port 52001 for Wireguard on both sides, so on each side I have to forward that port from my router to the Wireguard server. So, here's how I set everything up (with much thanks to @zOrg1331 for helping me get started). My instructions assume you have already installed the Wireguard tools packages.

STEP ONE -- GENERATE WIREGUARD KEYS

NOTE: The keys used below are not my keys. I found them in some other random Wireguard example.

Side A -- North America

wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs  SF8baNt/6uS8r6rAB7XNUyWrjo89Tf/q93Y1oG2YRWo=   
cat wgkeypubs  mnhaI1xPKVC/yKUmn77i3nuKtVfxnkg9ojIXd5CBn0E=

Side B -- Europe

wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs  iETL+NzI0VtxxCz62ZtG4N9ELoq1XLD6PmUb7fFir0g=
cat wgkeypubs  bGgYeUjhy+LRKMPL8lgcXlxKlpUJGuqa3mMgX9fw+ng=

In the next step, we will copy and paste Side A's public key into Side B's Wireguard configurationso that Side B can gain access to Side A. And vice versa.

STEP TWO -- CONFIGURE NETWORKING FOR WIREGUARD

On Side A -- North America -- modify /etc/netplan/00-installer-config.yaml and apply changes.

I have only one physical Ethernet port (enp1s0) on the Beelink computer that I use for Wireguard here, and I want it to have a static address of 192.168.15.20 and I want that to be part of a bridge br0. I want to set the IP address and default gateway (the LAN address of my American router) ONLY on the bridge interface. The physical interface should NOT have an IP Address. I also want to create the Wireguard tunnel so that it will have the address 172.16.0.1/24 and automatically connects to the European side.

Contents of /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:        
  ethernets:
    enp1s0:
      dhcp4: no

  bridges:
    br0:
      dhcp4: no
      addresses: [192.168.15.20/24]
      nameservers:
        addresses: [192.168.15.1]
      routes:
        - to: default
          via: 192.168.15.1
      interfaces:
        - enp1s0
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 52001
      key: SF8baNt/6uS8r6rAB7XNUyWrjo89Tf/q93Y1oG2YRWo=   
      addresses:
        - 172.16.0.1/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: [EUROPEAN SIDE DOMAIN NAME]:52001
          keys:
            public: bGgYeUjhy+LRKMPL8lgcXlxKlpUJGuqa3mMgX9fw+ng=
  version: 2

To apply, run the command "sudo netplan apply"

On Side B -- Europe -- modify /etc/netplan/00-installer-config.yaml and apply changes.

I have only one physical Ethernet port (enp1s0) on the Beelink computer that I use for Wireguard here, and I want it to have a static address 192.168.15.21 and I want that to be part of a bridge br0. I want to set the IP address and default gateway (my European router) only on the bridge. I also want to create the Wireguard tunnel so that it will have the address 172.16.0.2/24 and automatically connect to the European side.

Contents of /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:        
  ethernets:
    enp1s0:
      dhcp4: no

  bridges:
    br0:
      dhcp4: no
      addresses: [192.168.15.21/24]
      nameservers:
        addresses: [192.168.15.2]
      routes:
        - to: default
          via: 192.168.15.2
      interfaces:
        - enp1s0
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 52001
      key: iETL+NzI0VtxxCz62ZtG4N9ELoq1XLD6PmUb7fFir0g= 
      addresses:
        - 172.16.0.2/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: [NORTH AMERICAN SIDE DOMAIN NAME]:52001
          keys:
            public: mnhaI1xPKVC/yKUmn77i3nuKtVfxnkg9ojIXd5CBn0E=
  version: 2

To apply, run the command "sudo netplan apply"

STEP 3 CONFIRM THAT WIREGUARD IS RUNNING AND CONNECTED TO THE PEER ON THE OTHER SIDE

On the European side if I run "sudo wg show" I see:

interface: wg0
  public key: mnhaI1xPKVC/yKUmn77i3nuKtVfxnkg9ojIXd5CBn0E=
  private key: (hidden)
  listening port: 52001

peer: bGgYeUjhy+LRKMPL8lgcXlxKlpUJGuqa3mMgX9fw+ng=
  endpoint: [IP ADDRESS OF NORTH AMERICAN SIDE]:52001
  allowed ips: 172.16.0.0/24
  latest handshake: 1 minute, 8 seconds ago
  transfer: 13.27 GiB received, 9.44 GiB sent

Once Wireguard is running on both sides, I should be able to ping from one Wireguard server to the other across the Atlantic. For example, from 172.16.0.1 I get:
vpnadmin@andyvpntorre:/etc/netplan$ ping 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=130 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=126 ms
64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=134 ms
64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=138 ms

VERY IMPORTANT! Note, at this point, from the Wireguard server at 192.168.15.20 I cannot ping the Wireguard server at 192.168.15.21. In other words, the Wireguard servers don't know anything about how to cross the Atlantic on the 192.168.15.0/24 network. But they soon will.

STEP 4 -- CREATE GRETAP INTERFACES AND ADD THEM TO THE BRIDGE br0

I haven't yet tried putting this in my netplan configuration (I will soon and I'll come back here and edit this post) so after I have Wireguard running properly on each side and connected to the other side, I now create the GRETAP interface so that my Ethernet packets will flow from any computer on Side A to any computer on Side B (including from 192.168.15.20 to 192.168.15.21).

On Side A run the following commands:

sudo ip link add gretun type gretap local 172.16.0.1 remote 172.16.0.2 ignore-df nopmtudisc

This creates a virtual interface called "gretun" of the type "gretap". Packets will go from local IP address 172.16.0.1 to remote address 172.16.0.2. Note the addition of the two options "ignore-df" and "nopmtudisc". Without these, I found that packets larger than 1354 bytes could not get through the tunnel. For instance, web pages won't load across the tunnel. This is due to large packets not getting fragmented into pieces that are small enough to go through the tunnel. The large packets just get dropped without those options.

ip link set gretun up

This brings up the gretun interface

ip link set gretun master br0

This adds the gretun interface to my bridge "br0"

ip link set gretun mtu 1500

The default MTU that gets created for the GRETAP interface that we named "gretun" is too low and even with the "ignore-df" and "nopmtudisc" options. Setting the MTU to 1500 ensures that all normal network packets will get through.

On Side B, do the same thing except swap the IP addresses:

sudo ip link add gretun type gretap local 172.16.0.2 remote 172.16.0.1 ignore-df nopmtudisc
ip link set gretun up
ip link set gretun master br0
ip link set gretun mtu 1500

Assuming you have bridge-utils installed, you should see the following when you run the command "brctl show"

bridge name	bridge id		STP enabled	interfaces
br0		8000.5e085b91133c	no		enp1s0
						        gretun

STEP 5 -- BLOCK DHCP REQUESTS FROM CROSSING OVER THE VPN

On the Wireguard server on each side, I run the following commands:

ebtables -A INPUT --in-interface gretun --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
ebtables -A INPUT --in-interface gretun --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
ebtables -A FORWARD --out-interface gretun --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
ebtables -A FORWARD --out-interface gretun --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP

I will eventually put this into a script that runs a minute or so after the Wireguard-GRETAP bridge is up and connected.

FINAL NOTES

  • To make Wireguard + GRETAP work, you DO NOT need to enable IPV4 ip_forwarding. This works because the interface we named "gretun" (of type "gretap") is part of the bridge br0.
  • You do need to have a Domain Name or Dynamic DNS name for each side, or static public IP addresses, so that the two endpoints can automatically find each other and connect.
  • There is no routing involved when you put together Wireguard + GRETAP. This is just like having two Ethernet switches on your network, only there's a very long WAN cable between them (with both sides on the same subnet).
  • When a Wireguard interface gets created, it gets a default MTU of 1420. Changing this MTU to a bigger number doesn't seem to make any difference, nor does it solve the problem that only gets solved by adding the options "ignore-df nopmtudisc" to the command that creates the GRETAP interface.
  • I'm still considering whether I need to add a "persistent keepalive" option to my Wireguard configuration

@SandeepAthiyarath
Copy link

SandeepAthiyarath commented Sep 5, 2023

Excellent update @AndrewL733 . Thank you. I noticed one main difference is that you use single network card system with NATed from router.
Ideally it should not make difference with two cards as in my case. Another important thing I noticed that you use Allowd IPs complete sub-net.
I was thinking that this wont work . I am going to try options and update here soon.

@geckotdf
Copy link

geckotdf commented Sep 6, 2023

Hi!
I want to say thank you to all of you and the final guide of @AndrewL733
I was trying to achieve this to protect my home network behind a VPS with DDOS protection.
Because my home ISP don't allow me to open ports, WireGuard was the only approach.

In my case the IP of the VPS it's the public one, like a modem in bridge mode.
In that case I put two addreses with comma, one for the public IP of the VPS, and another for the VLAN of my network.
The same for the addresses in the nameservers.

In this escenario I enable in the VPS "the server" sysctl -w net.ipv4.ip_forward=1 because the traffic from the public IP must be NATed to a computer inside my home VLAN, and need to enable a iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE to expose the br0 interface to internet.
Finally in the Windows computer I have in my home need to add like a secondary gateway, in this scenario the public IP of the VPS.

And works awesome!

I add the persistent keepalive because happens to me some incoming packets mostly in UDP don't arrive to the computer behind the NAT, and some services that are monitored with UDP ports believe it's down, but with the persistent works great.

The part of the greetap interfaces I made it with a script in the startup of the ubuntu server in each side, then use in all cases aliases because in the home network the public IP can change time to time.

And Finally install UFW in the server side because have the public IP to BLOCK all except what I need to nat to my house, to avoid any possible issue.

Again, thanks to everyone for this process and sharing it!

@NikOverflow
Copy link

now it should be possible to use vrrp over wireguard.

@NikOverflow
Copy link

@zOrg1331 Thanks for your reply. Can you explain whether the configuration you have described would create a site-to-site bridge? The answer seems to be “no" but maybe I’m missing something. My ultimate goal is to create a site-to-site Wireguard VPN with layer 2 support, such that all computers at Site A can communicate with all computers at Site B, and so that all broadcast traffic goes over the VPN. I am currently able to achieve this with OpenVPN TAP and also with ZeroTier (about 1.3x the performance as OpenVPN). It would seem very useful to be able to use Wireguard for the same purpose.

Well, as for me, the whole purpose of this exercise was to link two networks on L2 level by using Wireguard.

Side note: I used it for a quite non-standard purpose. During a penetration test I had to establish a tunnel towards a corporate network via a corporate firewall and have L2 communications at the same time. By following the described method I had transparent (for userspace tools) L2 communications between a Wireguard client (remote VM) and the remote network. So the only difference between site-to-site VPN was that I did not have network with multiple hosts on one side.

Once you have GRE L2 tunnel working over Wireguard, the rest is up to you and your setup.

but what if you wanna connect multiple networks together?

@alexunderboots
Copy link

Спасибо за хорошую инструкцию. Нашел ошибку в своей конфигурации: на центральном сервере, соединяющем 2 wireguard был настроен маскарад

@alexunderboots
Copy link

alexunderboots commented Nov 17, 2023

please tell me why the solution does not work?:

ip link add vx0 type vxlan id 200 group 239.1.1.1 dev wg0 dstport 4789
ip link set vx0 master br0
ip link set vx0 up

@pmagro
Copy link

pmagro commented Jan 16, 2024

Hi all,

Thank you very much for your guides, they are really helpful.

@AndrewL733, how do you do the tunnel permanent? I'm loosing the setup after rebootin:

sudo ip link add gretun type gretap local 172.16.0.2 remote 172.16.0.1 ignore-df nopmtudisc
ip link set gretun up
ip link set gretun master br0
ip link set gretun mtu 1500

Thanks in advance!
Best regards

@geckotdf
Copy link

Hi @pmagro just login with sudo su - to be root, create a Start.sh in the /root folder then type chmod a+x /root/Start.sh to set the permission
Then put that code without the sudo in front in the Start.sh file, save it.
Then type crontab -e
Add at the end:

@reboot /root/Start.sh

Save it.

Done. Next time you start up your system will add that code.
Good Luck.

@pmagro
Copy link

pmagro commented Jan 16, 2024

Superb, @geckotdf, it worked, thanks!

@pmagro
Copy link

pmagro commented Jan 17, 2024

Can I ask another question? What would be the public IP address I will be detected with when accessing internet from each of the sites?

Thanks in advance!

Best regards
Pablo

@geckotdf
Copy link

In the other sites will have the public IP that have the gateway in that computers.
That not going to change, this expose layer 2.
Only if you do what I post a few months ago, in the server:

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE

Then, in the remote computer set in the gateway the public IP of the server.
But this only will work if the server have the public IP in the interface like a VPS or dedicated server.

For example I have more or less a setup like this to expose my services behind a server and avoid exposing my real IP, and prevent DDOS attacks in my services, similar to a Proxy, but allow UDP and TCP Traffic.

@jonfatino
Copy link

192.168.15.0/24

@AndrewL733 great guide.

I double checked everything and was able to ping 172.16.0.2 and 172.16.0.1 from both sides but traffic from my 192.168.x network is not passing through the bridge. Any suggestions?

@pmagro
Copy link

pmagro commented Jan 23, 2024

Hi @jonfatino, I can confirm that following Andrew's guide I made it work. Trying double checking everything or start from scratch if needed, but following it step by step, it works.

@Gaitonde007
Copy link

@AndrewL733 , Great explanation. But I did notice drastic difference when I bumped up the MTU of wg interface to 1550. I'm testing in a lab environment were all the devices have 10GBPS NIC. When wg0 interface is left to default of 1420 I got around 2GBPS of throughput. But after changing it to 1550 I'm getting throughput around 4GBPS.https://www.reddit.com/r/networking/comments/19ecn6v/comment/kjeetzs/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button redditor put dose similar with VXLAN and wg and has got 870+ on a 1GBPS link. I'm expecting atlease a throughput of 6GBPS with this setup. I hope I'm not missing anything.

Thanks

@geckotdf
Copy link

@Gaitonde007
Im not an expert but in general lines a big MTU can cause packet fragmentation and a little one to much overload.
So... probably at that fast speeds put a more tiny MTU like 1420 make load in the network and you are only able to get 2 GBPS, and a bigger one, allow to get 4 GBPS.
Take in mind the kind of traffic it's traveling too, it's not the same UDP packets than for example TCP.

Reading about what Andrew test, the default 1420 don't work very well like he said for navigation, so probably after testing get that 1500 it's the lowest working MTU quote:

The default MTU that gets created for the GRETAP interface that we named "gretun" is too low and even with the "ignore-df" and "nopmtudisc" options. Setting the MTU to 1500 ensures that all normal network packets will get through.

So... if you need more throughput can test changing the MTU, at the end of the day need to be set based on the network equipment and topology.
But it's a general line opinion probably a person with more experience can tell.

@magancete
Copy link

magancete commented Apr 5, 2024

@AndrewL733 thank you very much for the tutorial, I have finally been able to connect my two homes!

I have the following problem, I'm wondering if someone can help me:

  • In both homes A and B I have a router that broadcasts IPTV multicast content.
  • I have multicast receivers in both homes.
  • I want the multicast receiver in apartment B to receive content from apartment A.
  • Receivers and routers use IGMPv2.
  • On receiver B I have set a static IP with gateway to router A

I can't get it to work, in the firewalls of the two wireguard servers I have allowed all incoming/outgoing traffic to 224.0.0.0/4

@Handsome1080P
Copy link

for fake tcp,try phantun or mimc(debian 12 or kernel 6.1+),faster if you want.And I prefer use vxlan via wg.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment