wireguard, wireguard layer 2, wireguard over TCP

wireguard_layer2.md

Intro

This note describes how to connect two networks/devices/VMs over public network using Wireguard with Layer 2 support (ARP, IPv6 link-local, etc).

This can also be achieved using SSH and its "tap" tunnel, however, it does not provide the same level of latency and bandwidth as full-blown VPN such as Wireguard.

In addition, this note describes how to tunnel Wireguard over TCP connection. This may be of use if you encounter firewall in-between so, for instance, you can use TCP port 443 only.

Objective

Let's draw a network diagram of what we would like to have in as a result.

 Host on private LAN                                         Host on the Internet
 +---------------------------------+                       +-------------------------------+
 |   brtun bridge                  |                       |                  brtun bridge |
 | +-------------+                 |                       |                 +--------+    |
 | | ethX gretun |<->wg0<->udp2raw | <-Internet, TCP 443-> | udp2raw<->wg0<->| gretun |    |
 | +-------------+                 |                       |                 +--------+    |
 +---------------------------------+                       +-------------------------------+
 
 brtun: 192.168.0.200/24                                    brtun: 192.168.0.50/24
 wg0: 12.12.12.2/24                                         wg0: 12.12.12.1/24
 gretun: 12.12.12.2->12.12.12.1                             gretun: 12.12.12.1->12.12.12.2
 

Setting things up

Distinguishing between server and client here is based on the answer to the question: who can connect to Wireguard listener socket.

Server

Generating a key:

wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivs
ip a a 12.12.12.1/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubc) allowed-ips 0.0.0.0/0

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.1 remote 12.12.12.2
ip l s gretun up

Configuring a bridge which includes GRETAP interface. This step is in fact optional, but it allows you some flexibility: you can assign an IP address to this bridge interface and/or you can add additional interfaces to this bridge.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's assign an IP address from the target network to this bridge:

sudo ip a a 192.168.0.51/24 dev brtun

If we want Wireguard over TCP, launch UDP2RAW listener which will do its magic. You can get UDP2RAW from this Git repo: https://github.com/wangyu-/udp2raw-tunnel/releases/

./udp2raw_amd64 -s -l YOUR_PUBLIC_IP:443 -r 127.0.0.1:51820 -a --fix-gro

Client

Client is very similar to the server described above.

Generating a key:

wg genkey | tee wgkeyprivc | wg pubkey > wgkeypubc

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivc
ip a a 12.12.12.2/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key, endpoint address and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint YOUR_PUBLIC_IP:51820 persistent-keepalive 15

In case you want Wireguard over TCP, set endpoint address to a localhost interface:

wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint 127.0.0.1:7777 persistent-keepalive 15

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.2 remote 12.12.12.1
ip l s gretun up

Configuring a bridge which includes GRETAP interface.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's add a physical interface belonging to a LAN network to the bridge and start DHCP client:

ip l set ethX master brtun
dhcpcd ethX

If we want Wireguard over TCP, launch UDP2RAW listener on the localhost interface which connects to the corresponding public listener.

./udp2raw_amd64 -c -l 127.0.0.1:7777 -r YOUR_PUBLIC_IP:443 -a --fix-gro

Results

That's it. Now you will have Layer 2 connectivity between two separated machines over Wireguard VPN (optionally, over TCP). Enjoy your ARP and IPv6 link-local experience.

References

• https://notes.superlogical.ch/pages/note_wg/nolayer2/
• https://lafibre.info/vpn/tunnel-gre-sur-wireguard/&prev=search&pto=aue

now it should be possible to use vrrp over wireguard.

@zOrg1331 Thanks for your reply. Can you explain whether the configuration you have described would create a site-to-site bridge? The answer seems to be “no" but maybe I’m missing something. My ultimate goal is to create a site-to-site Wireguard VPN with layer 2 support, such that all computers at Site A can communicate with all computers at Site B, and so that all broadcast traffic goes over the VPN. I am currently able to achieve this with OpenVPN TAP and also with ZeroTier (about 1.3x the performance as OpenVPN). It would seem very useful to be able to use Wireguard for the same purpose.

Well, as for me, the whole purpose of this exercise was to link two networks on L2 level by using Wireguard.

Side note: I used it for a quite non-standard purpose. During a penetration test I had to establish a tunnel towards a corporate network via a corporate firewall and have L2 communications at the same time. By following the described method I had transparent (for userspace tools) L2 communications between a Wireguard client (remote VM) and the remote network. So the only difference between site-to-site VPN was that I did not have network with multiple hosts on one side.

Once you have GRE L2 tunnel working over Wireguard, the rest is up to you and your setup.

but what if you wanna connect multiple networks together?

Спасибо за хорошую инструкцию. Нашел ошибку в своей конфигурации: на центральном сервере, соединяющем 2 wireguard был настроен маскарад

please tell me why the solution does not work?:

ip link add vx0 type vxlan id 200 group 239.1.1.1 dev wg0 dstport 4789
ip link set vx0 master br0
ip link set vx0 up

Hi all,

Thank you very much for your guides, they are really helpful.

@AndrewL733, how do you do the tunnel permanent? I'm loosing the setup after rebootin:

sudo ip link add gretun type gretap local 172.16.0.2 remote 172.16.0.1 ignore-df nopmtudisc
ip link set gretun up
ip link set gretun master br0
ip link set gretun mtu 1500

Thanks in advance!
Best regards

Hi @pmagro just login with sudo su - to be root, create a Start.sh in the /root folder then type chmod a+x /root/Start.sh to set the permission
Then put that code without the sudo in front in the Start.sh file, save it.
Then type crontab -e
Add at the end:

@reboot /root/Start.sh

Save it.

Done. Next time you start up your system will add that code.
Good Luck.

Superb, @geckotdf, it worked, thanks!

Can I ask another question? What would be the public IP address I will be detected with when accessing internet from each of the sites?

Thanks in advance!

Best regards
Pablo

In the other sites will have the public IP that have the gateway in that computers.
That not going to change, this expose layer 2.
Only if you do what I post a few months ago, in the server:

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE

Then, in the remote computer set in the gateway the public IP of the server.
But this only will work if the server have the public IP in the interface like a VPS or dedicated server.

For example I have more or less a setup like this to expose my services behind a server and avoid exposing my real IP, and prevent DDOS attacks in my services, similar to a Proxy, but allow UDP and TCP Traffic.

192.168.15.0/24

@AndrewL733 great guide.

I double checked everything and was able to ping 172.16.0.2 and 172.16.0.1 from both sides but traffic from my 192.168.x network is not passing through the bridge. Any suggestions?

Hi @jonfatino, I can confirm that following Andrew's guide I made it work. Trying double checking everything or start from scratch if needed, but following it step by step, it works.

@AndrewL733 , Great explanation. But I did notice drastic difference when I bumped up the MTU of wg interface to 1550. I'm testing in a lab environment were all the devices have 10GBPS NIC. When wg0 interface is left to default of 1420 I got around 2GBPS of throughput. But after changing it to 1550 I'm getting throughput around 4GBPS.https://www.reddit.com/r/networking/comments/19ecn6v/comment/kjeetzs/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button redditor put dose similar with VXLAN and wg and has got 870+ on a 1GBPS link. I'm expecting atlease a throughput of 6GBPS with this setup. I hope I'm not missing anything.

Thanks

@Gaitonde007
Im not an expert but in general lines a big MTU can cause packet fragmentation and a little one to much overload.
So... probably at that fast speeds put a more tiny MTU like 1420 make load in the network and you are only able to get 2 GBPS, and a bigger one, allow to get 4 GBPS.
Take in mind the kind of traffic it's traveling too, it's not the same UDP packets than for example TCP.

Reading about what Andrew test, the default 1420 don't work very well like he said for navigation, so probably after testing get that 1500 it's the lowest working MTU quote:

The default MTU that gets created for the GRETAP interface that we named "gretun" is too low and even with the "ignore-df" and "nopmtudisc" options. Setting the MTU to 1500 ensures that all normal network packets will get through.

So... if you need more throughput can test changing the MTU, at the end of the day need to be set based on the network equipment and topology.
But it's a general line opinion probably a person with more experience can tell.

@AndrewL733 thank you very much for the tutorial, I have finally been able to connect my two homes!

I have the following problem, I'm wondering if someone can help me:

• In both homes A and B I have a router that broadcasts IPTV multicast content.
• I have multicast receivers in both homes.
• I want the multicast receiver in apartment B to receive content from apartment A.
• Receivers and routers use IGMPv2.
• On receiver B I have set a static IP with gateway to router A

I can't get it to work, in the firewalls of the two wireguard servers I have allowed all incoming/outgoing traffic to 224.0.0.0/4

for fake tcp,try phantun or mimc(debian 12 or kernel 6.1+),faster if you want.And I prefer use vxlan via wg.

@AndrewL733 - Your guide is really good, and it works. :) However, I do have an issue, and I'm a bit lost.

I have tried it between 2 data centers. When I install 2 Windows servers, one on each site (Site A and Site B), they do timeout when pinging each other. I did manage to get a speed test with over a Gigabit connection through the tunnel, but after the test, the connection dies. Then I need to reboot the WireGuard "switches" and wait 3-10 minutes to have a somewhat ok connection again. I have tried to change the MTU and also did an MTU test through the tunnel, but with no luck.

I can see some have commented on the high throughput and suggested changing the MTU. Has anyone figured it out? :)

The point of this for me is because I need to migrate some servers from one data center to another, and if I could make a big "switch", it would be nice to just take parts of the environment at a time.

I will paste my configuration down here, for Site A and Site B, and I hope you, or maybe someone else, have experienced the same issue and maybe have a solution.

Thank you for your time :)

Site A

#Ubuntu 22.04 LTS
#Remember to have Promiscuous mode, MAC address changes, and Forged transmits enabled on the interfaces.
#Change to root.
sudo su -
apt purge snapd -y;apt update -y ; apt full-upgrade -y;apt install wireguard bridge-utils openvswitch-switch-dpdk traceroute net-tools -y
reboot now
#Login again
sudo su -
wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs
cat wgkeypubs
Private: SiteAPrivateKey
Public: SiteAPublicKey

nano /etc/netplan/00-installer-config.yaml
#Change the netplan configuration. Modify it for the respective interfaces, IPs, and WireGuard keys.
# This is the network config written by 'subiquity'
network:
  ethernets:
    #Uplink/switching interface 10.76.0.0/24
    ens192:
      dhcp4: false
      dhcp6: false

  bridges:
    br1:
      dhcp4: false
      dhcp6: false
      #IP to Wireguard VM
      addresses: [10.76.1.10/24]
      #DNS - This can be changed to the Domain Controller if needed
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      routes:
        - to: default
          via: 10.76.1.1
      interfaces:
        - ens192
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 51820
      #Site A Privatekey
      key: SiteAPrivateKey
      addresses:
        - 172.16.0.1/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: SiteB_WAN:51820
          #Peers public key
          keys:
            public: SiteBPublicKey
          keepalive: 90
  version: 2

#Activate Layer2
#Create bash script, with execute permission.
nano ~/gretun_cmd
  #!/bin/bash
  ip link add gretun1 type gretap local 172.16.0.1 remote 172.16.0.2 ignore-df nopmtudisc
  ip link set gretun1 up
  ip link set gretun1 master br1
  ip link set gretun1 mtu 1500
chmod u+rwx,g+r-wx,o-rwx ~/gretun_cmd
bash ~/gretun_cmd

#Create bash script, to Block DHCP Req over WireGuard VPN. 
nano ~/block_dhcp_cmd
  #!/bin/bash
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
chmod u+rwx,g+r-wx,o-rwx ~/block_dhcp_cmd
bash ~/block_dhcp_cmd

#Now Layer2 is active

Site B

#Ubuntu 22.04 LTS
#Remember to have Promiscuous mode, MAC address changes, and Forged transmits enabled on the interfaces.
#Change to root.
sudo su -
apt purge snapd -y;apt update -y ; apt full-upgrade -y;apt install wireguard bridge-utils openvswitch-switch-dpdk traceroute net-tools -y
reboot now
#Login again
sudo su -
wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs
cat wgkeypubs
Private: SiteBPrivateKey
Public: SiteBPublicKey

nano /etc/netplan/00-installer-config.yaml
#Change the netplan configuration. Modify it for the respective interfaces, IPs, and WireGuard keys.
# This is the network config written by 'subiquity'
network:
  ethernets:
    #Uplink interface 10.76.2.0/24. This allows connection through WireGuard, with traffic passing through this interface.
    ens7:
      dhcp4: true
      dhcp6: false

    #Switching interface for bridge br1 10.76.1.0/24 
    #This is a dummy interface, BUT STILL REQUIRING Promiscuous mode, MAC address changes, and Forged transmits TO BE ACTIVE ON THE INTERFACE.
    #No gateway should be configured, as it goes through the WireGuard tunnel.
    ens3:
      dhcp4: false
      dhcp6: false

  bridges:
    br1:
      dhcp4: false
      dhcp6: false
      addresses: [10.76.1.11/24]
      #DNS - This can be changed to the Domain Controller if needed
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      #No routing needs to be configured; everything is routed through the WireGuard tunnel.
#      routes:
#        - to: default
#          via: 10.76.1.1
      interfaces:
        - ens3
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 51820
      #Site B Private key
      key: SiteBPrivateKey
      addresses:
        - 172.16.0.2/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: SiteA_WAN:51820
          #Peers public key
          keys:
            public: SiteAPublicKey
          keepalive: 90
  version: 2

#Activate Layer2
#Create bash script, with execute permission.
nano ~/gretun_cmd
  #!/bin/bash
  ip link add gretun1 type gretap local 172.16.0.2 remote 172.16.0.1 ignore-df nopmtudisc
  ip link set gretun1 up
  ip link set gretun1 master br1
  ip link set gretun1 mtu 1500
chmod u+rwx,g+r-wx,o-rwx ~/gretun_cmd
bash ~/gretun_cmd

#Create bash script, to Block DHCP Req over WireGuard VPN.
nano ~/block_dhcp_cmd
  #!/bin/bash
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
chmod u+rwx,g+r-wx,o-rwx ~/block_dhcp_cmd
bash ~/block_dhcp_cmd

#Now Layer2 is active