- I faced bandwidth issues between a WG Peer and a WG server. Download bandwidth when downloading from WG Server to WG peer was reduced significantly and upload bandwidth was practically non existent.
- I found a few reddit posts that said that we need to choose the right MTU. So I wrote a script to find an optimal MTU.
- Ideally I would have liked to have run all possible MTU configurations for both WG Server and WG Peer but for simplicity I choose to fix the WG Server to the original 1420 MTU and tried all MTUs from 1280 to 1500 for the WG Peer.
- On WG server, I started an
iperf3
server - On WG peer, I wrote a script that does the following:
wg-quick down wg0
- Edit MTU in the
/etc/wireguard/wg0.conf
file wg-quick up wg0
iperf3 -c 172.123.0.1 -J -t 5 -i 5
- This runs an
iperf3
client that connects to172.123.0.1
which is the WG Server gateway - The
iperf3
client runs for 5 seconds and stops and dumps a JSON output
- This runs an
- Max bandwidth provided by my ISP (1000Mbps Download, 50Mbps Upload)
- WG Server is a VPS running Ubuntu 20.04 on a cloud provider.
- WG Peer is a PC running Ubuntu 20.04 locally at home.
I followed this tutorial to setup my Wireguard configurations.
WG-server
# /etc/wireguard/wg0.conf
[Interface]
Address = 172.123.0.1/24
MTU = 1420
SaveConfig = true
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; iptables -t nat -A POSTROUTING -o ens10 -j MASQUERADE; ip6tables -t nat -A POSTROUTING -o ens10 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; iptables -t nat -D POSTROUTING -o ens10 -j MASQUERADE; ip6tables -t nat -D POSTROUTING -o ens10 -j MASQUERADE
ListenPort = 51820
PrivateKey = xxxxxxxxxxxxxxxxxx
[Peer]
PublicKey = xxxxxxxxxxxxxxxxxx
AllowedIPs = 172.123.0.2/32
Endpoint = X.X.X.X:61426
WG-peer
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = xxxxxxxxxxxxxxxxxx
ListenPort = 51820
Address = 172.123.0.2/24
MTU = 1384
[Peer]
PublicKey = xxxxxxxxxxxxxxxxxx
AllowedIPs = 172.123.0.0/24, 10.1.0.0/24
Endpoint = Y.Y.Y.Y:51820
PersistentKeepalive = 5
- As you can see in the image, the original MTU setting of 1420 for both peer and server gives abysmal bandwdith.
- I found that that MTU 1384 on the WG peer with 1420 on the WG server seems to almost have the best bandwidth.
- For WG Peer MTU 1384, the max upload bandwidth of 50Mbps of my ISP connection is achieved but I was only able to hit 550 Mbps for the download bandwidth where the max download bandwidth of my ISP connection is 1000 Mbps. This reduction in download bandwidth might be due to other factors but 550 Mbps was sufficient for my use cases so I stopped testing it further.
In case any one has any explanations for this behavior or have found some mistakes in my configurations or tests, please let me know.
Wireguard does not default to MTU 1500. It defaults to
1500 - 80
...but only if all other attempts to detect your connection MTU fail. I'm talking aboutwg-quick
helper script here. MTU config setting doesn't mean anything towg
(which expects you to manage ip/link/route/addr stuff yourself).This is because wireguard has 80 byte overhead for ipv6 and 60 byte overhead for ipv4. You may also find this page helpful.
Relevant code is here.
Here is the MTU setting logic from
wg-quick
:wg show wg0 endpoints
ip route get <endpoint address>
and looking formtu [0-9]+
in output.dev [^ ]+
) is taken instead and used inip link show dev <dev-for-route>
...and the mtu (mtu [0-9]+
) is extracted from that output.mtu
(on output ofip route get <addr>
orip link show dev <endpoint-dev>
...then wg-quick tries to get the mtu fromip route show default
mtu - 80
.** This means if your eth0 MTU is 1500 and you explicitly set MTU to 1500 in wg-quick config, 1500 will be used as-is (not 1420)...so you will have problems because your wg packets will end up larger than what is supported on the underlying device/route and so there will be fragmentation (slow because packets have to be broken up).
*** Is this right? If we have a
peer endpoint A
with1200 MTU
oneth1
and anotherpeer endpoint B
with1500 MTU
oneth2
, it appears to me that wg-quick keeps the1500 MTU
fromB
(minus 80 at the end)...meaning packets for peerA
may be larger than the1200 MTU
supported oneth1
causing fragmentation and slow bandwidth to peerA
? I would think we would want to take the smallest MTU...which reduces bandwidth (smaller packets) for peers that are connected over links/routes that support larger MTU...but avoiding fragmentation on the routes/links that have lower MTU support. Or did I misunderstand something? Maybe small packets are slower than fragmentation?You can manually repeat the wg-quick MTU detection process to see how it works. But ultimately it is relying on the MTUs the networking stack reports for routes/links to the endpoints. So if these MTUs are wrong or not-detectable...then there is a more general problem that will impact your networking performance for more than just wireguard. I think the suggestions here are good to make sure you are doing what you can to not prevent your networking stack from detecting MTUs correctly.
Of course if your ISP is doing stupid things, then you may need to resort to empirical testing.
Biggest footgun here is setting MTU manually in config rather than letting wg-quick do it's auto-detection. You can also let wg-quick detect/set the mtu and then look at what MTU was set by wg-quick (
ip link show dev wg0
) and then further reduce that to see if it helps. The MTU tradeoff is: bigger packets mean higher throughput...until it doesn't: if your packets are too big for the link/route then you have fragmentation and things get slow. If your packets are too small then it is also slow. Ideal MTU (largest packet without fragmentation) is: actual supported MTU by the route/device minus wg overhead. You can usemtu - 60
for instance if you know you will only use ipv4.If you have a mix of MTUs to your various peers...I would expect that choosing the SMALLEST would be the best compromise because I would expect throughput decrease of smaller packets to be less of a slowdown than fragmentation of too-large packets. But I'm not an expert. Interested for feedback on this point.