Wireguard Optimal MTU

optimal_mtu.md

About

• I faced bandwidth issues between a WG Peer and a WG server. Download bandwidth when downloading from WG Server to WG peer was reduced significantly and upload bandwidth was practically non existent.
• I found a few reddit posts that said that we need to choose the right MTU. So I wrote a script to find an optimal MTU.
• Ideally I would have liked to have run all possible MTU configurations for both WG Server and WG Peer but for simplicity I choose to fix the WG Server to the original 1420 MTU and tried all MTUs from 1280 to 1500 for the WG Peer.

Testing

• On WG server, I started an iperf3 server
• On WG peer, I wrote a script that does the following:
  • wg-quick down wg0
  • Edit MTU in the /etc/wireguard/wg0.conf file
  • wg-quick up wg0
  • iperf3 -c 172.123.0.1 -J -t 5 -i 5
    • This runs an iperf3 client that connects to 172.123.0.1 which is the WG Server gateway
    • The iperf3 client runs for 5 seconds and stops and dumps a JSON output

Setup

• Max bandwidth provided by my ISP (1000Mbps Download, 50Mbps Upload)
• WG Server is a VPS running Ubuntu 20.04 on a cloud provider.
• WG Peer is a PC running Ubuntu 20.04 locally at home.

Config

I followed this tutorial to setup my Wireguard configurations.

WG-server

# /etc/wireguard/wg0.conf
[Interface]
Address = 172.123.0.1/24
MTU = 1420
SaveConfig = true
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; iptables -t nat -A POSTROUTING -o ens10 -j MASQUERADE; ip6tables -t nat -A POSTROUTING -o ens10 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; iptables -t nat -D POSTROUTING -o ens10 -j MASQUERADE; ip6tables -t nat -D POSTROUTING -o ens10 -j MASQUERADE
ListenPort = 51820
PrivateKey = xxxxxxxxxxxxxxxxxx

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxx
AllowedIPs = 172.123.0.2/32
Endpoint = X.X.X.X:61426

WG-peer

# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = xxxxxxxxxxxxxxxxxx
ListenPort = 51820
Address = 172.123.0.2/24
MTU = 1384

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxx
AllowedIPs = 172.123.0.0/24, 10.1.0.0/24
Endpoint = Y.Y.Y.Y:51820
PersistentKeepalive = 5

Conclusions

• As you can see in the image, the original MTU setting of 1420 for both peer and server gives abysmal bandwdith.
• I found that that MTU 1384 on the WG peer with 1420 on the WG server seems to almost have the best bandwidth.
• For WG Peer MTU 1384, the max upload bandwidth of 50Mbps of my ISP connection is achieved but I was only able to hit 550 Mbps for the download bandwidth where the max download bandwidth of my ISP connection is 1000 Mbps. This reduction in download bandwidth might be due to other factors but 550 Mbps was sufficient for my use cases so I stopped testing it further.

In case any one has any explanations for this behavior or have found some mistakes in my configurations or tests, please let me know.

peer_mtu_vs_bandwidth.png

wireguard_peer_mtu.csv

thanks!

FYI, there's better data and plots that can be found on a very related project of mine. The plot above is only for a fixed server MTU of 1420 and different peer MTUs. The plot in the other repo is 2D which covers different values of both server and peer MTUs.

project: nr-wg-mtu-finder
example csv and png folder: examples

@nitred thx for input! script is running and will test my WG env...

@skydiablo Did you happen to finish running the test? I'm quite curious to know if the tool helped and if you were able to generate a plot?

I'm really unsure to make it sense on a different MTU on Server and client Side. I have calc my MTU by real logical facts.

@skydiablo That's alright, I was just curious. If you were able to calculate the MTU and it worked for you, that's great!

@nitred Wow! Great work work!

Have you tried to reproduce your test?

Have anyone tried to do this test in controlled environment (e.g. in LAN)?

Just tested bandwidth between router at work and cloud VPS server.
I haven't faced any issues using MTU 1420 at both sides.
I believe the issues with MTU larger than 1400 is related to @nitred's ISP.

Have you tried to reproduce your test?

Have anyone tried to do this test in controlled environment (e.g. in LAN)?

@Harliff I was able to get the same results (i.e. cutoff at 1400) when I repeated the test on a new cloud VM which acted as a WG server while my PC acted as WG peer. I also did a test where my Raspberry Pi (in LAN) acted as the WG server while my PC acted as a WG peer and it gave different results to the one above which didn't have any hard drop outs.

I believe the issues with MTU larger than 1400 is related to @nitred's ISP.

I think you're right that there's an issue somewhere between my router and cloud VPS server and very likely with my ISP.

However the point wasn't to say an MTU of 1420 is bad for everyone, it's just that there's an optimal MTU for every configuration of WG Server-Peer setups. Here's a image with a more extensive test which plots the bandwidths when WG Peer's and WG Server's MTUs are altered. I created a repo which one can use to create such plots - nitred/nr-wg-mtu-finder

AFAIK it makes no sense to set different MTUs for both ends of a connection. The MTU is basically limited by the headers of your ethernet frames. if you haven't enabled jumbo frames (MTU = 9000) for all your connections this is fixed to 1500. For example, I dropped mine to 1400 because I send IPv6 wireguard headers inside an IPv6 internet connnection. I needed the lower MTU so both fit into each other I could go a little higher if I used IPv4 for both because of the smaller header size of IPv4 frames. You pretty much always want to use the biggest possible MTU to maximaize your throughput and efficiency. It's very easy to calculate that. Just look at your headers.
There are very few factors out of your control that could have an effect on your MTU choice, like if you're connecting via a DSL line you have to make space for the PPPOE overhead or if your connecting through an overlay network like an MPLS cloud.

But for normal everyday connections you know all the components involved or have no control over them anyway. SO just calculate for what you know and use the biggest value possible.

@nitred Great job anyway! I'm not really familiar with all the tricks and calculations over MTUs, but page gave me more understanding how to fix this or that stuff, than reading tons of theoretical articles.

I would argue that the whole approach of trying to "tune" MTU is wrong to start with and any guide that you can find, even a good one, is both hard to know if it's applicable to your own circumstances and very prone to breakage as any network change can completely invalidate your configuration.
Not to mention that you (at most) only control your own half of the pipe and you will be communicating with a host that has it's own whole set of unknown and variable circumstances.

What you instead should try to do is make sure that the built in mechanism for auto-discovering MTU (Path MTU Discovery, or PMTUD) is enabled and functioning. Mainly this simply involves allowing ICMP traffic through your firewall and letting it take care of controlling your traffic. The functionality is even in the name, Internet Control Message Protocol.

However because of a long history of misinformation, and a false sense of security, most people are actively blocking all ICMP in their firewalls which can actually do quite a lot of damage to the quality of connections.

The other thing is to clamp MSS for TCP connections to PMTU, look up the --clamp-mss-to-pmtu option in iptables. This helps correct misbehaving clients however if PTMUD is truly broken you will still experience issues. And of course in the best case this is still only valid for TCP.

So my general advice for any budding network operator would be to do the following:

1. Allow ICMP and ICMPv6 in firewalls to allow PMTUD to work. If you desperately want to "hide", block only ICMP Echo/Reply types (ping). This is VERY important and if you only choose to do one thing, do this one.
2. Clamp MSS to PTMU in firewall.
3. Stop trying to micro-optimize your connection for performance,as it is extremely unlikely you are limited by Packets Per Second, which is what you are really tuning for by changing MTU. Only tinker with defaults if you are actually having problems.
4. If you are having problems, set a reasonably low value without being too extreme. I'd even recommend choosing the same value for IPv4 and IPv6, despite the header differences. IPv6 has 1280 default MTU, which is a MSS of 1260. Go with that for both protocols. You are extremely unlikely to ever notice the slightest difference in performance in a small network. However "MTU blackholing" should largely be a thing of the past so, again, it is very unlikely you will need to touch your MTU beyond defaults.

This is a very good article for anyone who finds this discussion interesting:
https://blog.apnic.net/2019/07/31/tcp-mss-values-whats-changed/
It suggests that staying

block only ICMP Echo/Reply types

Ping is the part of ICMP that doesn't need blocking. There are much more dangerous features of ICMP though, such as type 5 (redirect).

I was getting ~35 Mbits/sec inside of my tunnel with having a MTU of 1500 both on client and server (chosen at random), after testing your values (i know everyone should find out what their own are) im getting ~240 Mbits/sec
I had no idea this little setting changes so much, thank you!

@y0nei I'm glad it helped! I know what you mean, it's a single parameter that I was able to tweak and gain significant bandwidth improvement. In my case it went from ~0 Mbps to ~50Mbps. Then I was able to move on to what I really was using the VPN for.

Wow, I'm sitting 3000km from home right now with a very poor internet connection, and after reading this thread, I just reduced the MTU on both sides of my WireGuard connection to 1400 (as my understanding is that this is the size WireGuard will give the packets that go through the tunnel, right?). And now, after doing so, my VPN has become workable and feels quite fast, whereas it wasn't usable at all before.

So my understanding is that the VPN MTU should be below the MTU of the underlying real network connection so that the VPN packets "fit" into the network packets and won't be split up, which would cause a reduction in usable bandwidth. Is that correct?

block only ICMP Echo/Reply types

Ping is the part of ICMP that doesn't need blocking. There are much more dangerous features of ICMP though, such as type 5 (redirect).

Yeah I would personally never block Echo/Reply because it makes troubleshooting a big pain in the ass. But that sentence was more a comment on IF you want to block ICMP to "hide", only block (or rather, drop, so your computer wont reply at all) those two types and not all of ICMP (which is was most people do). Blocking ICMP entirely will break your connection in various ways.

As for redirects, most operating systems nowadays don't really accept them:
[svag@tuxxx] sysctl net.ipv4.conf.all.accept_redirects
net.ipv4.conf.all.accept_redirects = 0
[svag@tuxxx] sysctl net.ipv4.conf.default.accept_redirects
net.ipv4.conf.default.accept_redirects = 0

But blocking them in firewall anyway is good for a layered approach to security.

So my understanding is that the VPN MTU should be below the MTU of the underlying real network connection so that the VPN packets "fit" into the network packets and won't be split up, which would cause a reduction in usable bandwidth. Is that correct?

When MTU is not aligned to the actual MTU of the link, then every connection you make has to rely on Path MTU Discovery to work. But PMTUD is often broken due to firewalls and middleboxes. That results in packets sized above the actual MTU not reaching their target at all thus creating packetloss - especially with UDP; which is very noticable with DNS traffic and of course also with gaming.

block only ICMP Echo/Reply types

Ping is the part of ICMP that doesn't need blocking. There are much more dangerous features of ICMP though, such as type 5 (redirect).

Asuka, ping is primarily a utility...

Wireguard does not default to MTU 1500. It defaults to 1500 - 80 ...but only if all other attempts to detect your connection MTU fail. I'm talking about wg-quick helper script here. MTU config setting doesn't mean anything to wg (which expects you to manage ip/link/route/addr stuff yourself).

This is because wireguard has 80 byte overhead for ipv6 and 60 byte overhead for ipv4. You may also find this page helpful.

Relevant code is here.

Here is the MTU setting logic from wg-quick:

• if an mtu is set in the config file, the link is brought up with that mtu and we are done. 80 is NOT subtracted from the value.**
• otherwise, we move on to "auto detection" of MTU
• wg-quick grabs all endpoints (IPs of peers) from the wireguard interface (using wg0 as example)
  • wg show wg0 endpoints
• attempts to parse mtu for each endpoint by running ip route get <endpoint address> and looking for mtu [0-9]+ in output.
  • if no mtu is found in the output, then the device for the route (dev [^ ]+) is taken instead and used in ip link show dev <dev-for-route>...and the mtu (mtu [0-9]+) is extracted from that output.
• the largest of all endpoint mtus found are kept***
• if at the end of checking all wg peer endpoints (routes/devices) we still did not find ANY set mtu (on output of ip route get <addr> or ip link show dev <endpoint-dev>...then wg-quick tries to get the mtu from ip route show default
• if we still have not found any mtu, wireguard defaults to 1500.
• whatever mtu value we have at this point...wg-quick brings up the link with mtu - 80.

** This means if your eth0 MTU is 1500 and you explicitly set MTU to 1500 in wg-quick config, 1500 will be used as-is (not 1420)...so you will have problems because your wg packets will end up larger than what is supported on the underlying device/route and so there will be fragmentation (slow because packets have to be broken up).

*** Is this right? If we have a peer endpoint A with 1200 MTU on eth1 and another peer endpoint B with 1500 MTU on eth2, it appears to me that wg-quick keeps the 1500 MTU from B (minus 80 at the end)...meaning packets for peer A may be larger than the 1200 MTU supported on eth1 causing fragmentation and slow bandwidth to peer A? I would think we would want to take the smallest MTU...which reduces bandwidth (smaller packets) for peers that are connected over links/routes that support larger MTU...but avoiding fragmentation on the routes/links that have lower MTU support. Or did I misunderstand something? Maybe small packets are slower than fragmentation?

You can manually repeat the wg-quick MTU detection process to see how it works. But ultimately it is relying on the MTUs the networking stack reports for routes/links to the endpoints. So if these MTUs are wrong or not-detectable...then there is a more general problem that will impact your networking performance for more than just wireguard. I think the suggestions here are good to make sure you are doing what you can to not prevent your networking stack from detecting MTUs correctly.

Of course if your ISP is doing stupid things, then you may need to resort to empirical testing.

Biggest footgun here is setting MTU manually in config rather than letting wg-quick do it's auto-detection. You can also let wg-quick detect/set the mtu and then look at what MTU was set by wg-quick (ip link show dev wg0) and then further reduce that to see if it helps. The MTU tradeoff is: bigger packets mean higher throughput...until it doesn't: if your packets are too big for the link/route then you have fragmentation and things get slow. If your packets are too small then it is also slow. Ideal MTU (largest packet without fragmentation) is: actual supported MTU by the route/device minus wg overhead. You can use mtu - 60 for instance if you know you will only use ipv4.

If you have a mix of MTUs to your various peers...I would expect that choosing the SMALLEST would be the best compromise because I would expect throughput decrease of smaller packets to be less of a slowdown than fragmentation of too-large packets. But I'm not an expert. Interested for feedback on this point.

@nitred : thank you so much! It is very helpful. Just an observation. It seems that you don't need to put MTU in the wg0.conf for server, as it is automatically done. Per @mattpr .

# ip a
...
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000

Only on the client they should use a smaller MTU size via wg conf.

Thanks to @Svaag , I am adding this to the iptables firewall on the server, and the client's test confirmed it worked without making the conf mod for MTU size.

iptables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1384

It seems for some clients, 1400 (MTU 1420-20) is not good enough, thus I am not using: --clamp-mss-to-pmtu

Ref: https://blog.vitlabuda.cz/2022/10/15/clamping-tcp-mss-using-iptables.html