I'm a user of mwan3 and contribute to its development in a small way by mainly providing feedback with my multi WAN setup and maintaining the beast of it's documentation on the OpenWrt wiki (feedback and contributors welcome).
This setup ultimately requires the use of a NAT6 firewall script. NAT6 is currently broke with fw3 and LuCI, so this is an important helper script to workaround this current limitation.
The NAT6 configuration requirements are explained in more detail on the OpenWrt wiki.
This gist is aims to document my configuration for others.
I use version 2.8.15. Currently this is the latest stable build version for the 19.07 branch. Right now mwan3 is currently undergoing some major development to be made compatiable with the next major version of OpenWrt, 20.x, but it is considered to be quite bleeding edge currently. If you are using snapshot builds in a production scenario (because you clearly like to live dangerously!) you might want to give version 2.10 onwards a try, but be warned, here be dragons! Be prepared to debug and report issues to the maintainers of mwan3 on the OpenWrt packages issue tracker.
You don't need to have a router that has more than WAN port by design, you can instead reconfigure the switch ports and convert one or more LAN ports into independent WAN ports, this is achieved through VLANs. This offers a few advantages mainly:
- Flexibility of creating multiple WAN ports as required
- No need to run multiple routers in your network
- No need to purchase specific routers with dual or multiple WAN ports
- No need for expensive enterprise kit. (yay).
I use a Linksys WRT3200ACM as my main router and this has good OpenWrt support (ignoring the Marvell WiFI driver drama) overall and is VLAN capable.
If you can beat this amount of network interfaces, you've done well! This isn't going to be your typical dual WAN setup, but it does help having a lot of diverse connections to really test mwan3 and it's functionality. With the default firewall mask 0x3F00, mwan3 supports up to 60 physical or logical interfaces, so we are comfortably below any limitations here.
Using the interface name, rather than device name, this is my full list of network interfaces as a summary:
wan- Virgin Media DOCSIS 3.0 Cable (DHCP)wan6- Not active. Reserved for when this provider enables IPv6wanb- Vodafone Business Broadband (PPPoE, Static IPv4)wanb6- Not active. Reserved for when this provider enables IPv6wanc- EE 4G broadband (Static IPv4 configuration to another router)wanc6- EE 4G broadband (DHCPv6 auto configuration to another router)aaisp- Andrews and Arnold L2TP tunnel (Static IPv4)aaisp6- Andrews and Arnold L2TP tunnel (delegated /48 IPv6 prefix)henet- Hurricane Electric 6in4 IPv6 tunnel (bound towanb)wg- Wireguard client to Mullvad.netwg6- Static /128 ULA IPv6 address provided by configwgb- Wireguard client to Mullvad.netwgb6- Static /128 ULA IPv6 address provided by config
All IPv6 interfaces that are in use have been split. This is achieved using alias interfaces. This is currently required because mwan3 does not currently handle dual stack interfaces. It expects any interface to be one specific family i.e. IPv4 or IPv6 not both.
docsismodem- My cable modem for mywan. Allows me to access the modem at 192.168.100.1. This is in bridge mode.vigormodem- My Draytek Vigor 130 DSL modem doing passthrough forwanb, configuring my router with a static address within it's network range 192.168.2.0/24, so I can access the modem information.
Clearly my network setup has a lot of network interfaces, the main reason for this is due to wanting IPv6 connectivity across all WANs configured. This however is a problem when your provider itself doesn't provide it :(! In order to have proper failover for both IPv4 and IPv6 for each WAN, IPv6 needs to exist in some form for each member policy after all. While not having IPv6 wouldn't break anything, I'd essentially have unbalanced IPv6 policies, which could lead to interesting results with failover.
The summary of my network interfaces and connections is:
- Three physical indepedent WAN connections from three different providers (WAN, WANB and WANC, with their IPv6 interface counterparts). Not all
wanX6interfaces are used, because they aren't actually direct IPv6 from the WAN connection in some cases as explained above) - Two Wireguard connections from Mullvad.net connecting to two different tunnel endpoints (different colo providers)
- Two tunnel based connections (L2TP and 6in4).
I pretty much use mwan3 for failover, because my independent internet connections are not equal, so load balancing doesn't really make a lot of sense. The exception to this is for my two Wireguard connections, these are load balanced.
Most of the NAT6 configuration and heavy lifting is handled by the firewall.nat6 firewall script. Once this is in place, to activate NAT6 on a firewall zone, you need to add option masq6 '1' to the zone in /etc/config/firewall. Essentially, if you have option masq '1' set on a zone already, add option masq6 '1' as well. There is also a masq6_privacy option. This essentially makes the outbound IPv6 address a temporary address that changes regularly, the idea being to make it harder to track you, but I wouldn't rely on this as a privacy feature, instead use TOR or a VPN. Equally not all network interfaces support it even when enabled.
All my WAN interfaces are attached to the WAN zone that is provided by default as it's the most easiest solution to build off. The exception is the Wireguard interfaces, as they don't have the same firewall requirements as the physical WAN network interfaces, given no inbound configuration is needed. So another Wireguard firewall zone exists. In addition, I have made an unrelated change to the WAN zone by changing the Input and Forward chains to DROP rather than REJECT, just to stop wasted CPU cycles of my router sending a response to bots or script kiddies probing my IPs, given I have multiple external static addresses configured on my route.
The firewall.nat6 script can be partially broken if your firewall configuration doesn't indiviually list network interfaces like so:
config zone
option name 'wan'
option output 'ACCEPT'
option masq '1'
option mtu_fix '1'
option masq6 '1'
option masq6_privacy '1'
option forward 'DROP'
option input 'DROP'
list network 'wan'
list network 'wan6'
....
If uci show firewall outputs something like the example below:
firewall.@zone[1].network='wan wan6...'
Amend /etc/config/firewall to the list network format shown above accordingly, otherwise part of the script will fail when reporting network devices inspected, as it will not be able to loop through the network interfaces assigned to that specific firewall zone.
When installing the NAT6 script /etc/firewall.nat6 make sure to add this file path to the /etc/sysupgrade.conf file so it is kept on any further firmware upgrades, otherwise the NAT6 config will be lost on any sysupgrade and you'll have to re-apply it.
mwan3 currently doesn't perform any IPv6 related helper configuration, because it tries to remain agnostic and instead would expect the user to configure whatever setup they want. All of the NAT6 configuration is mostly at the firewall level and doesn't require any specific configuration in mwan3 other than appropriately configuring interfaces and policies as you would with IPv4.
When dealing with multiple independent IPv6 interfaces you may encounter permission denied or issues when trying to use ping or traceroute. This is due to a kernel protection relating to source addresses routing behaviour. This is why several route6 rules are present in the network config with a target of ::/0, this is to avoid this issue so the interfaces allow IPv6 traffic properly. Otherwise, it will be broken and mwan3 will mark the interfaces as down because the ping test mwan3track will perform will fail, getting a permission denied respone from the kernel.
This is something that occurrs with OpenWrt 19.07 and possibly older versions. I don't believe you have to implement this for snapshot builds, given the upstream kernel and mwan3 changes present in version 2.10 and above.
L2TP and possibly other PPP based network interfaces seem to be broken when fwmark is applied to packets for these interfaces, this was discovered when using L2TP and mwan3 together. Massive credit to @aaronjg who helped debug this issue. It seems as soon as a single fwmark is applied, routing would break horribly. I explain the issue a bit more detail over on my Medium article.
The fix is to apply the following iptables rule as a custom rule to workaround the issue, preventing mwan3 from marking these packets and triggering the issue, although it is not a specific issue with mwan3 but fwmark and routing in the kernel.
iptables -t mangle -I OUTPUT -d 90.155.53.19 -p udp --dport 1701 --sport 1701 -j RETURN90.155.53.19 is the L2TP endpoint of AAISP.
This doesn't occur on snapshot builds and was fixed in OpenWrt release 19.07.5, with backported kernel patches.
I'll acknowledge NAT6 has many disadvantages and I don't neccersarily recommend it is the solution for everyone. Some people will hate me for even mentioning NAT and IPv6 together. An alternative is NETMAP however this requires you to have a prefix to be mapped for every ULA prefix defined and in my case this isn't possible for two of my WAN connections.
Inbound IPv6 communication is still possible with NAT6. If you have at least one prefix delegated by your ISP (ideally as static), inbound rules can be configured through standard firewall rules.
My use of NAT6 is because I ultimately have a mix of some WANs with a delegated a prefix and others not. In addition, Mullvad's IPv6 configuration for Wireguard also uses NAT6, because they only provide a /128 ULA address on your side of the tunnel, without NAT6 here, you cannot configure Wireguard to work across the LAN, given the ULA is only usable on the router without NAT6.
Generally IPv6 masquerading makes IPv6 balancing/failover work well with mwan3. I welcome anyone using mwan3 and IPv6 with alternative configurations to provide their thoughts however. While IPv6 has been around for a long time, its adoption is still slow and therefore the use of IPv6 and mwan3 is suspected to be low, so sharing your experiences and configurations will help improve mwan3 and IPv6 support going forward.