Skip to content

Instantly share code, notes, and snippets.

@marfillaster
Last active June 18, 2024 13:59
Show Gist options
  • Save marfillaster/7a136ea826815ac22f2849e099a1c6a1 to your computer and use it in GitHub Desktop.
Save marfillaster/7a136ea826815ac22f2849e099a1c6a1 to your computer and use it in GitHub Desktop.
MikroTik RouterOS v7 dual DHCP WAN recursive failover w/ PCC load-balancing; and recursive ECMP
# feb/11/2022 11:00:55 by RouterOS 7.2rc3
# software id = 9QK9-C798
#
# model = RB5009UG+S+
# serial number = XXXXXXXXXX
/ip settings set allow-fast-path=no
/interface bridge add admin-mac=FF:FF:FF:FF:FF:FF auto-mac=no name=bridge
/interface bridge port add bridge=bridge ingress-filtering=no interface=ether3
/interface bridge port add bridge=bridge ingress-filtering=no interface=ether4
/interface bridge port add bridge=bridge ingress-filtering=no interface=ether5
/interface list add name=WAN
/interface list add name=LAN
/interface list member add interface=bridge list=LAN
/interface list member add interface=ether1 list=WAN
/interface list member add interface=ether2 list=WAN
#/interface bridge port add bridge=bridge ingress-filtering=no interface=ether6
#/interface bridge port add bridge=bridge ingress-filtering=no interface=ether7
#/interface bridge port add bridge=bridge ingress-filtering=no interface=ether8
#/interface bridge port add bridge=bridge ingress-filtering=no interface=sfp-sfpplus1
/ip address add address=192.168.88.1/24 interface=bridge network=192.168.88.0
/ip dns static add address=192.168.88.1 name=router.lan
/ip pool add name=pool1 ranges=192.168.88.10-192.168.88.254
/ip dhcp-server add address-pool=pool1 interface=bridge name=dhcp1
/ip dhcp-server network add address=192.168.88.0/24 dns-server=192.168.88.1 gateway=192.168.88.1
/ip dhcp-client add interface=ether1 add-default-route=no script=":if (\$bound=1) do={\r\
\n /ip/route/set [find where comment=\"ISP1\"] gateway=\$\"gateway-address\"\r\
\n}\r\
\n\r\
\n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
\n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
\n" use-peer-dns=no use-peer-ntp=no
/ip dhcp-client add interface=ether2 add-default-route=no script=":if (\$bound=1) do={\r\
\n /ip/route/set [find where comment=\"ISP2\"] gateway=\$\"gateway-address\"\r\
\n}\r\
\n\r\
\n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
\n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]" use-peer-dns=no use-peer-ntp=no
/routing table add fib name=to_ISP1
/routing table add fib name=to_ISP2
/ip route
# recursive routes for ECMP default gateways, dst-address are public DNS servers
add distance=1 dst-address=9.9.9.9/32 gateway=ether1 scope=10 target-scope=10 comment=ISP1
add distance=1 dst-address=8.26.56.26/32 gateway=ether2 scope=10 target-scope=10 comment=ISP2
# ECMP default gateways
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=8.26.56.26 scope=10 target-scope=11
# recursive routes for default gateways, dst-address are public DNS servers
add dst-address=64.6.64.6/32 gateway=ether1 scope=10 comment="ISP1"
add dst-address=208.67.220.220/32 gateway=ether1 scope=10 comment="ISP1"
add dst-address=208.67.222.222/32 gateway=ether2 scope=10 comment="ISP2"
add dst-address=64.6.65.6/32 gateway=ether2 scope=10 comment="ISP2"
# load-balanced w/ auto failover default gateways
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=64.6.64.6 routing-table=to_ISP1 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=64.6.65.6 routing-table=to_ISP1 scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=208.67.222.222 routing-table=to_ISP2 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=208.67.220.220 routing-table=to_ISP2 scope=10 target-scope=11
/ip firewall address-list add address=192.168.88.0/24 list=local
/ip firewall mangle
add action=accept chain=prerouting comment="bridge access" dst-address-list=local in-interface-list=LAN
# WAN to LAN
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related in-interface=ether1 new-connection-mark=ISP1_conn \
passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related in-interface=ether2 new-connection-mark=ISP2_conn \
passthrough=yes
# PCC mangles
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!local dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!local dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1_conn dst-address-list=!local new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn dst-address-list=!local new-routing-mark=to_ISP2 passthrough=yes
# masquerade
/ip firewall nat add action=masquerade chain=srcnat ipsec-policy=out,none out-interface-list=WAN
@marfillaster
Copy link
Author

@bwraith that’s what the ECMP default gateway routes are for. I added it for DNS resolver but I think it applies to the traffic you described also.

@bwraith
Copy link

bwraith commented Jul 15, 2022

I realize that because I only added the parts relating to the public ping responders to a configuration I got from another place (that only ping-checked the ISP device, not outside connectivity, and therefore did not work if the ISP's device worked, but the ISP itself had no service), I did not have the last two chain=output commands in the mangle section. Could you comment on why my configuration still worked and what those chain=output mangle rules do? Also, don't understand why the ECMP routes don't conflict in some way with the PCC routes. Sorry if this is too many questions.

@bwraith
Copy link

bwraith commented Jul 15, 2022

I also notice that if I add those chain=output mangle rules, my traceroute command works but does not list 192.168.88.1 (the router itself), and it times out on that first line before reporting the rest of the paths correctly.

@marfillaster
Copy link
Author

@bwraith noticed the tracert issue too. I added this mangle rule and before the pcc rules.

/ip firewall mangle
add action=accept chain=prerouting comment=traceroute dst-address-list=!local in-interface-list=LAN protocol=icmp

I'm not really sure what kind of traffic are still getting processed in chain=output. They match marked connections that have no routing mark yet.

@oakwhiz
Copy link

oakwhiz commented Jul 22, 2022

Interesting config, I'm curious if there is any specific reason that line 86 has connection-state=new while line 84 does not have that.

@marfillaster
Copy link
Author

@oakwhiz good catch! That shouldn't be there.

@bwraith
Copy link

bwraith commented Jul 22, 2022

@bwraith noticed the tracert issue too. I added this mangle rule and before the pcc rules.

/ip firewall mangle
add action=accept chain=prerouting comment=traceroute dst-address-list=!local in-interface-list=LAN protocol=icmp

I'm not really sure what kind of traffic are still getting processed in chain=output. They match marked connections that have no routing mark yet.
I tried the above mangle rule and a few variations, and I still get the traceroute hanging up. Are the ECMP routes necessary for this to work? I tried just putting in a default route in the main table and that did not seem to help. I wonder if there is a firewall rule that I'm missing? Thanks for any insight.

@marfillaster
Copy link
Author

You can either have the dhcp clients add the default routes in which case the secondary default route will be used as failover. Unlike ECMP, both routes will be used.

For traceroute from LAN, I think ECMP is not necessary.

I realized the accept mange rule for ICMP will not be processed by the succeeding PCC rules. I don't have a yet the solution.

@bwraith
Copy link

bwraith commented Jul 22, 2022

I added the ECMP routes and still I get the traceroute delayed and not displaying result for the router itself. Thanks again for any insight.

@bwraith
Copy link

bwraith commented Jul 22, 2022

If I remove the chain=output mangle rules at the end, things still seem to work, but maybe I'm not noticing something important that is happening with those rules. Unfortunately, I guess I need to get myself a course or tutorial on the details of routing rules.

@marfillaster
Copy link
Author

Yeah the output chain is still a mystery to me. Even in https://help.mikrotik.com/docs/display/ROS/Firewall+Marking, it was not explained.

@marfillaster
Copy link
Author

@bwraith I removed the ICMP accept rule and updated the output to check for destination. It seems to work.

@bwraith
Copy link

bwraith commented Jul 22, 2022

@bwraith I removed the ICMP accept rule and updated the output to check for destination. It seems to work.

Yes, that seems to have worked for me, too. What is the difference between dst-address-list and dst-address-type. I see that both can be set to !local, but that you added an address list in the firewall.

@oakwhiz
Copy link

oakwhiz commented Jul 24, 2022

I haven't yet had success with the routing side of the config.

@lucasasdelli
Copy link

Hello, thanks for posting it. Unfortunately with the two ECMP default gateways rules I get invalid and unreachable. I'm not sure if they should use the main route of one of the two PCC ones.

@oakwhiz
Copy link

oakwhiz commented Jul 27, 2022

Testing several variants of this config with PCC set to destination only, and with either A) both ISPs connected, or B) a single ISP interface set up with a working link that has IP transit blocked via a switch, shows incorrect behavior. For example with both ISPs connected, traffic marked for ISP1 will only go through ISP2. Blocking ISP2 so that it silently fails breaks all default-destined traffic, but traffic destined for the directly connected on-link subnet at ISP1 still works.

My thought is that something is wrong with the routing, and my next step is to see if adding a 3rd layer of nexthop routes would actually fix this. In this configuration there are only two layers: 0.0.0.0/0 default to fake gateway inside routing tables, and fake gateway via real gateway inside the main table. However not all resources on this topic mention the intermediate nexthop, and in v7 the target-scope must be adjusted to match (might be 12>11>10 or 13>12>11).

Any discussion would be appreciated as the Mikrotik documentation is always a bit terse, ambiguous, and not so thorough.

@oakwhiz
Copy link

oakwhiz commented Jul 27, 2022

I encountered this bug: https://forum.mikrotik.com/viewtopic.php?t=185950
and in addition I was using a gateway which was not set up to forward to a default route - after using the proper gateway things are starting to improve.

@oakwhiz
Copy link

oakwhiz commented Jul 28, 2022

I have posted a partial configuration which might help someone: https://gist.github.com/oakwhiz/55b4043e99320129323496ffd5087f05
I should mention though, there are some minor errors in there and various things set up for debugging/testing purposes, I wouldn't recommend copying it.

@dipan29
Copy link

dipan29 commented Nov 30, 2022

Can anyone please explain what this area is doing?

/ip dhcp-client add interface=ether1 add-default-route=no script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP1\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n" use-peer-dns=no use-peer-ntp=no

I have one Static WAN where I need to setup but how do I add the script?

@marfillaster
Copy link
Author

@dipan29 That is for resetting all connections whenever the DHCP client changes its IP. If your other uplink is a static IP, you only need one DHCP client.

@dipan29
Copy link

dipan29 commented Dec 1, 2022

@marfillaster thanks for the response,

Also, I tried doing the same and gracefully it works for the local network right out of the box. As in for all interface lists LAN. However, My requirement was a little different and could not manage to get it working post Router OS 7.

I would require something similar but for the following scenario

  • Having one DHCP Client and one with Static IP or PPPoE Client (here the gateway caused an issue when I tried to replicate cause for ether1 when its PPPoE the gateway IP isn't defined)
  • Ensuring that both the WAN IPs can be used for DST-NAT - for any local subnet
  • I would be having generally two subnets - one for home lab and one for normal home users. For the home lab I won't really want to do load balancing but just ensure that the DST-NAT for that network works from both the WANs.
  • For normal home users, I would be okay if it goes out through ether1 (PPPoE/Static) most of the times but have an address-list called DUAL say that can only get Load Balancing when tagged to it. In other words, only the addresses tagged as DUAL can have the load balancing.

I tried finding a lot of guides on the internet but could not have something from reddit or mikrotik forum that serves my purpose (some claim to, but they just don't get connected).
Also, I am new to this mikrotik, would be glad if you can guide me with some exact details or something from the internet that works.

@rexsllemel
Copy link

rexsllemel commented Feb 27, 2023

I'm using this config but, when the ISP 1 has no internet, it uses the ISP 2, however, when the ISP 1's internet get's back, the recursive will still mark ISP 1 as no internet. It only uses ISP 2's internet the whole time, unless I'll restart the mikrotik.

@nhan6310
Copy link

@marfillaster thanks for the response,

Also, I tried doing the same and gracefully it works for the local network right out of the box. As in for all interface lists LAN. However, My requirement was a little different and could not manage to get it working post Router OS 7.

I would require something similar but for the following scenario

  • Having one DHCP Client and one with Static IP or PPPoE Client (here the gateway caused an issue when I tried to replicate cause for ether1 when its PPPoE the gateway IP isn't defined)
  • Ensuring that both the WAN IPs can be used for DST-NAT - for any local subnet
  • I would be having generally two subnets - one for home lab and one for normal home users. For the home lab I won't really want to do load balancing but just ensure that the DST-NAT for that network works from both the WANs.
  • For normal home users, I would be okay if it goes out through ether1 (PPPoE/Static) most of the times but have an address-list called DUAL say that can only get Load Balancing when tagged to it. In other words, only the addresses tagged as DUAL can have the load balancing.

I tried finding a lot of guides on the internet but could not have something from reddit or mikrotik forum that serves my purpose (some claim to, but they just don't get connected). Also, I am new to this mikrotik, would be glad if you can guide me with some exact details or something from the internet that works.

I can help you with all your requests

@aaronbolton
Copy link

Has the variable $leaseBound change to $bound now?

@dipan29
Copy link

dipan29 commented Aug 12, 2023

Hi @nhan6310 how can we connect to get this configuration set?

@quinont
Copy link

quinont commented Aug 12, 2023

I tested this on a MikroTik RB4011, and it worked perfectly. However, I'm encountering an issue that perhaps you could assist with: the load balancing is happening constantly (meaning a PC is continuously switching between the two WANs). Is there a way to configure WAN balancing based on the client's IP?

@aaronbolton
Copy link

aaronbolton commented Aug 12, 2023

@quinont the easiest way to to change this is to change the pre routing mangle rule to included a src address list and then you can specify which IP get load balanced or not

Below is how mines looks

add action=mark-connection chain=prerouting
connection-mark=no-mark dst-address-type=
!local in-interface-list=LAN
new-connection-mark=ISP1_conn passthrough=yes
per-connection-classifier=
both-addresses-and-ports:2/0 src-address-list=
MultiWAN-Clients
add action=mark-connection chain=prerouting
connection-mark=no-mark dst-address-type=
!local in-interface-list=LAN
new-connection-mark=ISP2_conn passthrough=yes
per-connection-classifier=
both-addresses-and-ports:2/1 src-address-list=
MultiWAN-Clients

@yosijoe
Copy link

yosijoe commented Dec 21, 2023

any one can give me recursive fail over gate way
im using NTH
my ip : gateway
my dns :8.8.8.8,8.8.4.4,1.1.1.1
wan1 192.168.1.1
wan2 192.168.254.254
wan3 192.168.0.1

@adrianolima83
Copy link

Jovem, que configuração excelente. Testamos em laboratório de bancada, fizemos testes de estresse, usamos conexões diferentes (pppoe, client, etc), e tudo funcionou perfeitamente. Meus parabéns e muito obrigado pelo compartilhamento.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment