Disable WAN Interface on CARP Backup

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

$ifkey = 'wan';

if ($type === "MASTER") {
    log_error("enable interface '$ifkey' due CARP event '$type'");
    $config['interfaces'][$ifkey]['enable'] = '1';
    write_config("enable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
} else {
    log_error("disable interface '$ifkey' due CARP event '$type'");
    unset($config['interfaces'][$ifkey]['enable']);
    write_config("disable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
}

Script that allows to have WAN DHCP with single IP address (forced to use same MAC address on WAN interface) running CARP only on LAN interface.

---

In theory, only one WAN will be active at once and independent of the node.
They should reuse the same DHCP lease from your ISP.
If a node is BACKUP, the optionally created gateway will kick in for the backup node to have internet connection via the MASTER.
This should also work with all WAN related services, due we completely disable the WAN connection and "restart" it in the case of MASTER.

---

1. to not mess with the current DHCP lease on WAN or have temporary the same MAC twice on the WAN, you should remove the cable on the WAN port during setup.

2. Setup CARP for LAN interface according to the doc's.

3. [optional] If one WAN DHCP lease should be shared between MASTER and BACKUP, set the MAC address of the WAN interface of MASTER and BACKUP to the same.

4. [optional] Create an additional WAN gateway:

   Field	Value
   Interface	LAN
   IP address	the LAN CARP VIP address
   Upstream Gateway	☑
   Priority	255 or something higher than the WAN default gateway.

   This allows the BACKUP firewall to have internet connection over the MASTER during the time the WAN interface is disabled.

5. Execute on the MASTER and on the BACKUP:

   curl -sL -H "Cache-Control: no-cache" \
     https://gist.githubusercontent.com/spali/2da4f23e488219504b2ada12ac59a7dc/raw/10-wancarp \
     --output /usr/local/etc/rc.syshook.d/carp/10-wancarp && \
     chmod +x /usr/local/etc/rc.syshook.d/carp/10-wancarp
   

6. Put the MASTER once into "Persistent CARP Maintenance Mode" and then leave the "Maintenance Mode".
   This should disable the WAN interface no the BACKUP node.

---

I'm in the middle of working my issue and I saw your post. I'm trying to do this with Dual WAN.. do I have to have a separate script for each WAN interface?

Assuming you just want to disable both WAN interfaces on the backup and enable both on the master, you can just duplicate the script with a any suffix in the filename and adjust the $ifkey variable to for the second WAN interface.

A bit cleaner solution would be adapt the script to allow to define an array for $ifkey variable to it can loop over the interfaces.

Thank you for the response.. I have that working but am now chasing DHCP issues. Neither DHCP server working (Leaving both active). I'll get there :)

Spali,
I fired off an update today and both WAN interfaces on my firewalls started going crazy.. I kept seeing the scripts invoked to up and down my WAN.. have to work for a few hours so I turned off gateway monitoring and chmod -x both scripts and it seems to have stabilized. Are you seeing anything similar after update?

Are you seeing anything similar after update?

sorry, didn't update yet to 22.1.

Works well, many thanks.

A few suggestions though:

• Maybe the script should also trigger when CARP is DISABLED. Now, if I go into the GUI and disable CARP, the WAN stays active, and plays havoc on my other server. Not sure if that is a state that can be triggered upon however.

• In my case/my vendor I had to make sure both WAN not only had the same MAC, but also the same ClientID, in order to get the same WAN-IP. Which oddly works in OPNsense by going into "advanced" and copying the MAC into the Host. (if you do this not in advanced, it interprets the MAC as a string.

Hey @cybermaus ... on your second point.. where did you find advanced to add the MAC into the host? I had it working by adding the spoofed MAC into interfaces/WAN.

My provider needed the Client-ID the same too, and for some reason, OPNsense hides/merges that in the hostname.
You need advanced, otherwise OPNsense interprets the whole thing as a string rather then 6 bytes.

When you do not specify hostname, OPNsense sends the MAC as Client-ID, sp that is why I had to spoof the MAC into the Client-ID (and on OPNsense, thus in the hostname) to get my old IP back (that I had unmodified for 6 years now). Anything else, and I got a new IP

(see image, I hope it arrives, it does not show on preview)

Got it.. I found it right about when the email came in. Many thanks. I'm testing again with the scripts enabled. So far so good. Was stable for a couple months before I upgraded :)

thanks for the script. it works pretty well! I think CARP itself is picky with the maintenance mode stuff ... I've had to trigger it on the 'master node' by disabling CARP or rebooting that node for the 'backup' to kick-in.

one thing I haven't gotten to work is creating the 'lower priority gateway' for lan interface on the 'backup node. When I try to go to the 'firmware' section on the backup node, that's when I see that it is not loading (just keeps spinning). Firewall logs show all traffic is being allowed on both node's fw logs. I've also cleared the arp on the backup node for the vip associated with the LAN interface. Have even tried to use just the IP of the master node on LAN as the gateway and still does not work for internet access on the backup.

I am having the exact same behavior as @vc1cv1

PERFECT!!! (except when ONE interface decides to go MASTER)

Made some adjustments to this script for my needs and to update to 23.1:
https://gist.github.com/kronenpj/e90258f12f7a40c4f38a23b609b3288b

Had to make some edits to make this work: https://gist.github.com/tlyakhov/15172db645d01dc67a0b585096f28ab3

This script worked great for me. I did, however, have a weird issue that I worked around. Thought someone else might find the solution useful. I'm running OPNSense on Proxmox 8. Failover worked well with this script (did everything it was supposed to do), but traffic going through what was the standby firewall (that is now active after failover) wouldn't work. A network capture after a failover verified that outbound traffic from the firewall was not being NAT'd.

I noticed that on the normal Primary firewall, auto-generated firewall NAT rules were in place. However, on the standby firewall there were no auto-generated NAT rules in place.

I put static NAT rules in place on both firewalls, sync'd them and this works like a charm. Drop maybe 4 ICMP Echo replies on a failover. I drop nothing on fail-back.

Thanks for the great work, everyone.

@flyboy747 would you care you elaborate how you did this? I'm on more or less the same setup as you, and having the same struggles atm. Thanks in advance :)

@flyboy747 would you care you elaborate how you did this? I'm on more or less the same setup as you, and having the same struggles atm. Thanks in advance :)

Yeah - So what I did was on the primary firewall, I navigated to Firewall, NAT, Outbound, changed the "Mode" to "Manual Outbound NAT rule generation". Then I added a new outbound NAT rule. I selected the "Outbound" interface, in my case "Comcast". I selected IPV4 (I don't run V6), any protocol, source address "LAN Net", source port "any", destination address and port both set to "any" and translation target "Interface Address". Everything else besides the Description are blank. You can put whatever you want in the Description, of course.

Then, I tested on the primary firewall to make sure everything was working properly still. You may have to add the ISAKMP outbound if you use that protocol for anything (I didn't). Mine worked great still.

If you have two ISPs, do this all again for the 2nd ISP but change the "Outbound Interface" to correspond to the 2nd ISP.

Then, I did a HA Sync and the outbound NAT rules populated in the Standby side. Now, when I fail over to the standby firewall everything works like it should.

Good luck. I'm really happy with the way OPNSense and these scripts worked out.

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature:
function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature:
function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839:
} elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876:
} elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

So this is a year later, and I'm using 24.1.1. I have a static IP address and we figured we could make it work. We have Cox internet.

We set it up and it seemed to work once. Then it stopped working. Nothing we did seemed to work. I Rebooted both firewalls. I even powered down the Secondary and for the life of me I could not get back to the gateway. Checked and double checked the config and all was correct. On a hunch, I re-saved and applied gateway details and wan details. With no changes. Presto, back working. So it appears something on the back end got changed that is not visible to the GUI. Another things is when the fail over takes place, and the WAN is taken off line on the primary/slave, it disappears in the gui to not reappear again. So then the system just takes a random interface and uses that when brought back up. In my case the built in wifi interface.

Any help appreciated.

Edit: I just now happened to see the post directly above. That explains a lot, now to figure a way around that.

I'm invested in fixing this script for my own purposes so I'll get back to you here if I get it working.

That would be awesome oasis9. For what it is worth 24.1.2 is now out and I see some changes to interface changes. That MAY change how things work. I've spent so much time on this, that I'm not even willing to update for fear of it borking the system again. I have other fires to put out for now. ;)

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature: function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature: function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839: } elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876: } elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

interface_reset($interface)

This fixed it for me on 24.1.2, thank you :)

thank you all for your efforts for getting a solution at this usecase. Someone created a Featurerequst at opnsense which i also commented: opnsense/core#7333

I'll still try to get this solution to work properly (even with the comment from @oasis9 and @stevencoutts) -. but i don't get a new ipv6 after these commands are fired:

$config['interfaces'][$ifkey]['enable'] = '1';
interfaces_bring_up($ifkey);
interface_configure(false, $ifkey, true, true);
write_config("enable interface '$ifkey' due CARP event '$type'", false);
usleep(200 * 1000);

if i manually do an configctl interface reconfigure wan on the console - i instantly get an ipv6/route. Exactly the same if i press the "Reload" Button in the commands from the UI (interfaces/overview).

it seems, that interface_configure is not "enough" for getting a working connection with ipv4 AND ipv6 AND a default route

here is a screenshot only from executing by the script:

and here a Screenshot after pressing reload:

i've tried the scripts from @kronenpj @tlyakhov and also from another thread this one

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

I haven't tested my script for IPv6 specifically yet but I believe it's working (I can verify later). My first WAN interface is public IP cable and that WAN interface between instances shares the same MAC address. My second WAN interface is Starlink and is CG-NAT'd so that interface between instances has a different MAC address between interfaces. In both instances, I usually see an IPv6 WAN address per interface.

This would be nice!

I only have 1 WAN Interface with CGNAT - so ipv6 is very important ;-)