Disable WAN Interface on CARP Backup

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

$ifkey = 'wan';

if ($type === "MASTER") {
    log_error("enable interface '$ifkey' due CARP event '$type'");
    $config['interfaces'][$ifkey]['enable'] = '1';
    write_config("enable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
} else {
    log_error("disable interface '$ifkey' due CARP event '$type'");
    unset($config['interfaces'][$ifkey]['enable']);
    write_config("disable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
}

Hey @cybermaus ... on your second point.. where did you find advanced to add the MAC into the host? I had it working by adding the spoofed MAC into interfaces/WAN.

My provider needed the Client-ID the same too, and for some reason, OPNsense hides/merges that in the hostname.
You need advanced, otherwise OPNsense interprets the whole thing as a string rather then 6 bytes.

When you do not specify hostname, OPNsense sends the MAC as Client-ID, sp that is why I had to spoof the MAC into the Client-ID (and on OPNsense, thus in the hostname) to get my old IP back (that I had unmodified for 6 years now). Anything else, and I got a new IP

(see image, I hope it arrives, it does not show on preview)

Got it.. I found it right about when the email came in. Many thanks. I'm testing again with the scripts enabled. So far so good. Was stable for a couple months before I upgraded :)

thanks for the script. it works pretty well! I think CARP itself is picky with the maintenance mode stuff ... I've had to trigger it on the 'master node' by disabling CARP or rebooting that node for the 'backup' to kick-in.

one thing I haven't gotten to work is creating the 'lower priority gateway' for lan interface on the 'backup node. When I try to go to the 'firmware' section on the backup node, that's when I see that it is not loading (just keeps spinning). Firewall logs show all traffic is being allowed on both node's fw logs. I've also cleared the arp on the backup node for the vip associated with the LAN interface. Have even tried to use just the IP of the master node on LAN as the gateway and still does not work for internet access on the backup.

I am having the exact same behavior as @vc1cv1

PERFECT!!! (except when ONE interface decides to go MASTER)

Made some adjustments to this script for my needs and to update to 23.1:
https://gist.github.com/kronenpj/e90258f12f7a40c4f38a23b609b3288b

Had to make some edits to make this work: https://gist.github.com/tlyakhov/15172db645d01dc67a0b585096f28ab3

This script worked great for me. I did, however, have a weird issue that I worked around. Thought someone else might find the solution useful. I'm running OPNSense on Proxmox 8. Failover worked well with this script (did everything it was supposed to do), but traffic going through what was the standby firewall (that is now active after failover) wouldn't work. A network capture after a failover verified that outbound traffic from the firewall was not being NAT'd.

I noticed that on the normal Primary firewall, auto-generated firewall NAT rules were in place. However, on the standby firewall there were no auto-generated NAT rules in place.

I put static NAT rules in place on both firewalls, sync'd them and this works like a charm. Drop maybe 4 ICMP Echo replies on a failover. I drop nothing on fail-back.

Thanks for the great work, everyone.

@flyboy747 would you care you elaborate how you did this? I'm on more or less the same setup as you, and having the same struggles atm. Thanks in advance :)

@flyboy747 would you care you elaborate how you did this? I'm on more or less the same setup as you, and having the same struggles atm. Thanks in advance :)

Yeah - So what I did was on the primary firewall, I navigated to Firewall, NAT, Outbound, changed the "Mode" to "Manual Outbound NAT rule generation". Then I added a new outbound NAT rule. I selected the "Outbound" interface, in my case "Comcast". I selected IPV4 (I don't run V6), any protocol, source address "LAN Net", source port "any", destination address and port both set to "any" and translation target "Interface Address". Everything else besides the Description are blank. You can put whatever you want in the Description, of course.

Then, I tested on the primary firewall to make sure everything was working properly still. You may have to add the ISAKMP outbound if you use that protocol for anything (I didn't). Mine worked great still.

If you have two ISPs, do this all again for the 2nd ISP but change the "Outbound Interface" to correspond to the 2nd ISP.

Then, I did a HA Sync and the outbound NAT rules populated in the Standby side. Now, when I fail over to the standby firewall everything works like it should.

Good luck. I'm really happy with the way OPNSense and these scripts worked out.

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature:
function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature:
function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839:
} elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876:
} elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

So this is a year later, and I'm using 24.1.1. I have a static IP address and we figured we could make it work. We have Cox internet.

We set it up and it seemed to work once. Then it stopped working. Nothing we did seemed to work. I Rebooted both firewalls. I even powered down the Secondary and for the life of me I could not get back to the gateway. Checked and double checked the config and all was correct. On a hunch, I re-saved and applied gateway details and wan details. With no changes. Presto, back working. So it appears something on the back end got changed that is not visible to the GUI. Another things is when the fail over takes place, and the WAN is taken off line on the primary/slave, it disappears in the gui to not reappear again. So then the system just takes a random interface and uses that when brought back up. In my case the built in wifi interface.

Any help appreciated.

Edit: I just now happened to see the post directly above. That explains a lot, now to figure a way around that.

I'm invested in fixing this script for my own purposes so I'll get back to you here if I get it working.

That would be awesome oasis9. For what it is worth 24.1.2 is now out and I see some changes to interface changes. That MAY change how things work. I've spent so much time on this, that I'm not even willing to update for fear of it borking the system again. I have other fires to put out for now. ;)

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature: function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature: function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839: } elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876: } elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

interface_reset($interface)

This fixed it for me on 24.1.2, thank you :)

thank you all for your efforts for getting a solution at this usecase. Someone created a Featurerequst at opnsense which i also commented: opnsense/core#7333

I'll still try to get this solution to work properly (even with the comment from @oasis9 and @stevencoutts) -. but i don't get a new ipv6 after these commands are fired:

$config['interfaces'][$ifkey]['enable'] = '1';
interfaces_bring_up($ifkey);
interface_configure(false, $ifkey, true, true);
write_config("enable interface '$ifkey' due CARP event '$type'", false);
usleep(200 * 1000);

if i manually do an configctl interface reconfigure wan on the console - i instantly get an ipv6/route. Exactly the same if i press the "Reload" Button in the commands from the UI (interfaces/overview).

it seems, that interface_configure is not "enough" for getting a working connection with ipv4 AND ipv6 AND a default route

here is a screenshot only from executing by the script:

and here a Screenshot after pressing reload:

i've tried the scripts from @kronenpj @tlyakhov and also from another thread this one

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

I haven't tested my script for IPv6 specifically yet but I believe it's working (I can verify later). My first WAN interface is public IP cable and that WAN interface between instances shares the same MAC address. My second WAN interface is Starlink and is CG-NAT'd so that interface between instances has a different MAC address between interfaces. In both instances, I usually see an IPv6 WAN address per interface.

This would be nice!

I only have 1 WAN Interface with CGNAT - so ipv6 is very important ;-)

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

This stopped working for me but now seems
To be working again :)

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

@klingon888 what did you exactly change?

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

Yep, the script as is still works fine for me. I've made no changes since upgrading to recent versions of opn and I'm 24.x

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly.

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly.

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

I am using the gw groups too. I don't have this issue with my 2nd's gws getting deleted when it fails back / forth.

Does your general or audit log show what's going on with it?

Apologies if understanding wrong, but if you want your backup's default route through the primary (or vice versa), I've added a GW indeed but you'll have to look at the priority e.g. when other GWs are down your LAN should prevail, but not prevail when their up.

Embarrassed to say, it now works properly, e.g. the WAN-on-LAN gateway gives access to internet when backup, it becomes secondary weh the firewall is master, and when the firewall goes back to become a backup only the WAN gateway is delete, the WAN-on-LAN gateway doesn't get deleted. Not sure why it wasn't working in the last couple of months after I installed it. Maybe the new 24.1.7 makes a difference...