Disable WAN Interface on CARP Backup

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

$ifkey = 'wan';

if ($type === "MASTER") {
    log_error("enable interface '$ifkey' due CARP event '$type'");
    $config['interfaces'][$ifkey]['enable'] = '1';
    write_config("enable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
} else {
    log_error("disable interface '$ifkey' due CARP event '$type'");
    unset($config['interfaces'][$ifkey]['enable']);
    write_config("disable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
}

@flyboy747 would you care you elaborate how you did this? I'm on more or less the same setup as you, and having the same struggles atm. Thanks in advance :)

Yeah - So what I did was on the primary firewall, I navigated to Firewall, NAT, Outbound, changed the "Mode" to "Manual Outbound NAT rule generation". Then I added a new outbound NAT rule. I selected the "Outbound" interface, in my case "Comcast". I selected IPV4 (I don't run V6), any protocol, source address "LAN Net", source port "any", destination address and port both set to "any" and translation target "Interface Address". Everything else besides the Description are blank. You can put whatever you want in the Description, of course.

Then, I tested on the primary firewall to make sure everything was working properly still. You may have to add the ISAKMP outbound if you use that protocol for anything (I didn't). Mine worked great still.

If you have two ISPs, do this all again for the 2nd ISP but change the "Outbound Interface" to correspond to the 2nd ISP.

Then, I did a HA Sync and the outbound NAT rules populated in the Standby side. Now, when I fail over to the standby firewall everything works like it should.

Good luck. I'm really happy with the way OPNSense and these scripts worked out.

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature:
function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature:
function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839:
} elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876:
} elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

So this is a year later, and I'm using 24.1.1. I have a static IP address and we figured we could make it work. We have Cox internet.

We set it up and it seemed to work once. Then it stopped working. Nothing we did seemed to work. I Rebooted both firewalls. I even powered down the Secondary and for the life of me I could not get back to the gateway. Checked and double checked the config and all was correct. On a hunch, I re-saved and applied gateway details and wan details. With no changes. Presto, back working. So it appears something on the back end got changed that is not visible to the GUI. Another things is when the fail over takes place, and the WAN is taken off line on the primary/slave, it disappears in the gui to not reappear again. So then the system just takes a random interface and uses that when brought back up. In my case the built in wifi interface.

Any help appreciated.

Edit: I just now happened to see the post directly above. That explains a lot, now to figure a way around that.

I'm invested in fixing this script for my own purposes so I'll get back to you here if I get it working.

That would be awesome oasis9. For what it is worth 24.1.2 is now out and I see some changes to interface changes. That MAY change how things work. I've spent so much time on this, that I'm not even willing to update for fear of it borking the system again. I have other fires to put out for now. ;)

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature: function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature: function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839: } elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876: } elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

interface_reset($interface)

This fixed it for me on 24.1.2, thank you :)

thank you all for your efforts for getting a solution at this usecase. Someone created a Featurerequst at opnsense which i also commented: opnsense/core#7333

I'll still try to get this solution to work properly (even with the comment from @oasis9 and @stevencoutts) -. but i don't get a new ipv6 after these commands are fired:

$config['interfaces'][$ifkey]['enable'] = '1';
interfaces_bring_up($ifkey);
interface_configure(false, $ifkey, true, true);
write_config("enable interface '$ifkey' due CARP event '$type'", false);
usleep(200 * 1000);

if i manually do an configctl interface reconfigure wan on the console - i instantly get an ipv6/route. Exactly the same if i press the "Reload" Button in the commands from the UI (interfaces/overview).

it seems, that interface_configure is not "enough" for getting a working connection with ipv4 AND ipv6 AND a default route

here is a screenshot only from executing by the script:

and here a Screenshot after pressing reload:

i've tried the scripts from @kronenpj @tlyakhov and also from another thread this one

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

I haven't tested my script for IPv6 specifically yet but I believe it's working (I can verify later). My first WAN interface is public IP cable and that WAN interface between instances shares the same MAC address. My second WAN interface is Starlink and is CG-NAT'd so that interface between instances has a different MAC address between interfaces. In both instances, I usually see an IPv6 WAN address per interface.

This would be nice!

I only have 1 WAN Interface with CGNAT - so ipv6 is very important ;-)

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

This stopped working for me but now seems
To be working again :)

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

@klingon888 what did you exactly change?

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

Yep, the script as is still works fine for me. I've made no changes since upgrading to recent versions of opn and I'm 24.x

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly.

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly.

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

I am using the gw groups too. I don't have this issue with my 2nd's gws getting deleted when it fails back / forth.

Does your general or audit log show what's going on with it?

Apologies if understanding wrong, but if you want your backup's default route through the primary (or vice versa), I've added a GW indeed but you'll have to look at the priority e.g. when other GWs are down your LAN should prevail, but not prevail when their up.

Embarrassed to say, it now works properly, e.g. the WAN-on-LAN gateway gives access to internet when backup, it becomes secondary weh the firewall is master, and when the firewall goes back to become a backup only the WAN gateway is delete, the WAN-on-LAN gateway doesn't get deleted. Not sure why it wasn't working in the last couple of months after I installed it. Maybe the new 24.1.7 makes a difference...