Skip to content

Instantly share code, notes, and snippets.

@spali
Last active November 7, 2024 16:57
Show Gist options
  • Save spali/2da4f23e488219504b2ada12ac59a7dc to your computer and use it in GitHub Desktop.
Save spali/2da4f23e488219504b2ada12ac59a7dc to your computer and use it in GitHub Desktop.
Disable WAN Interface on CARP Backup
#!/usr/local/bin/php
<?php
require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");
$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';
if ($type != 'MASTER' && $type != 'BACKUP') {
log_error("Carp '$type' event unknown from source '{$subsystem}'");
exit(1);
}
if (!strstr($subsystem, '@')) {
log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
exit(1);
}
$ifkey = 'wan';
if ($type === "MASTER") {
log_error("enable interface '$ifkey' due CARP event '$type'");
$config['interfaces'][$ifkey]['enable'] = '1';
write_config("enable interface '$ifkey' due CARP event '$type'", false);
interface_configure(false, $ifkey, false, false);
} else {
log_error("disable interface '$ifkey' due CARP event '$type'");
unset($config['interfaces'][$ifkey]['enable']);
write_config("disable interface '$ifkey' due CARP event '$type'", false);
interface_configure(false, $ifkey, false, false);
}
@stevencoutts
Copy link

Heya all, looks like 24.1 is removing interface_bring_down (not to be confused with interfaces_bring_up) in favor of interface_reset. This may break your scripts after upgrade, once it's released. https://github.com/opnsense/core/blob/stable/24.1/src/etc/inc/interfaces.inc

Old signature: function interface_bring_down($interface = 'wan', $ifacecfg = false)

New signature: function interface_reset($interface = 'wan', $ifacecfg = false, $suspend = false)

$suspend = true seems to prevent the removal of VIPs, v6 and v4 addresses, and prevents ondemand ppp from being killed. There are other nuances I can't seem to get my head around, mainly to do with line 839: } elseif (!is_ipaddrv6($ifcfg['ipaddrv6'])) {

and its counterpart 876: } elseif (!is_ipaddrv4($ifcfg['ipaddr'])) {

..these checks don't exist in 23.7, and I'm confused as to the usefulness of reacting only when an address is configured but invalid? Perhaps I'm misinterpreting.

So the old method call should be able to be replaced with interface_reset($interface, false, false) for it to function in OPNsense 24.1. 24.1 is still in active development though, so this is potentially subject to change.

Edit: To be honest I'm a little confused at the fact that some scripts call interface_bring_down($interface, true) as I'm seeing a condition at the start of that function as early as 21.7 that will return immediately if $ifacecfg is not strictly false and is also not an array. I think that function call was effectively blocked from execution? The same condition exists in interface_reset($interface, $ifacecfg = false, $suspend = false) so $ifacecfg should either be false or an array, if it is set. I'd think $suspend should be set to false, as it retains static addresses, routes and gateway information, which are no longer present. So my invocation would be interface_reset($interface) as the defaults are suitable.

Thanks all for your efforts, this has been a very useful tool for my circumstances.

interface_reset($interface)

This fixed it for me on 24.1.2, thank you :)

@skl283
Copy link

skl283 commented Apr 1, 2024

thank you all for your efforts for getting a solution at this usecase. Someone created a Featurerequst at opnsense which i also commented: opnsense/core#7333

I'll still try to get this solution to work properly (even with the comment from @oasis9 and @stevencoutts) -. but i don't get a new ipv6 after these commands are fired:

$config['interfaces'][$ifkey]['enable'] = '1';
interfaces_bring_up($ifkey);
interface_configure(false, $ifkey, true, true);
write_config("enable interface '$ifkey' due CARP event '$type'", false);
usleep(200 * 1000);

if i manually do an configctl interface reconfigure wan on the console - i instantly get an ipv6/route. Exactly the same if i press the "Reload" Button in the commands from the UI (interfaces/overview).

it seems, that interface_configure is not "enough" for getting a working connection with ipv4 AND ipv6 AND a default route

here is a screenshot only from executing by the script:
image

and here a Screenshot after pressing reload:
image

i've tried the scripts from @kronenpj @tlyakhov and also from another thread this one

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

@willjasen
Copy link

willjasen commented Apr 1, 2024

So, does someone have any hints for getting this done without an manual step or - in best case - only in php without console command?

I haven't tested my script for IPv6 specifically yet but I believe it's working (I can verify later). My first WAN interface is public IP cable and that WAN interface between instances shares the same MAC address. My second WAN interface is Starlink and is CG-NAT'd so that interface between instances has a different MAC address between interfaces. In both instances, I usually see an IPv6 WAN address per interface.

@skl283
Copy link

skl283 commented Apr 1, 2024

This would be nice!

I only have 1 WAN Interface with CGNAT - so ipv6 is very important ;-)

@klingon888
Copy link

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

@garryevanson99
Copy link

This stopped working for me but now seems
To be working again :)

@skl283
Copy link

skl283 commented Jun 1, 2024

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

@klingon888 what did you exactly change?

@vc1cv1
Copy link

vc1cv1 commented Jun 1, 2024

Thanks @spali. Now in 2024, your script still works with single WAN IP on 24.1.7.

I got side tracked reading thru all the other scripts and posts abt changes in 24.1 breaking scripts etc and wasted a day going around in circles and finally decided to use yr script exactly as is and it works great.

I added a few more lines to stop most of the services when its on backup firewall. I also had problems in the beginning with ipv6 not working. I noticed that you need to let both machines run for a day or so to let it properly sync. Then bring the firewalls up/down to test and now even ipv6 works. So for anyone who's found this script, it still works.

Yep, the script as is still works fine for me. I've made no changes since upgrading to recent versions of opn and I'm 24.x

@vecchiae
Copy link

vecchiae commented Jun 6, 2024

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly.
Screenshot 2024-06-06 at 6 46 01 PM

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

@vc1cv1
Copy link

vc1cv1 commented Jun 6, 2024

I am on OPNsense 24.1.8, the Spali script works properly for my single WAN connections. I also want to have the backup firewall to access internet, and thus I created a gateway in my backup firewall as per point 4 (optional) from Spali instructions. It works properly. Screenshot 2024-06-06 at 6 46 01 PM

However, when the backup becomes master and then reverts to backup, both the WAN and WAN-on-LAN gateways are deleted. I suppose that only the WAN interface should be deleted. Not sure if the script is not meant to retain the backup WAN-on-LAN gateway, or if I am doing something wrong.

I am using the gw groups too. I don't have this issue with my 2nd's gws getting deleted when it fails back / forth.

Does your general or audit log show what's going on with it?

@CoMPaTech
Copy link

Apologies if understanding wrong, but if you want your backup's default route through the primary (or vice versa), I've added a GW indeed but you'll have to look at the priority e.g. when other GWs are down your LAN should prevail, but not prevail when their up.

@vecchiae
Copy link

vecchiae commented Jun 7, 2024

Embarrassed to say, it now works properly, e.g. the WAN-on-LAN gateway gives access to internet when backup, it becomes secondary weh the firewall is master, and when the firewall goes back to become a backup only the WAN gateway is delete, the WAN-on-LAN gateway doesn't get deleted. Not sure why it wasn't working in the last couple of months after I installed it. Maybe the new 24.1.7 makes a difference...

@woodshoes
Copy link

Well 24.7.2 has removed the "interfaces_bring_up" function - see release notes here
image
Any ideas by chance? PHP is not my strong suit & my failovers don't bring up the interface any more.

@woodshoes
Copy link

Found another gist here that works w the new function "legacy_interface_flags"

@garryevanson99
Copy link

Found another gist here that works w the new function "legacy_interface_flags"

Hi - this is still working for me fyi on version 24.7.2 - if I failover my carp, my wan is disabled on primary

if I fail back wan is disabled on secondary

@edward-scroop
Copy link

edward-scroop commented Sep 11, 2024

The script should also ignore the third state of INIT as i keep seeing it cause a failover , despite it being harmless.

line 11 can be changed to
if ($type != 'MASTER' && $type != 'BACKUP' && $type != 'INIT') {

and line 28 can be changed to
} else if ($type === "BACKUP") {
or ignored.

@raegedoc
Copy link

raegedoc commented Sep 13, 2024

Thank you @spali for this script, simple and efficient without frills.

As of totay with OPNsense 24.7.4, the script works perfectly except for the fact that once the backup is promoted master and then demoted to backup again, the default route (System -> Routes -> Status) is not set back to the LAN VIP as it was set initially as stated by @skl283.

That problem limits the backup's ability to have internet access while being backup.

To fix that situation, change/add these:

  1. Following $ifkey = 'wan' add $lan_vip = 'YOUR_LAN_VIP' and set to your correct LAN_VIP / LAN CARP VIP
  2. Following interface_configure in the BACKUP section add both...
    exec('/sbin/route del default >&1', $ifc, $ret);
    exec('/sbin/route add default ' . $lan_vip . ' >&1', $ifc, $ret);
  3. At the end of the script, add the missing "?>"
  4. Add the suggestions provided by @edward-scroop for the previous post to mine. (!= INIT and else BACKUP)

NOTE : The 4th step in spali's instructions is not optional anymore. A WAN-to-LAN Gateway is required.

This is it :)

@skl283
Copy link

skl283 commented Sep 15, 2024

Hi @raegedoc are you sure that you you use this gist?
There ist only an else case line 28 to 33 - which should used, if the system is in the Backup case... or are you using this gist? There is explicit an Backup Section.

Perhaps you could post or do a fork of this Script?

@willjasen
Copy link

willjasen commented Sep 15, 2024

I’m throwing this here with little knowledge otherwise with my abandoned script, but a challenge I had to overcome dealt with multiple interfaces being decided as “failed” such that the backup connection would take over. May not be relevant now with the recent updates but throwing it out there - https://gist.github.com/willjasen/6ae0f47bca36ced2bd52b2fefc2bc21e

@raegedoc
Copy link

raegedoc commented Sep 15, 2024

Hi @raegedoc are you sure that you you use this gist? There ist only an else case line 28 to 33 - which should used, if the system is in the Backup case... or are you using this gist? There is explicit an Backup Section.

Perhaps you could post or do a fork of this Script?

Hi @skl283, I tried them all from 2 weeks ago and none was giving me back internet access on my backup node after being promoted primary and demoted back to backup again. Only these small adds would fix it all while keeping the script very light and clean.

I forgot to mention I incorporated the suggestions @edward-scroop did in the post previous to mine : https://gist.github.com/spali/2da4f23e488219504b2ada12ac59a7dc?permalink_comment_id=5185710#gistcomment-5185710

Here is a link to my gist : https://gist.github.com/raegedoc/093ba815b6b3f2bc2ff327f48c60f3a9

Open to your ideas :)

@edward-scroop
Copy link

@raegedoc do you have the gateway monitoring setup for the WAN gateway? Because I have it set up and when it switches back to master, it sets the priority of the backup WAN gateway to defunct which removes it from the route selection.

@raegedoc
Copy link

raegedoc commented Sep 15, 2024

@edward-scroop, Yes I have gateway monitoring set for my WAN gateway of both primary and backup. The problem is not with my primary node switching back to master but my backup node switching back to being a backup. This way, backup has internet access for receiving it OPNsense updates and news Annoncements

For clarity, here is my primary configuration for the WAN link when primary is primary and backup is backup :

image

...and for my backup configuration. Blue arrow point to the fields where MY_CARP_LAN_VIP is specifed.

image
image

@edward-scroop
Copy link

edward-scroop commented Sep 15, 2024

From your screenshots, the monitor ip is empty and the disable gateway monitoring is checked. That would mean gateway monitoring is disabled.

I think what is happening is as your WAN gateway has a higher priority than the LAN gateway and with no gateway monitoring, the backup has no way to tell the WAN gateway is down and it then doesn't have a reason to swap to the LAN gateway.

To fix it either set the LAN gateway to a priority higher than the WAN gateway, or set a monitor ip of 1.1.1.1 and uncheck the disable gateway monitoring box.

@raegedoc
Copy link

raegedoc commented Sep 15, 2024

Hi, WAN Gateway has priority 254 and WAN-to-LAN has 255 (so WAN > WAN-to-LAN).

Anyway, I tried your trick and worse, my backup has no internet access when backup. Default route has shown still point default gateway to the WAN IP that connects to nothing when backup.

image

Interfaces: Diagnostics: Ping to 1.1.1.1 has 100% loss :(

Since fixing the default gateway (with route delete followed by add CARP_LAN_IP) while being backup of a functional primary node, it might have been the missing trick with my setup that is pretty standard when theISP provided only a public DHCP WAN IP.

I'll keep the setup I shared earlier. Thank's for sharing edward-scroop.

@edward-scroop
Copy link

edward-scroop commented Sep 16, 2024

The LAN gateway needs a priority higher than 254. The smaller the value, the higher the priority.

@raegedoc
Copy link

The LAN gateway needs a priority higher than 254. The smaller the value, the higher the priority.

It's the case, LAN has priority 255

@edward-scroop
Copy link

I meant, the LAN needs a priority of 1-253.

@bitcoredotorg
Copy link

I upgraded to 24.7.6 today, and our syshook.d scripts that call interface_configure() appears to now crash when an undefined function eventually is called (see my stack trace below). See my post on opnsense forums: https://forum.opnsense.org/index.php?topic=20972.msg216770#msg216770 for the customizations I run, but I'd imagine Spali's version is equally as affected. I submitted a crash report, but did not create an issue on the opnsense github.

I believe we need to be using a more well supported method to enable/disable interfaces in these syshook scripts. The 'interface' PHP functions seem to be in heavy development in 24.7, and many functions seem to be considered 'legacy' methods or becoming deprecated. Or, perhaps this is just a bug.

As a workaround, if you don't want to roll-back, you can comment the $config line, write_config, and interface_configure calls and instead use shell_exec("/sbin/ifconfig {$interface['if']} up"); and shell_exec("/sbin/ifconfig {$interface['if']} down"); instead, but this is less reliable and has other undesirable effects. For example, when only using interface up/down commands, the backup device needs it's WAN interface left as enabled - under that condition, in the event of a reboot, you'll want to manually trigger a failover cycle to have the backup device's WAN interface in "down" state, else you'll have both interfaces up and enabled. Again, we need to find the most well supported way to enable/disable interfaces, and go from there.

[22-Oct-2024 13:17:14 America/New_York] PHP Fatal error: Uncaught Error: Call to undefined function system_routing_configure() in /usr/local/etc/inc/interfaces.inc:3777
Stack trace:
#0 /usr/local/etc/inc/interfaces.inc(2498): interfaces_restart_by_device(false, Array, false)
#1 /usr/local/etc/rc.syshook.d/carp/10-wancarp(24): interface_configure(false, 'opt3', false, false)
#2 {main}
thrown in /usr/local/etc/inc/interfaces.inc on line 3777

As a side note, others are having trouble with carp maintenance mode not working at all (not triggering a failover, as one would expect): opnsense/core#7877

@toddgonzo74
Copy link

Anyone find a fix for this issue yet?

@skl283
Copy link

skl283 commented Nov 7, 2024

i haven't tried it yet, but does this issue also occur at 24.7.8? @bitcoredotorg perhaps you tried the update?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment