OpnSense 23.1 - Disable WAN + OPT2 Interfaces during CARP Failover

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("system.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

// Add more interfaces that need to be disabled/enabled after a CARP event.
//$iface_aliases = array('wan', 'opt2');
//$iface_names = array('wan' => 'igc0', 'opt2' => 'gif0');
$iface_aliases = array('wan', 'wan');
$iface_names = array('wan' => 'igc0');
$dhcp_ifaces = array('lan', 'opt3', 'opt1');

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

if ($type === "MASTER") {
    if ($config['interfaces']['wan']['enable'] == 0) {
      foreach ($iface_aliases as $ifkey) {
          // $iface_name = $iface_names[$ifkey];
          log_error("enable interface '$ifkey' due CARP event '$type'");
          $config['interfaces'][$ifkey]['enable'] = '1';
          interfaces_bring_up($ifkey);
          interface_configure(false, $ifkey, true, true);
          write_config("enable interface '$ifkey' due CARP event '$type'", false);
          // usleep(200 * 1000);
          //foreach ($dhcp_ifaces as $dhkey) {
          //  $config['dhcpd'][$dhkey]['enable'] = true;
          //}
      }
    } else {
      log_msg("Carp '$type' duplicate event triggered.");
    }
} else {
    if ($config['interfaces']['wan']['enable'] == 1) {
      foreach ($iface_aliases as $ifkey) {
          // $iface_name = $iface_names[$ifkey];
          log_error("disable interface '$ifkey' due CARP event '$type'");
          //foreach ($dhcp_ifaces as $dhkey) {
          //  $config['dhcpd'][$dhkey]['enable'] = false;
          //}
          interface_reset($ifkey);
          unset($config['interfaces'][$ifkey]['enable']);
          interface_configure(false, $ifkey, true, false);
          exec('/sbin/ifconfig ' . escapeshellarg($ifkey) . 'down 2>&1', $ifc, $ret);
          write_config("disable interface '$ifkey' due CARP event '$type'", false);
      }
    } else {
      log_msg("Carp '$type' duplicate event triggered.");
    }
}
?>

I've made another update to the script in an attempt to reduce the switch-over / recovery time. On my setup the unbound daemon took upwards of 30 minutes to begin serving DNS. The current version of this script reduces that to under 5 minutes. The "cause" was the daemon being restarted each time a WAN interface was changed, which was approximately 14 times in my case.

This change reduces that to two, partially by removing opt2 from the list of interfaces to bring down and partially to only run the configuration change twice.

Unfortunately this has identified two problems, which may be bugs:

1. I need to bring the WAN interface up twice either because it doesn't properly request a DHCP address or it takes "too long" to get one from my ISP.
2. ~~Failing over from primary->secondary takes longer than secondary->primary. I don't know the cause of the asymmetry, but I'm going to bring it up in the forums and possibly file a bug. Presumably the fail-back does the same work but it takes ~1 second to transition back to the primary firewall.~~ Update - This was possibly due to a 'bad' secondary VM image. I've cloned the primary to the secondary and it appears that the failover is symmetric now.