Skip to content

Instantly share code, notes, and snippets.

@mzpqnxow
Last active December 23, 2022 16:39
Show Gist options
  • Save mzpqnxow/00b7b4b8cfea6d67d4d23b2a08c59b39 to your computer and use it in GitHub Desktop.
Save mzpqnxow/00b7b4b8cfea6d67d4d23b2a08c59b39 to your computer and use it in GitHub Desktop.
Multiple default routes in Azure - solving with policy-based routing

Azure and Public IP Addresses

In Microsoft Azure, even when a VM has a public IP address internface, the system sees it as a private network address. It also uses a private IP address as the gateway. The only way to "fix" this without making routing table changes is to use setsockopt() with SO_BINDTODEVICE, which is a privileged operation. In addition to that (and this can't be worked around in any generic way) you will need to do the standard socket bind() with the private IP address corresponding to the interface you want to use.

Solving With Policy Based Routing

The new generation of network configuration tools (I guess they're not that new, but they replaced route and ifconfig) makes policy-based routing pretty easy, especially if you don't require anything too special

Set Up Name Mappings for Routing Tables (Optional)

The userspace iproute2 tools (specifically, ip) references this file to resolve routing table names to numbers. This has no functional impact on routing, it just makes commands more readable

$ cat /etc/iproute2/rt_tables     
#
# reserved values
#
255	local
254	main
253	default
0	unspec
# Custom routing tables
1 wan1
2 wan2
3 wan3
4 wan4

Create Dedicated Routing Tables

To create 4 new (non-default/non-main) routing tables with specific devices as their default route, use the following:

# ip route add default via 10.0.14.1 dev eth1 table wan1
# ip route add default via 10.0.14.1 dev eth2 table wan2
# ip route add default via 10.0.14.1 dev eth3 table wan3
# ip route add default via 10.0.14.1 dev eth4 table wan4

Also, provide a default route on the main routing table. You will have very, very odd symptoms (unanswered ARP from the source interface) if you don't add these, though I'm not entirely sure why that is ...

# ip route add default via 10.0.14.1 metric 101 dev eth1
# ip route add default via 10.0.14.1 metric 102 dev eth2
# ip route add default via 10.0.14.1 metric 103 dev eth3
# ip route add default via 10.0.14.1 metric 104 dev eth4

Note this is typical of Azure with multiple interfaces. Unless you go out of your way to configure each NIC to use a different Virtual Network entirely, each will have a different public IP addresses on different public networks but they will have internal addresses on the same subnet, using the same gateway

Assign Traffic to Routing Tables Based on Source IP and Destination IP

Add this point, the indepdendant routing tables are created. However, no traffic will be assigned to respect/consult those tables until rules are applied. Rules are generally based on destination or source IP address. In this case, we need to use the private IP address of the respective device to assign traffic over to the corresponding table. The terms used here ("egress" and "ingress") aren't perfectly accurate, but they'll do

Add The Egress Rules

# ip rule add from 10.0.14.7 table wan1
# ip rule add from 10.0.14.8 table wan2
# ip rule add from 10.0.14.9 table wan3
# ip rule add from 10.0.14.10 table wan4

Add The Ingress Rules

# ip rule add to 10.0.14.7 table wan1
# ip rule add to 10.0.14.8 table wan2
# ip rule add to 10.0.14.9 table wan3
# ip rule add to 10.0.14.10 table wan4

Done

You are pretty much set now. However, whatever application(s) you use will need to support binding to a specific interface. This is supported by nearly all network tools and especially server software such as proxies, web servers, OpenSSH, etc. usually via configuration file or command line switch. In the event that the application you're using doesn't support binding to a specific interface by IP, you can add a simple bind() call on the socket immediately after it is created. You will want to bind to the private IP address of the device you want to use, and you'll want to specify port 0. Port 0 instructs the kernel TCP/IP stack to choose an available ephemeral port for the socket.

Addendum

Some possibly related things you may run into

ARP-Related sysctl Settings

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.default.rp_filter
sysctl net.ipv4.conf.default.arp_filter
sysctl net.ipv4.conf.all.arp_filter

IP Forwarding sysctl Settings

I don't think you'll need to deal with enabling packet forwarding, but just for reference, forwarding is typically disabled by default. You can enabled it several different ways with sysctl:

sysctl net.ipv4.conf.all.forwarding
sysctl net.ipv4.conf.default.forwarding

Make sure you know what you're doing if you're going to start messing around with packet forwarding ;)

More Addendum: Other Ways Of Doing This

You can do a lot of powerful things with iptables, obviously. From some random blog, here's a quite involved way to do this. It's not friendly or specific to a configuration as above, consider it a reference if you want to look into other solutions

 echo 1 > /proc/sys/net/ipv4/ip_forward  
 echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter  
create a veth pair, assign to it a private addresses, disable rp_filter on one end:
 ip link add serverintf type veth peer name distributeintf  
 ip ad ad 10.10.10.1/30 dev serverintf  
 ip ad ad 10.10.10.2/30 dev distributeintf  
 ip link set serverintf up
 ip link set distributeintf up
 echo 0 > /proc/sys/net/ipv4/conf/serverintf/rp_filter  
 echo 0 > /proc/sys/net/ipv4/conf/distributeintf/rp_filter  
 ip rule add from 10.10.10.1 lookup 1010101  
 ip route add default dev serverintf via 10.10.10.2 table 1010101
use DNAT to route connections to the server:
 iptables -t nat -A PREROUTING -p udp -m multiport --dports $serverports -j DNAT --to-destination 10.10.10.1
 iptables -t raw -A PREROUTING -i distributeintf -j CT --notrack
 # for each interface eth$i with ip $ip and gateway $gateway
 echo 2 > /proc/sys/net/ipv4/conf/eth$i/rp_filter
 ip rule add from $ip lookup $((i + 1))
 ip route add default dev eth$i via $gateway table $((i + 1))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment