Connecting back to IPVS VIPs from backend/real servers

README.md

Scenario: IPVS is used to route traffic to various ports on some VIP to different sets of backend servers. Those backend servers are configured with local routes for that VIP in order to accept incoming packets as local INPUT.

$ ip route add local 194.197.235.245 dev lo

Problem: The backend servers cannot themselves connect to the other services on the same IPVS VIP, because the VIP is locally routed.

$ ip ro get 194.197.235.245
local 194.197.235.245 dev lo table local src 194.197.235.245 
    cache <local> 

Updated solution

Leave the local VIP route in the local route table, but bypass it for output route lookups for the VIP:

# move the local table lookup later in the rules
$ ip rule add pref 100 lookup local
$ ip rule del pref 0 lookup local

# add normal local routes for the VIP
$ ip -4 route add local 194.197.235.245 dev eth-test
$ ip -6 route add local 2001:2060:41:f0::f5 dev eth-test

# bypass the local route lookup for output route lookups
$ ip -4 rule add to 194.197.235.245 iif lo pref 10 table main
$ ip -6 rule add to 2001:2060:41:f0::f5 iif lo pref 10 table main

Original solution

Unidirection local routes no longer work in newer Linux versions:

$ ip rule add to 194.197.235.245 iif eth-test table 100
$ ip route add local 194.197.235.245 dev eth-test table 100

The outgoing/incoming packets are routed correctly, but the return packets are dropped:

$ ip route get 194.197.235.245
194.197.235.245 via 10.100.0.1 dev eth-test src 10.100.100.38 
    cache 
$ sudo ip route get 194.197.235.245 from 192.0.2.1 iif eth-test
local 194.197.235.245 from 192.0.2.1 dev lo table 100 
    cache <local>  iif eth-test
$ sudo ip route get 192.0.2.1 from 194.197.235.245 # OUTPUT path when iif is unspecified
RTNETLINK answers: Invalid argument

This is because ip_route_output now does a lookup on the src IP, and expects it to have a local route in the local routing table:

• https://elixir.bootlin.com/linux/v4.9.184/source/net/ipv4/route.c#L2669
• https://elixir.bootlin.com/linux/v4.9.184/source/net/ipv4/route.c#L2256

This shows up in netstat -s | grep 'dropped because of missing route', and the ktrace stopping on ip_route_output_flow => __ip_dev_find => fib_table_lookup.

ktrace

 ------------------------------------------
 0)    ip-4599     =>    ip-4607    
 ------------------------------------------

 0)               |  inet_rtm_getroute() {
 0)               |    __alloc_skb() {
 0)               |      kmem_cache_alloc_node() {
 0)   0.156 us    |        _cond_resched();
 0)   0.058 us    |        _cond_resched();
 0)   0.056 us    |        memcg_kmem_put_cache();
 0)   1.908 us    |      }
 0)               |      __kmalloc_reserve.isra.35() {
 0)               |        __kmalloc_node_track_caller() {
 0)   0.056 us    |          kmalloc_slab();
 0)               |          kmem_cache_alloc_node_trace() {
 0)   0.056 us    |            _cond_resched();
 0)   0.058 us    |            _cond_resched();
 0)   0.058 us    |            memcg_kmem_put_cache();
 0)   1.500 us    |          }
 0)   2.415 us    |        }
 0)   2.975 us    |      }
 0)   0.154 us    |      ksize();
 0)   6.568 us    |    }
 0)               |    ip_route_output_flow() {
 0)               |      __ip_route_output_key_hash() {
 0)               |        __ip_dev_find() {
 0)   0.456 us    |          fib_get_table();
 0)   0.802 us    |          fib_table_lookup();
 0) + 10.361 us   |        }
 0) + 15.212 us   |      }
 0) + 19.421 us   |    }
 0)               |    kfree_skb() {
 0)               |      skb_release_all() {
 0)   0.101 us    |        skb_release_head_state();
 0)               |        skb_release_data() {
 0)               |          skb_free_head() {
 0)               |            kfree() {
 0)   0.088 us    |              ___cache_free();
 0)   0.651 us    |            }
 0)   1.132 us    |          }
 0)   1.603 us    |        }
 0)   2.615 us    |      }
 0)               |      kfree_skbmem() {
 0)               |        kmem_cache_free() {
 0)   0.070 us    |          ___cache_free();
 0)   0.690 us    |        }
 0)   1.140 us    |      }
 0)   4.637 us    |    }
 0) + 43.348 us   |  }