Sharing IPv4 exits for clients in layer 3 mesh networks using 464xlat

gistfile1.txt

This gist describes the setup in freifunk-gluon (OpenWrt based)

Sharing IPv4 internet access over IPv6 in babel mesh networks

gluon-464xlat

    Using the kernel module https://github.com/ayourtch/nat46/tree/master/nat46/modules, we have built two simple packages for providing IPv4 internet access to clients over IPv6-only networks.

    The setup is not conform with RFC 6877 in every point, but it works quite good for our purpose.

    Nodes providing IPv4 internet access offer a stateful NAT64.

    Other nodes can use these NAT64s (called PLATs in the context of 464xlat) to offer IPv4 internet access to their clients using a technique called SIIT.

    SIIT means "Stateless IP/ICMP Translation"  also called Stateless NAT46 or in the context of 464xlat and in the following called "CLAT" (Customer side translator).

    This procedure is nearly transparent and thus offers a fully functional IPv4 exit to the client.


    The package gluon-464xlat-clat provides a CLAT. (https://tools.ietf.org/html/rfc6145)

    A CLAT translates IPv4 packets to IPv6 packets prefixing the source address with a /96 prefix called CLAT prefix unique to the node serving the client and the destination address with a /96 prefix called PLAT prefix unique to that IPv4 exit.

      CLAT prefix: 2001:db8:1234::/96 PLAT prefix: 64:ff9b::/96

    CLIENT -> IPv4 (Source=192.168.0.1, Destination=1.2.3.4) -> CLAT -> IPv6 (Source=2001:db8:1234::192.168.0.1, Destination=64:ff9b::1.2.3.4) -> PLAT

    PLAT -> IPv6 (Source=64:ff9b::1.2.3.4, Destination=2001:db8:1234::192.168.0.1) -> CLAT -> IPv4 (Source=1.2.3.4, Destination=192.168.0.1) -> CLIENT


    The package gluon-464xlat-plat provides a NAT64 configuration (we call its interface PLAT). (https://tools.ietf.org/html/rfc6146)

    A PLAT SNATs IPv6 packets to a pool of IPv4 addresses

      PLAT prefix: 64:ff9b::/96, IPv4 pool: 10.23.0.9/32 (port-range: 9000-10000)

    CLAT -> IPv6 (Source=2001:db8:1234::192.168.0.23, Destination=64:ff9b::1.2.3.4) -> NAT64 -> IPv4 (Source=10.23.0.9:9876, Destination=1.2.3.4) -> INTERNET

    INTERNET -> IPv4 (Source=1.2.3.4, Destination=10.23.0.9:9876) -> NAT64 -> IPv6 (Source=64:ff9b::1.2.3.4, Destination=2001:db8:1234::192.168.0.23) -> CLAT

    Technical notes:

    Gluon-xlat-plat uses a little hack to provide a NAT64. 

    Actually it consists of a NAT66 which SNATs incoming traffic to a single IPv6 address

    and a NAT46 device which outputs, when fed by the NAT66, IPv4 traffic originating from a single IPv4 address.

    This IPv4 traffic gets routed to the masquerade in the WAN zone where it reaches the internet.

    Replies from the internet enter the NAT46 device and get translated to IPv6, are translated back to their CLAT addresses by NAT66 and finally reach the CLAT -> Client.


    On devices with enough memory (>=64MB) and enough flash (e.g. supernodes) we plan to use Jool (https://www.jool.mx/)  in the future (If benchmarks are better).

    Jool is able to directly SNAT the incoming IPv6 traffic to the WAN IPv4 address, but it is unable to avoid collisions with the kernel IPv4 masquerading running on the WAN device unless it is configured with a dedicated SNAT port range.



Babel-specific general 464xlat setup

    Glossary:

    Node: router participating in a gluon babel mesh network

    Clatnode: a node which uses a platnode to provide IPv4 exit to its clients

    Platnode: node which offers IPv4 exit to other nodes

      Note: Nodes can be clatnode and platnode at the same time (e.g. using its own IPv4 exit for clients)


    We introduced two new site-config-domain-parameters clat_range and plat_range which hold /48-prefixes.

    Nodes calculate their own PLAT or CLAT prefix by appending their MAC address to the /48 prefixes to obtain dedicated /96 prefixes.

    e.g. clat_range=fddd:1337:1337::/48, MAC=12:34:56:78:9a:bc -> clat_prefix=fddd:1337:1337:1234:5678:9abc::/96

    Note: 64:ff9b:1::/48 can be used for this purpose as defined by RFC8215


    /96-routes in clat_range and plat_range need to be redistributed by babel:

      (babelconfig)

    redistribute ip clat_range::/48 eq 96  allow

    redistribute ip plat_range::/48 eq 96  allow


    Nodes can be distinguished between clatnodes which have a CLAT device and platnodes which have a PLAT device.

    Platnodes can also be clatnodes at the same time having also a CLAT device which traffic is directly routed to its own PLAT device.

    This is required for allowing its clients to roam.


    General clatnode configuration:

    To receive replies the clatnodes CLAT prefix needs to be redistributed by babel.

    A clatnode needs to select a PLAT to use as an IPv4 exit for its clients

    Our naive approach is to use the metrics provided by babel:

    Initially prefixd checks for /96 routes in the plat_range and selects the one with the lowest metric.

    Prefixd dynamically reconfigures the CLAT device if a specific threshold is exceeded regarding the metric differences (PLAT with much better metric found) or if connectivity issues are detected (route lost, ping fail etc.).

    If prefixd selects a new PLAT it needs to inform l3roamd - l3roamd has socket hooks for that:

    set_platprefix Informs l3roamd that a PLAT with a better metric has been selected 

    this PLAT will be used for new leases of clients - old clients still use their old PLAT

    therefore l3roamd needs to handle all clients which are using the old PLAT as if they roamed to it (method described below)

    del_platprefix Informs l3roamd that a specific PLAT is not reachable anymore

    needs to delete all ip6tables/ipset rules using this PLAT


    General platnode configuration:

    A platnode provides a PLAT to other nodes and additionally itself for usage as an IPv4 exit

    Therefore it possesses a dedicated /96 PLAT prefix in the plat_range and distributes a route to this prefix via babel.

    Every platnode is reachable via address plat_prefix:192.0.0.1/29 to receive management packets for roaming purposes.



l3roamd 464xlat specific roaming behaviour

    Our goal was to allow seamless IPv4 roaming without TCP-connections breaking. The following concept achieves this theoretically.

    Nonetheless we had to trade of between simplicity and perfectionism for an initial concept. 

    Thus TCP connections break in this concrete example if the client roamed to another node and its lease runs out. 

    You can simply counter this effect by setting a very high leasetime. We don't recommend this for mesh network setups like Freifunk. 

    Clients will more likely be annoyed by a broken connection as their IPv4 exit is 5 wireless hops away than they will be annoyed that one TCP connection breaks (e.g. website stuck loading - build better browsers!).

    Let's get to the point.

    In this example we consider a client that roams from node A to node B using node C as its PLAT exit that initially associated on node D.

    depends on: +kmod-ipt-nat-extra


    What happens on node A?

    l3roamd receives CLAIM from node B

    Replies with INFO packet (see above for its content)

    Releases route to ULA-IPv6-Unicast address generated from the clients MAC address

    Deletes any ip6tables/ipset rules for that client if existing


    What happens on Node B?

    The new client associates - Node B only knows the clients MAC address at this point.

    l3roamd on node B sends a CLAIM to the ULA-IPv6-Unicast address generated from the clients MAC address

    If another node (A) served this client before, it is listening on that address and replies with an INFO packet which holds along with other information:

    CLATD - node D CLAT prefix the client used when it received its lease

    CLIENTIP - IPv4 address the Client used

    LEASETIME - remaining IPv4 leasetime in seconds

    PLATC - PLAT prefix the client used when it received its lease

    When node B receives this INFO, it sends a ROAM packet to PLATC:192.0.0.1 with following data: 

    CLATD

    CLIENTIP

    LEASETIME

    CLATB - bode B's own CLAT prefix

    Node B now listens to the ULA-IPv6-Unicast address generated from the clients MAC address


    Creates an individual ipset proamD for holding the roamed clients addresses which use PLATD

    ipset create proamD hash:ip family inet6

    NETMAPs the destination to PLATC of outgoing traffic from roamed clients that use PLATC

    ip6tables -t  nat -A PREROUTING -i clat -m set --match-set proamD src  -j NETMAP --to PLATC/96

    NETMAPs the source of incoming traffic to PLATB if its source is PLATC and the destination is a roamed client that uses PLATC

    ip6tables -t nat -A POSTROUTING -o clat -s PLATC/96 -m set --match-set proamD dest -j NETMAP --to PLATB/96

    Adds CLATB:CLIENTIP to ipset

    ipset add proamD CLATB:CLIENTIP timeout LEASETIME

    Sets an ARP entry for the clients MAC address with CLIENTIP


    What happens on node C?

    When the exitnode receives a ROAM packet, it:

    Needs to delete any old ipset rules for that client if existing

    MAP the destination of outgoing traffic for that client from CLATD to CLATB

    MAP the source of incoming traffic from that client from CLATB to CLATD


    Create one ipset rofromD for holding the clients new addresses (e.g. CLATB:CLIENTIP) that roamed from CLAT D

    ipset create rofromD hash:ip family inet6

    Create one ipset rotoB for holding the clients old address (e.g. CLATD:CLIENTIP) that roamed to CLAT B

    ipset create rotoB hash:ip family inet6

    MAP the source of incoming traffic from NEWCLAT to CLATD if its source is a roamed client

    [ -I(NSERT) to set rule before PLAT NAT66 ]

    ip6tables -t nat -I POSTROUTING -o plat -m set --match-set rofromD src -j NETMAP --to CLATD/96

    MAP the destination to CLATB of outgoing traffic destined to the client

    ip6tables -t nat -A PREROUTING -i plat -m set --match-set rotoB dest -j NETMAP --to CLATB/96

    Add ipset rules for client to activate translation

    ipset add rotoB CLATD:CLIENTIP timeout LEASETIME

    ipset add rofromD CLATB:CLIENTIP timeout LEASETIME


    What happens on node D?

    nothing.


Security considerations

    There is a (several) security issue with the roaming process. If an attacker can make out a clients IPv4 address, its initial CLAT prefix used and the PLAT it uses, the attacker can fake a ROAM message and send it to the PLAT.

    The attacker can e.g. simply set the LEASETIME in this packet to 1 seconds (worse 0) to block IPv4 internet access for the client until its lease really times out or redirect the traffic.

    This issue is not severe compared to other possibilities given to an attacker in open mesh infrastructure, but it is only fair to note it here.