tc_lunatik.md

Traffic Controller Bindings for Lunatik

Table of contents

1. What is visible
2. Uses
3. What can TC do today?
4. Components
5. Why Lunatik?
6. LuaXDP
7. TC Actions
8. Example

What is visible

Reassembled packets are visible at this stage. Linear portions of network packets can be inspected here. We have access to skb (socket buffer) at this level. Examples include:

• Ethernet header
• IP header
• TCP/UDP header
• IP ToS/DSCP bits
• Inspect ports, flags, etc.

It does NOT see:

• Reassembled TCP stream
• Full application-layer view

So for instance, parsing HTTP headers reliably at TC layer is not possible.

Read more here

TC gets struct __sk_buff allocated which XDP does not, so there are a few things which we can leverage at TC:

• tc_index
• tc_classid
• struct bpf_sock* sk

Uses

1. ingress: policing, we don't have control over the remote sending the packets, as soon as we inspect it, we have used our bandwidth. So we cannot shape the ingress traffic but we can drop packets based on policy

2. egress: shaping, we can control the bandwidth to be used for sending outbound packets here, hence we can shape it

Primarily used for modifying scheduler settings but for packets, think of simulating low bandwidth to see how your system performs. As an example, only allowing 100MB/s on a network to test your applications.

Classic TC u32 filter can match TCP source port and apply shaping

tc filter add dev eth1 parent 1:0 protocol ip prio 1 \
   u32 match ip protocol 6 0xff \
   match ip sport 80 0xffff \
   flowid 1:10

And eBPF TC filters can match anything in the linear portion of the packet.

Good examples on Traffic Shaping

Another use is direct action by eBPF programs, basically attach eBPF program to TC and set the classid of the packets dynamically, refer to this commit.

So we could shape packets by sending them to defined classes (for instance, qdisc setup involving HTB).

What can TC do today?

1. Queueing and Scheduling

• HTB (Hierarchical Token Bucket)
• FQ-CoDel: Fair queuing with active queue management to reduce bufferbloat.
• TBF: rate limiting
• netem: delay, jitter, packet loss, duplication, reordering

2. Declarative Classification

• u32 classifier: Traditional bitmask-based matching on packet headers like protocol, src/dst IP, ports, TOS/DSCP
• example: shape HTTP traffic (port 80/443) differently from SSH (port 22)
• tc flower (flow classifier): rich matching (L2–L4 fields)
• VLAN, MPLS, tunnels, hardware offload to NIC

3. eBPF-based TC Programs

• Two modes
  • Classifier mode -> return classid
  • Direct-action mode -> return TC_ACT_*
• Arbitrary packet parsing within linear skb region
• Stateful logic via BPF maps
• Dynamic class assignment

Components

traditional element	Linux component
shaping	class
scheduling	qdisc, can be simple such as the FIFO or complex, containing classes and other qdiscs, such as HTB
classifying	filter, performs the classification through the agency of a classifier object. Linux classifiers cannot exist outside of a filter
policing	policer exists only as part of a filter
dropping	to drop traffic requires a filter with a policer which uses "drop" as an action

Read more on components: Traffic-Control-HOWTO

tc uses a queue structure to temporarily store and organize packets. In the tc subsystem, the corresponding data structure and algorithm control mechanism are abstracted as qdisc (Queueing discipline).

qdisc exposes two callback interfaces for enqueuing and dequeuing packets externally, and internally hides the implementation of queuing algorithms.

In qdisc, we can implement complex tree structures based on filters and classes. Filters are mounted on qdisc or class to implement specific filtering logic, and the return value determines whether the packet belongs to a specific class.

When a packet reaches the top-level qdisc, its enqueue interface is called, and the mounted filters are executed one by one until a filter matches successfully. Then the packet is sent to the class pointed to by that filter and enters the qdisc processing process configured by that class.

The tc framework provides the so-called classifier-action mechanism, that is, when a packet matches a specific filter, the action mounted by that filter is executed to process the packet.

The existing tc provides eBPF with the direct-action mode, which allows an eBPF program loaded as a filter to return values such as TC_ACT_OK as tc actions, instead of just returning a classid like traditional filters and handing over the packet processing to the action module.

Read more on qdisc here

Several qdiscs are implemented in /net/sched/sch_*.c

picture taken from here

Why Lunatik?

The TC currently has mature scheduling primitives. The classifiers are mostly declarative and C based. There is no high level programmable policy layer.

Enter Lunatik. Lunatik enables safe Lua script execution in BPF context. Lua will act as the glue language here, providing a scripting capabilities to the kernel. For instance regex matching is very painful in pure eBPF, but very natural in Lua. So we want to match a particular header with a pattern, we could parse the packet using eBPF (fast path) and offload the complex policy to Lua (string matching).

This allows us to script the otherwise monotonous tc dynamically. We could have a decoupled architecture where Lua policy is hot reloadable without modifying loaded eBPF program.

The key is to extend TC's programmability.

LuaXDP

XDP doesn't allow LKM to be called directly. It's only extensible through eBPF (ref here)

The design uses eBPF kfuncs to run a custom function defined in Lunatik. This allows Lua callback to be run for the verdict when a packet hits the XDP hook. The below diagram summarizes the flow

Design

flowchart LR
    A[NIC] --> B[XDP hook]
    B --> C[eBPF program]
    C --> D[bpf_luaxdp_run]
    D --> E[Lunatik]
    E --> F[Lua callback]

Loading

and then Lua finally return XDP verdict.

TC Actions

Defined in the kernel's uapi, the following TC actions exist:

#define TC_ACT_UNSPEC	(-1)
#define TC_ACT_OK		0
#define TC_ACT_RECLASSIFY	1
#define TC_ACT_SHOT		2
#define TC_ACT_PIPE		3
#define TC_ACT_STOLEN		4
#define TC_ACT_QUEUED		5
#define TC_ACT_REPEAT		6
#define TC_ACT_REDIRECT		7
#define TC_ACT_TRAP		8 

From the Lua binding we can add support as follows:

tc.action.OK
tc.action.SHOT
tc.action.PIPE
tc.action.RECLASSIFY
tc.action.REDIRECT

Example

Consider this DNS shaper consisting

• eBPF hook for past path
• eBPF map for maintaining state
• Lua based string matching for TC verdict (classid)

1. Setup TC configuration

• Add a root HTB
• Define 3 classes -> 20mbit for video calling, 10mbit for streaming netflix and 5mbit for bulk transfers

tc qdisc del dev eth0 root 2>/dev/null
tc qdisc add dev eth0 root handle 1: htb default 30

tc class add dev eth0 parent 1: classid 1:10 htb rate 20mbit  # Zoom
tc class add dev eth0 parent 1:20 classid 1:20 htb rate 10mbit  # Streaming
tc class add dev eth0 parent 1: classid 1:30 htb rate 5mbit   # S3/Default

2. Attach eBPF program which would call our Lua callback in case of DNS responses recieved

// tc.bpf.c
SEC("classifier")
int luatc_dns_shaper(struct __sk_buff *skb)
{
    // is this a DNS response?
    // if yes -> call Lua for string matching + map update
    if (is_dns_response(skb)) {     // not very frequent
        return bpf_luatc_run(skb);  // Lua handles parsing + map write
    }

    // fast path: all other traffic
    // lookup ip in map populated by Lua
    __u32 dst_ip = get_dst_ip(skb);
    __u32 *classid = bpf_map_lookup_elem(&ip_class_map, &dst_ip);
    if (classid) {
        skb->tc_classid = *classid;
    }

    return TC_ACT_OK;
}

tc filter add dev eth0 ingress bpf da obj tc.bpf.o sec classifier

3. On Lua side we handle the shaping, by maintaining class id in a map

-- classifier.lua
local tc  = require("tc")
local map = require("map")

local ip_class_map = map.open("ip_class_map") -- need to finalize the API for maps

local policy = {
    { pattern = "%.zoom%.us$",          classid = 0x00010010 },
    { pattern = "%.netflix%.com$",      classid = 0x00010020 },
    { pattern = "%.youtube%.com$",      classid = 0x00010020 },
    { pattern = "s3%.amazonaws%.com$",  classid = 0x00010030 },
}

local function classify_domain(domain)
    for _, rule in ipairs(policy) do
        if domain:match(rule.pattern) then
            return rule.classid
        end
    end
    return nil
end

local function handler(pkt)
    local domain = parse_dns_name(pkt, 42 + 12)
    local classid = classify_domain(domain)
    if classid then
        -- store all A record IPs -> classid
        store_dns_answers(pkt, classid, ip_class_map)
    end
    return tc.action.OK
end

tc.attach(handler)