Skip to content

Instantly share code, notes, and snippets.

@rtreffer

rtreffer/qos-multi-network.md

Last active April 20, 2017 11:12
Show Gist options
  • Save rtreffer/f85974ca0cf654c9466d to your computer and use it in GitHub Desktop.
Save rtreffer/f85974ca0cf654c9466d to your computer and use it in GitHub Desktop.
Download ZIP
QoS on a 5 ethernet port system with 2 networks and management.
Raw
qos-multi-network.md

Ports

  • 1: Main Network in
  • 2: Main Network out
  • 3: Guest Network in
  • 4: Guest Network out
  • 5: MGMT

Interfaces

eth0: cpu switch port, tagged
eth0.1-5: switch ports

tin: virtual interface for input (paired with in)
tout: virtual interface for output (paired with out)

br1: retagging eth0.1 to tin.1
br2: retagging eth0.2 to tout.1
br3: retagging eth0.3 to tin.2
br4: retagging eth0.4 to tout.2

br0: in,out

Network flow

   eth0.1    eth0.2    eth0.3    eth0.4    eth0.5
     |         |         |         |         |
    br1       br2       br3       br4        |
     |         |         |         |         |
    tin.1    tout.1     tin.2    tout.2      |
     |         |         |         |         |
     +----+--- | --------+         |         |
          |    |                   |         |
          |    +-----+-------------+         |
         in          |                       |
          |         out                      |
          |          |                       |
          +-- br0 ---+                     (cpu)

An tc egress on in/out will yield good QoS properties.

Virtual testsetup on linux

Create the virtual interfaces

   ip netns add tc-switch
   ip netns add tc-switch-port1
   ip netns add tc-switch-port2
   ip netns add tc-switch-port3
   ip netns add tc-switch-port4
   for i in 1 2 3 4; do
     ip netns exec tc-switch-port$i ip link add veth0 type veth peer name eth$i
     ip netns exec tc-switch-port$i ip link set dev eth$i netns tc-switch
     ip netns exec tc-switch-port$i ip addr add 172.12.10.$i/24 dev veth0
     ip netns exec tc-switch-port$i ip link set dev veth0 up
     ip netns exec tc-switch ip link set dev eth$i up
   done

This results in the following setup

   namespace       | device | peer                     | Note
   -----------------------------------------------------------------------
   tc-switch-port1 | veth0  | tc-switch / eth1         | 172.12.10.1/24
   tc-switch-port2 | veth0  | tc-switch / eth2         | 172.12.10.2/24
   tc-switch-port3 | veth0  | tc-switch / eth3         | 172.12.10.3/24
   tc-switch-port4 | veth0  | tc-switch / eth4         | 172.12.10.4/24
   -----------------------------------------------------------------------
   tc-switch       | eth1   | tc-switch-port1 / veth0  | eth0.1 on real hw
   tc-switch       | eth2   | tc-switch-port2 / veth0  | eth0.2 on real hw
   tc-switch       | eth3   | tc-switch-port3 / veth0  | eth0.3 on real hw
   tc-switch       | eth4   | tc-switch-port4 / veth0  | eth0.4 on real hw
   -----------------------------------------------------------------------

Create the in/out interfaces

   ip netns exec tc-switch ip link add tin type veth peer name in
   ip netns exec tc-switch ip link add tout type veth peer name out
   for dev in tin tout; do
     ip netns exec tc-switch ip link set dev $dev up
     for vlan in 1 2 ; do
       ip netns exec tc-switch ip link add link $dev name $dev.$vlan type vlan id $vlan
       ip netns exec tc-switch ip link set dev $dev.$vlan up
     done
   done
   ip netns exec tc-switch ip link set dev in up
   ip netns exec tc-switch ip link set dev out up

Which results in (everything in namespace tc-switch)

   device | peer          | Note
   ----------------------------------------------------
   tin    | in            | tagged / vlan splitted side
   tin.1  | tin, vlan 1   | vlan tag 1 on tin
   tin.2  | tin, vlan 2   | vlan tag 2 on tin
   tin.3  | tin, vlan 3   | vlan tag 3 on tin
   tin.4  | tin, vlan 4   | vlan tag 4 on tin
   in     | tin           | trunk side
   ----------------------------------------------------
   tout   | out           | tagged / vlan splitted side
   tout.1 | tout, vlan 1  | vlan tag 1 on tout
   tout.2 | tout, vlan 2  | vlan tag 2 on tout
   tout.3 | tout, vlan 3  | vlan tag 3 on tout
   tout.4 | tout, vlan 4  | vlan tag 4 on tout
   ----------------------------------------------------

Create the required bridges

   ip netns exec tc-switch brctl addbr br0
   ip netns exec tc-switch ip link set dev br0 up
   for i in 1 2 3 4; do
     ip netns exec tc-switch brctl addbr br$i
     ip netns exec tc-switch ip link set dev br$i up
     ip netns exec tc-switch brctl addif br$i eth$i
   done
   ip netns exec tc-switch brctl addif br1 tin.1
   ip netns exec tc-switch brctl addif br2 tout.1
   ip netns exec tc-switch brctl addif br3 tin.2
   ip netns exec tc-switch brctl addif br4 tout.2
   ip netns exec tc-switch brctl addif br0 in
   ip netns exec tc-switch brctl addif br0 out

This creates all noted bridges

   device | port devices     | Note
   ---------------------------------------------
   br1    | eth1, tin.1      | retagging bridge
   br2    | eth2, tout.1     | retagging bridge
   br3    | eth3, tin.2      | retagging bridge
   br4    | eth4, tout.2     | retagging bridge
   ---------------------------------------------
   br0    | in, out          | main trunk bridge
   ---------------------------------------------

Test that everything works as expected

  for s in 1 2 3 4; do
    for t in 1 2 3 4; do
      if [ "$2" != "$t" ]; then
        ip netns exec tc-switch-port$s ping -c1 -w1 172.12.10.$t &> /dev/null && \
	echo "172.12.10.$s -> 172.12.10.$t (port $s -> $t)"
      fi
    done
  done

If everything works as expected you'll see

  172.12.10.1 -> 172.12.10.2 (port 1 -> 2)
  172.12.10.2 -> 172.12.10.1 (port 2 -> 1)
  172.12.10.3 -> 172.12.10.4 (port 3 -> 4)
  172.12.10.4 -> 172.12.10.3 (port 4 -> 3)

TC/QoS

Everyone has a different idea on how QoS should work. Here is my aim

  • Strict fairness among all clients (on an ip basis)
  • Understandable error case (e.g. running bittorent may degrade your inet)
  • Low, predictable latency (aka short queues)
  • Dual egress (somehow real ingeress QoS)

Basically every source ip (egress on out) / every target ip (egress in) on every vlan should be independant. Every IP in every vlan should be an independant queue.

We will use handle beef: as the basis, vlans will be queued under a major number representing the vlan id.

  # qos, egress towards inet
  ip netns exec tc-switch tc qdisc add dev out root handle cafe: hfsc default ffff
  ip netns exec tc-switch tc class add dev out parent cafe: classid cafe:ffff hfsc sc rate 10mbit ul rate 10mbit
  ip netns exec tc-switch tc class add dev out parent cafe:ffff classid cafe:f001 hfsc ls rate 7mbit rt umax 1kb dmax 1ms rate 10mbit
  ip netns exec tc-switch tc class add dev out parent cafe:ffff classid cafe:f002 hfsc ls rate 3mbit rt umax 1kb dmax 1ms rate 10mbit
  ip netns exec tc-switch tc qdisc add dev out parent cafe:f001 handle 1: sfq divisor 2048
  ip netns exec tc-switch tc qdisc add dev out parent cafe:f002 handle 2: sfq divisor 2048
  ip netns exec tc-switch tc filter add dev out parent 1: handle 1 protocol all flow hash keys src divisor 2048
  ip netns exec tc-switch tc filter add dev out parent 2: handle 2 protocol all flow hash keys src divisor 2048
  ip netns exec tc-switch tc filter add dev out protocol all parent cafe: prio 1 \
    basic match 'meta(vlan mask 0xfff eq 0x1)' flowid cafe:f001
  ip netns exec tc-switch tc filter add dev out protocol all parent cafe: prio 2 \
    basic match 'meta(vlan mask 0xfff eq 0x2)' flowid cafe:f002

  # qos, egress towards lan
  ip netns exec tc-switch tc qdisc add dev in root handle cafe: hfsc default ffff
  ip netns exec tc-switch tc class add dev in parent cafe: classid cafe:ffff hfsc sc rate 100mbit ul rate 100mbit
  ip netns exec tc-switch tc class add dev in parent cafe:ffff classid cafe:f001 hfsc ls rate 70mbit rt umax 10kb dmax 1ms rate 100mbit
  ip netns exec tc-switch tc class add dev in parent cafe:ffff classid cafe:f002 hfsc ls rate 30mbit rt umax 10kb dmax 1ms rate 100mbit
  ip netns exec tc-switch tc qdisc add dev in parent cafe:f001 handle 1: sfq divisor 2048
  ip netns exec tc-switch tc qdisc add dev in parent cafe:f002 handle 2: sfq divisor 2048
  ip netns exec tc-switch tc filter add dev in parent 1: handle 1 protocol all flow hash keys src divisor 2048
  ip netns exec tc-switch tc filter add dev in parent 2: handle 2 protocol all flow hash keys src divisor 2048

  ip netns exec tc-switch tc filter add dev in protocol all parent cafe: prio 1 \
    basic match 'meta(vlan mask 0xfff eq 0x1)' flowid cafe:f001
  ip netns exec tc-switch tc filter add dev in protocol all parent cafe: prio 2 \
    basic match 'meta(vlan mask 0xfff eq 0x2)' flowid cafe:f002

Note: I have 11 MBit/s upload speed. The out ports are connected to the gateway and should thus not exceed 11 MBit/s minus a safety margin. That's why rate is 10 MBit/s and ceil is 11 MiBit/s (and not 11 MBit/s). Same for the other direction where 110Mbit is deployed and 100Mbit is roughly usable.

We need one leaf class per possible client

Teardown

   for r in 1 2 ; do
   for brdev in br{0,1,2,3,4}; do
     for dev in {t,}in.{1,2} {t,}out.{1,2} in out eth{1,2,3,4}; do
       ip netns exec tc-switch brctl delif $brdev $dev 2> /dev/null
     done
     ip netns exec tc-switch ip link set dev $brdev down 2> /dev/null
     ip netns exec tc-switch brctl delbr $brdev 2> /dev/null
   done
   done
   for dev in {v,}eth{1,2,3,4} t{in,out}.{1,2} in out ; do
     ip netns exec tc-switch ip link delete dev $dev type vlan || \
     ip netns exec tc-switch ip link delete dev $dev type veth
   done
   for ns in tc-switch{,-port{1,2,3,4}}; do ip netns delete $ns ; done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment