Notes: OpenWRT / Codel on TL-WDR4300

gistfile1.md

Overview

I'm tired of massive delay (500-1000 ms pings) while uploading large files. QoS helps some, but a significant portion remains due to bufferbloat (100-150ms). This leads to delay in VoIP calls and generally sluggish web browsing while uploading.

The new Codel algorithm in OpenWRT / Attitude Adjustment should help a lot. The results below show only ~5-6ms of added latency during uploads (i.e. 14ms vs 500+ms before!). Insane.

These instructions are for the TP-Link TL-WDR4300, because I got a good deal. If you have the money, buy a Netgear WNDR3800 and install CeroWRT, you'll probably see even better results.

Why the TP-Link TL-WDR4300

In short, its fully supported in OpenWRT Attitude Adjustment and works well for my needs:

• Cheap
• Simultaneous dual-band (2.4/5GHz)
• Well supported in OpenWRT
• Atheros chipsets:
  • Up to date drivers (req'd for Attitude Adjustment / Codel)
  • Very advanced capabilities via the ath9k driver
  • Want to standardize on Atheros since I often need WDS (Any combination of Broadcom WDS / RALink WDS / Atheros WDS does not work well together)
• Detachable antennas / cheap availability of high-gain dual-band antennas.
  • (Hoping I can get by with a single router without needing WDS or repeaters).

The TL-WDR3600 is almost identical (2x2 MIMO on 5GHz instead of 3x3 on the TL-WDR4300). Setup / functionality should be the same.

Other good routers

Netgear WNDR3800 or anything supported in Attitude Adjustment.

I believe that currently rules out all Broadcom devices (for now?). I think they only have binary drivers and they only work on 2.4/2.6. The Codel work requires 3.3 or higher. Most Cisco/Linksys, Asus routers are using Broadcom. :-/

Why OpenWRT and not Tomato / dd-wrt / {my favorite firmware}

I think most of them are stuck on old Linux kernels (2.4/2.6), but Codel only works on 3.3+.

I think its primarily so they can support Broadcom drivers.

Initial thoughts

• Yes, codel is doing its job. Magnificently. Results below.
• I can barely even tell when I'm uploading. Latency is great. Browsing used to be very sluggish during uploads ("ughh" every time Google Music uploader starts). So far its seems much snappier, almost like I can't tell I'm uploading.
• Jitter during uploads is less than my sub-20ms target (one single g.711u packet). This means a shorter jitter buffer resulting in less delay.
• OpenWRT flashed fine right from the default TP-Link firmware. No tftp or serial cable required.
• OpenWRT is weird (but awesome!). Default install does not include web interface. Must:
  • Flash firmware
  • Telnet to 192.168.1.1 (ssh not enabled yet)
  • Set up /etc/config/network for your internet connection only if your ISP not using DHCP (e.g. you're using DSL/PPPoE).
  • Once you have internet access, then you can use 'opkg install luci' to install the web interface.
  • Most (all?) other config can happen via web interface (http://192.168.1.1) after that.
• No multiple SSID support so far (atleast in the GUI). I'm surprised honestly. Would have thought these Atheros chipsets supported that. Might be a bug with newer builds of OpenWRT (looking into it).
• Wireless (RF) output seems a little low. I've heard there's a flag you can set so OpenWRT will let you set it higher. Requires compiling your own OpenWRT build.

Results

• Sample pings to my "first hop" during a large upload, listed cronologically (newest at bottom):

    Reply from 207.118.111.1: bytes=32 time=9ms TTL=254  <- QoS neither installed, nor enabled.  No uploads in progress
    Reply from 207.118.111.1: bytes=32 time=9ms TTL=254
    Reply from 207.118.111.1: bytes=32 time=9ms TTL=254
    Reply from 207.118.111.1: bytes=32 time=9ms TTL=254

    Reply from 207.118.111.1: bytes=32 time=336ms TTL=254    <-- Upload started. 300-500ms (no QoS)
    Reply from 207.118.111.1: bytes=32 time=542ms TTL=254
    Reply from 207.118.111.1: bytes=32 time=446ms TTL=254
    Reply from 207.118.111.1: bytes=32 time=342ms TTL=254

    Reply from 198.125.244.1: bytes=32 time=22ms TTL=254  <-- QoS (via Codel) Enabled, Upload in progress, this ICMP (ping) packets are not prioritized via QoS.  Avg 16.09ms.
    Reply from 198.125.244.1: bytes=32 time=12ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=19ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=21ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=14ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=21ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=16ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=21ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=18ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=17ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=19ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=15ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=14ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=14ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=10ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=24ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=14ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=16ms TTL=254
    Reply from 198.125.244.1: bytes=32 time=20ms TTL=254

    
    Reply from 98.125.244.1: bytes=32 time=10ms TTL=254  <-- QoS (via Codel) Enabled, Upload in progress, ICMP (ping) packets prioritized via QoS ('priority' queue).  Avg 13.74ms.
    Reply from 98.125.244.1: bytes=32 time=13ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=11ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=22ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=19ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=12ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=12ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=15ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=18ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=10ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=13ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=9ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=21ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=18ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=10ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=17ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=19ms TTL=254
    Reply from 98.125.244.1: bytes=32 time=22ms TTL=254

• Codel is definitely in use (notice the fq_codel mentions below):

root@OpenWrt:/etc/config# tc -s -d qdisc show

qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 44528859 bytes 64845 pkt (dropped 0, overlimits 0 requeues 1)
 backlog 0b 0p requeues 1

qdisc hfsc 1: dev pppoe-wan root refcnt 2 default 30
 Sent 22510040 bytes 18630 pkt (dropped 3, overlimits 33984 requeues 0)
 backlog 368b 1p requeues 0

qdisc fq_codel 100: dev pppoe-wan parent 1:10 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
 Sent 82016 bytes 1684 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 429 drop_overlimit 0 new_flow_count 915 ecn_mark 0
  new_flows_len 0 old_flows_len 1

qdisc fq_codel 200: dev pppoe-wan parent 1:20 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
 Sent 8685 bytes 82 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 82 ecn_mark 0
  new_flows_len 1 old_flows_len 0

qdisc fq_codel 300: dev pppoe-wan parent 1:30 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
 Sent 22411576 bytes 16803 pkt (dropped 2156, overlimits 0 requeues 0)
 backlog 368b 1p requeues 0
  maxpacket 1492 drop_overlimit 0 new_flow_count 863 ecn_mark 0
  new_flows_len 0 old_flows_len 1

qdisc fq_codel 400: dev pppoe-wan parent 1:40 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
 Sent 4917 bytes 59 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 477 drop_overlimit 0 new_flow_count 22 ecn_mark 0
  new_flows_len 1 old_flows_len 0

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 2874823 bytes 17490 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

qdisc mq 0: dev wlan0 root
 Sent 39473 bytes 262 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

qdisc mq 0: dev wlan1 root
 Sent 37948 bytes 250 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

qdisc hfsc 1: dev ifb0 root refcnt 2 default 30
 Sent 3329407 bytes 17488 pkt (dropped 1, overlimits 1729 requeues 0)
 backlog 0b 0p requeues 0

qdisc fq_codel 100: dev ifb0 parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 57011 bytes 399 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 560 drop_overlimit 0 new_flow_count 116 ecn_mark 0
  new_flows_len 0 old_flows_len 1

qdisc fq_codel 200: dev ifb0 parent 1:20 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

qdisc fq_codel 300: dev ifb0 parent 1:30 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 3272224 bytes 17087 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1518 drop_overlimit 0 new_flow_count 1439 ecn_mark 0
  new_flows_len 1 old_flows_len 1

qdisc fq_codel 400: dev ifb0 parent 1:40 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

Step by Step

1. Load the following webpages so you have access while offline: http://wiki.openwrt.org/doc/uci/network

2. Download openwrt for the tl-wdr4300. You want the squashfs-factory version. http://downloads.openwrt.org/snapshots/trunk/ar71xx/ http://downloads.openwrt.org/snapshots/trunk/ar71xx/openwrt-ar71xx-generic-tl-wdr4300-v1-squashfs-factory.bin

3. Flash using web admin

4. Telnet to 192.168.1.1

5. Bring up the network so we can install LuCI (the web interface)

   Optional (if your WAN connection doesn't use DHCP): Config per http://wiki.openwrt.org/doc/uci/network (edit /etc/config/network) when done, restart the network to make changes take effect: /etc/init.d/network restart

6. Install LuCI: http://wiki.openwrt.org/doc/howto/luci.essentials#installation opkg update opkg install luci

   Enable the LuCI service (so that it will be started at every boot) and start uHTTPd: /etc/init.d/uhttpd enable /etc/init.d/uhttpd start

7. Install QoS web interface module for LuCI: opkg install luci-app-qos

8. Follow the QoS Quick Start Guide: http://wiki.openwrt.org/doc/uci/qos#quick.start.guide:

   root@OpenWrt:/etc/config# opkg install qos-scripts root@OpenWrt:/etc/config# uci set qos.wan.upload=700 # Choose an appropriate number for your connection. My upload speed it ~768kbps upload speed, but its good to be a little conservative. root@OpenWrt:/etc/config# uci set qos.wan.download=10000 # Choose an appropriate number for your connection. My download speed is ~10500kbps, but its good to be a little conservative. root@OpenWrt:/etc/config# uci commit qos

   root@OpenWrt:/etc/config# /etc/init.d/qos start root@OpenWrt:/etc/config# /etc/init.d/qos enable

9. Log into LuCI web interface (http://192.168.1.1)

10 LuCI > System > Administration * Set password * Enable SSH Access ("Dropbear Instance") on lan interface * Save & Apply

11. Reboot router (if no LuCI > Network > QoS tab. It didn't show up for me until after a reboot)

12. LuCI > Network > QoS

    • Interfaces > WAN > Enable: checked

13. LuCI > Network > Wifi > Radio0 > Edit

    • Note: radio0 seems to be 2.4GHz

    • General Setup > Enable

    • General Setup

    • Device Configuration > General Setup tab:

      • Channel
      • Transmit Power

    • Device Configuration > Advanced Settings tab:

    • Interface Configuration > General Setup

      • ESSID (Network Name)
      • Mode (probably Access Point, but whatever you want)
      • Network: lan

    • Interface Configuration > Wireless Security tab

      • Encryption: WPA2-PSK
      • Cipher: Force CCPM (AES) (TKIP is less secure, but you may need it if you have an old device that can't connect)
      • Key:

    • Save & Apply

Thanks for posting your notes, this helped me decide to buy one.

Thank you petedoyle. I got tired of the tp-link tl-wr1043nd router. Thanks to you I decided to buy this model.

Dave Täht's debloat Lua script sets up fq_codel on the wlan ports by creating separate queues on top of the multi-queue (mq) for the different 802.11e priorities (vo := voice, vi := video, be := best effort, bk := background):

tc qdisc del dev wlan0 root
tc qdisc add dev wlan0 handle 1 root mq
tc qdisc add dev wlan0 parent 1:1 handle 10 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan0 parent 1:2 handle 20 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan0 parent 1:3 handle 30 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan0 parent 1:4 handle 40 fq_codel limit 1000 quantum 1000

tc qdisc del dev wlan1 root
tc qdisc add dev wlan1 handle 1 root mq
tc qdisc add dev wlan1 parent 1:1 handle 10 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan1 parent 1:2 handle 20 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan1 parent 1:3 handle 30 fq_codel limit 1000 quantum 1000
tc qdisc add dev wlan1 parent 1:4 handle 40 fq_codel limit 1000 quantum 1000

The script further reduces the hardware queue size:

ifconfig wlan0 txqueuelen 50
ifconfig wlan1 txqueuelen 50

echo "2" > /sys/kernel/debug/ieee80211/phy0/ath9k/qlen_vo
echo "4" > /sys/kernel/debug/ieee80211/phy0/ath9k/qlen_vi
echo "12" > /sys/kernel/debug/ieee80211/phy0/ath9k/qlen_be
echo "12" > /sys/kernel/debug/ieee80211/phy0/ath9k/qlen_bk

echo "2" > /sys/kernel/debug/ieee80211/phy1/ath9k/qlen_vo
echo "4" > /sys/kernel/debug/ieee80211/phy1/ath9k/qlen_vi
echo "12" > /sys/kernel/debug/ieee80211/phy1/ath9k/qlen_be
echo "12" > /sys/kernel/debug/ieee80211/phy1/ath9k/qlen_bk

This will also combat bufferbloat within your own network.

You could simply run the debloat script like this:

IFACE=wlan0 QMODEL=fq_codel lua debloat 

Src: https://github.com/dtaht/ceropackages-3.3/blob/master/net/debloat/files/debloat

1. fiddling with txqueuelen does not affect fq_codel. The equivalent setting is the "limit" parameter and there is generally no need to knock it much 1000 for most circumstances, unless you are severely memory limited.

2. the debloat script presently uses some dubious settings for it's quantum on wifi. I really don't know what is good, I've tried values as high as 4500 and as low as 300 on wifi and most of that gets buried deep in the device drivers. The values for setting the ath9k buffering settings were derived from experiment, and undoubtably will change once felix finishes reorganizing that layer of the stack. At the moment I generally recomend disabling 802.11e on wifi almost entirely for best aggregation, this too will change.

3. openwrt's qos system does not work properly with native ipv6 presently. Cero's does, so do several others.

4. I'm glad your results rock. :)

5. From a QoS script perspective (where you are using fq_codel primarily and where it does the most good presently), fq_codel runs on everything openwrt, dd-wrt, and the linux kernel supports. Yes, even broadcom.

If however you are running it on the underlying device, (like the raw wlan0 or raw eth0 device), BQL support is needed on the ethernet side, and as I wrote above work continues on making wireless work even better. I have no data on broadcom's wifi devices. I do like atheros's chipset a lot.