Skip to content

Instantly share code, notes, and snippets.

@CpuID

CpuID/plug Secret

Created March 24, 2016 04:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save CpuID/9240ca9e77604211a57d to your computer and use it in GitHub Desktop.
Save CpuID/9240ca9e77604211a57d to your computer and use it in GitHub Desktop.
SYN plug troubleshooting
nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: --update plug --buffer
# do some stuff while we have a buffer building, reload processes, etc
nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: --update plug --release-indefinite
# goal: plug SYN packets on TCP 8080 hitting an IPv4 address on eth0, when plug qdisc is "plugged", then released to flush them.
ip link add ifb0 type ifb
tc qdisc add dev ifb0 root handle 1: prio bands 4
tc qdisc add dev ifb0 parent 1:1 handle 10: pfifo limit 1000
tc qdisc add dev ifb0 parent 1:2 handle 20: pfifo limit 1000
tc qdisc add dev ifb0 parent 1:3 handle 30: pfifo limit 1000
nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: plug --limit 1048576
nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: --update plug --release-indefinite
tc filter add dev ifb0 protocol ip parent 1:0 prio 1 handle 1 fw classid 1:4
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip prio 1 u32 match ip dport 8080 0xffff flowid 1:1 action mirred egress redirect dev ifb0
# tried this at one point, with no success (invalid ipt action)
#tc filter add dev eth0 parent ffff: protocol ip prio 1 u32 match ip dport 8080 0xffff flowid 1:1 action ipt -j MARK --set-mark 1 action mirred egress redirect dev ifb0
# alternative via iptables, not sure if its the right table/chain though?
iptables -t mangle -I OUTPUT -p tcp --dport 8080 --syn -j MARK --set-mark 1
@CpuID
Copy link
Author

CpuID commented Mar 24, 2016

Notes:

  • run within a Docker container, with the NET_ADMIN capability added
  • startup = run at container boot, to setup the link/qdisc's etc
  • plug = run at plug time, when the feature is required

@CpuID
Copy link
Author

CpuID commented Mar 24, 2016

Issue: SYN packets never end up in the plug qdisc when looking at tc -s qdisc show

@CpuID
Copy link
Author

CpuID commented Mar 24, 2016

Attempting to perform HAProxy reloads using this guide as a basis: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html

@keepclean
Copy link

Hello.

I'm also trying to find the correct way to restart some balancer program.
The program is running on port 4444.
So, first iteration on that way is that script:

modprobe ifb numifbs=1
ip link set dev ifb0 up

tc qdisc add dev ifb0 root handle 1: prio bands 4
tc qdisc add dev ifb0 parent 1:1 handle 10: pfifo limit 1000
tc qdisc add dev ifb0 parent 1:2 handle 20: pfifo limit 1000
tc qdisc add dev ifb0 parent 1:3 handle 30: pfifo limit 1000

nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: plug --limit 1048576
nl-qdisc-add --dev=ifb0 --parent=1:4 --id=40: --update plug --release-indefinite

tc filter add dev ifb0 protocol ip parent 1:0 prio 1 handle 1 fw classid 1:4

tc qdisc add dev enp0s8 ingress
tc filter add dev enp0s8 parent ffff: protocol ip prio 1 u32 match ip dport 4444 0xffff flowid 1:1 action skbedit mark 1 pipe action mirred egress redirect dev ifb0

I'm not sure that last line is exactly right, because all packets goes to 40.

$ tc -s q s dev ifb0
qdisc prio 1: root refcnt 2 bands 4 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 209330 bytes 2567 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 10: parent 1:1 limit 1000p
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 20: parent 1:2 limit 1000p
 Sent 140 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 30: parent 1:3 limit 1000p
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc plug 40: parent 1:4
 Sent 209190 bytes 2565 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

first result - http://pastebin.com/dfmPdbju

@joshughes
Copy link

joshughes commented Jan 18, 2017

I am unable to get the plug to create... is there any trick to that?
I have the latest kernel and NET_ADMIN but just get an error when ever I go to create the plug.

root@c06b5076b636:/#  nl-qdisc-add --dev=eth0 --parent=1:4 --id=40: plug --limit 1048576
Adding qdisc plug dev eth0 id 40: parent 1:4
  refcnt 0 no options
Error: Unable to add qdisc: Object not found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment