- A local copy of AFL with Doug Birdwell's networking patch
- A binary dump of a BGP OPEN packet
- Latest copy of Free Range Routing
- Patience
AFL is designed to fuzz applications which accept input from a file or stdin. In order to convince it to send fuzzed data over a network link it needs some modifications. Thankfully someone else has done this already, and you can get the patched version of AFL here: https://github.com/jdbirdwell/afl. Go ahead and git clone that.
One additional patch is necessary for our purposes:
diff --git a/afl-fuzz.c b/afl-fuzz.c
index 46d1002..20d3b7d 100644
--- a/afl-fuzz.c
+++ b/afl-fuzz.c
@@ -8205,13 +8205,14 @@ int main(int argc, char** argv) {
*N_found1 = 0;
N_hostspec = N_found1 + 3;
}
+#if 0
if (!(
(strcmp("localhost", N_hostspec) == 0)
|| (strcmp("::1", N_hostspec) == 0)
|| (strcmp("127.0.0.1", N_hostspec) == 0)
)
) FATAL("-N: only hosts allowed are localhost, ::1, and 127.0.0.1");
-
+#endif
if (strcmp("localhost",N_hostspec) == 0) {
N_hints.ai_family = AF_UNSPEC;
} else if (strcmp("::1",N_hostspec) == 0) {
@@ -8277,13 +8278,14 @@ int main(int argc, char** argv) {
*N_found1 = 0;
N_hostspec = N_found1 + 3;
}
+#if 0
if (!(
(strcmp("localhost", N_hostspec) == 0)
|| (strcmp("::1", N_hostspec) == 0)
|| (strcmp("127.0.0.1", N_hostspec) == 0)
)
) FATAL("-N: only hosts allowed are localhost, ::1, and 127.0.0.1");
-
+#endif
if (N_hints.ai_family == AF_UNSPEC) {
if (getaddrinfo(N_hostspec, N_servicename, &N_hints, &N_results) != 0) {
FATAL( "-N: getaddrinfo() lookup failed");
This just removes an unnecessary check that the specified address bind for AFL's network socket is localhost. This is required because BGPd will listen on localhost so we need to bind to another address. After applying this patch you can build AFL with $ make and you should be ready to roll.
Make sure you don't include the TCP/IP headers as AFL will handle this for us later on. For your convenience, here's a hexdump of a BGP OPEN I made:
vagrant@host /vagrant> xxd bgp-open.bin
0000000: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000010: 0055 0104 fdea 0009 0a00 0102 3802 0601 .U..........8...
0000020: 0400 0100 0102 0805 0600 0100 0100 0202 ................
0000030: 0280 0002 0202 0002 0641 0400 00fd ea02 .........A......
0000040: 0645 0400 0101 0102 0649 0402 7232 0002 .E.......I..r2..
0000050: 0440 0280 78 .@..
You can turn this back into binary with
$ xxd -r <hexdump.in>
This one's easy:
$ git clone git@github.com:freerangerouting/frr.git
AFL works by executing the fuzz target, feeding it fuzzed input, watching for new code paths or crashes, and then killing the process. Since it was not designed to work with network services, which typically never exit, we need to make some modifications to BGPd so it will exit at an appropriate spot. The general idea is that we want to force an exit if BGPd reaches a state that we can consider a successful parse of the input packet. In this case, since we're fuzzing BGP OPEN, we can just exit (with success) if the BGP state changes to OpenConfirm, meaning it received and successfully processed a BGP OPEN from a peer. Here's a patch for that:
diff --git a/bgpd/bgp_fsm.c b/bgpd/bgp_fsm.c
index 60a6475..99e433f 100644
--- a/bgpd/bgp_fsm.c
+++ b/bgpd/bgp_fsm.c
@@ -975,6 +975,11 @@ bgp_fsm_change_status (struct peer *peer, int status)
peer->host,
LOOKUP (bgp_status_msg, peer->ostatus),
LOOKUP (bgp_status_msg, peer->status));
+ if (peer->status == OpenConfirm)
+ {
+ zlog_debug ("--->>> AFL RUN COMPLETE");
+ exit (0);
+ }
}
As mentioned in the AFL homepage, AFL monitors the executable to be fuzzed by inserting instrumentation code at compile time. This lets it watch what code paths a given input triggers and allows it to detect crashes, hangs, and interesting test cases to explore. So we need to configure, build and install Free Range Routing using its modified version of GCC that will insert this code for us. The way to do this is:
cd frr
./bootstrap.sh
CC=/home/vagrant/afl-network/afl-gcc CFLAGS='-std=gnu99' ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --libexecdir=\${prefix}/lib/quagga --disable-maintainer-mode --enable-dependency-checking --enable-exampledir=/usr/share/doc/quagga/examples/ --localstatedir=/var/run/quagga --sbindir=/usr/lib/quagga --sysconfdir=/etc/quagga --enable-ospfclient=yes --enable-ospfapi=yes --enable-multipath=64 --enable-user=quagga --enable-group=quagga --enable-vty-group=quaggavty --enable-configfile-mask=0640 --enable-logfile-mask=0640 --enable-rtadv --enable-gcc-rdynamic --with-libpam --enable-systemd=yes --enable-poll=yes --enable-vtysh=yes --enable-pimd=yes --enable-static=yes --enable-bgp-vnc=no
AFL_HARDEN=1 make -j 4
systemctl stop quagga
rm -rf /var/log/quagga/*.log
rm -rf /var/log/quagga/*.gz
rm -rf /var/support/old*
rm -rf /var/support/core/*
rm /usr/lib/quagga/*
make install
You will need to change the path to afl-gcc to wherever you cloned it to. And if you use a shell other than bash you will probably need to change the way the environment variables are set. For fish you can do it like this, by putting env before the variables.
env CC=/home/vagrant/afl-network/afl-gcc CFLAGS='-std=gnu99' ./configure <...snip...>
Assuming everything went to plan, you now have an instrumented copy of BGPd (and everything else) installed and ready for fuzzing.
BGPd will only accept connections from peers it knows about. We need to configure BGPd to expect the OPEN packets we'll be sending it, and we need to modify the OPEN packets to reflect that peer information. This is pretty easy but will require manually editing some hex.
First we need to choose what interface we'll be using for AFL and assign it an IP. I ran all of this on Cumulus VX, which comes out of the box with two swp interfaces:
vagrant@host ~> ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:35:24:2b brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:32:32:5d brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:c5:e4:e4 brd ff:ff:ff:ff:ff:ff
I chose swp2 here. Bring up the interface and give it the IP 10.0.1.2/24 (or whatever):
$ ip link set swp2 up
$ ip addr add 10.0.1.2/24 dev swp2
Now we need to configure BGPd to look for a peer on this address. Here's the config file I used:
vagrant@host ~> sudo cat /etc/quagga/bgpd.conf
no log monitor
no log monitor
no log monitor
!
log file /var/log/quagga/quagga.log
!
log syslog informational
!
log timestamp precision 6
!
debug bgp keepalives
debug bgp neighbor-events
router bgp 65001
bgp router-id 10.0.0.1
neighbor 10.0.1.2 remote-as 65002
timers bgp 1 3
!
line vty
!
Note the neighbor line. We've configured the peer's router-id as 10.0.1.2 and its ASN as 65002. So the BGP OPENs we send will need to reflect this, or BGPd will return early because it checks those fields very early on in the packet parsing function, and won't parse the rest of the packet, resulting in most of our fuzzing efforts going to waste. Thus we need to open up the hexdump of our BGP OPEN and make a couple of modifications. Or you can use a packet editor. I chose to edit the hex because it's faster. So going back to that hexdump earlier:
0000000: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000010: 0055 0104 fdea 0009 0a00 0102 3802 0601 .U..........8...
0000020: 0400 0100 0102 0805 0600 0100 0100 0202 ................
0000030: 0280 0002 0202 0002 0641 0400 00fd ea02 .........A......
0000040: 0645 0400 0101 0102 0649 0402 7232 0002 .E.......I..r2..
0000050: 0440 0280 78 .@..
Hovering over the appropriate packet information in Wireshark will show you which bytes to edit.
Reverse the edited dump back into a binary with xxd -r, and you have your packet.
So, let's recap what should have happened by now:
- You have a fuzzed version of Free Range Routing compiled and installed
- You have a network interface designated for use by AFL, with an IP address assigned to it
- You have a peer configured in the bgp config file with that same IP address and an ASN of your choosing
- Your BGP OPEN packet has the 'My AS' field set to that ASN, and the 'BGP Identifier' field set to that same IP address
If all of this is true, we're ready to start fuzzing. First of all, stop Free Range Routing if it's running. When run as a systemd service, BGPd will read the wrong configuration file, and systemd will restart it if it notices that the daemon goes down, which is a problem since AFL needs to have control over that.
$ systemctl stop quagga
Now make a directory called 'bgpd' or whatever you want under testcases/ in the afl directory and put your fuzzed bgp packet in there:
vagrant@host ~/afl> mkdir testcases/bgpd
vagrant@host ~/afl> cp ~/bgp-open.bin testcases/bgpd
Make a directory for AFL to store its findings in:
vagrant@host ~/afl> mkdir findings
We are now ready to run AFL.
Doug Birdwell's network patch has two modes of operation. In the first mode, AFL opens a TCP connection to the target, sends its fuzzed input, and then closes the connection. In the second mode, it opens a TCP connection, waits to receive a packet (which it ignores) and then sends the fuzzed input. While technically it's possible to use either mode, one of the first things BGPd does after receiving an OPEN is try to get some additional information about the sender by doing some lookups with the connection. Since the connection will be closed by the time it gets there, this will fail and we'll get an early return from the OPEN parsing. Thankfully the first thing BGP does when it comes up is send an OPEN to each of its configured peers, so we can use the second mode and wait to receive that packet before sending our response. This will ensure that BGPd gets all the way to OpenConfirm (if it doesn't crash ;- ) and then hits the exit (0) we put in place earlier.
Thus the command to use is:
vagrant@host ~/afl> sudo ./afl-fuzz -i testcases/bgpd/ -o findings/ -D 1000 -t 500 -L -N tcp://10.0.1.2:179 /usr/lib/quagga/bgpd -f /etc/quagga/bgpd.conf
Argument breakdown:
- -i: input directory containing our OPEN packet
- -o: directory to store results in
- -D: time to wait for the program to come up before sending any network data (ms)
- -t: time to wait for program to exit itself before killing the program (ms)
- -L: listen mode, discussed above
- -N: specification of protocol, address and port to bind on
- the program to run with any arguments
Note that because we only have BGPd exit if it is successful in parsing the packet, if it rejects the packet for any reason or hangs it will stay alive and AFL will kill it after 500ms. AFL will mark this as a hang. Thus for our purposes we can just ignore hangs reported by AFL since those will almost always be a result of the parser rejecting the fuzzed packet.
Also note that by default, BGPd binds to port 179 on every address. Luckily, AFL will bind to the address it uses before bringing BGPd up, and so BGPd won't bind to that one. Otherwise BGPd would start sending BGP OPENs to itself which screws everything up...
If you did everything correctly and I didn't make any mistakes writing this, you should see AFL do some dry runs, warn about slow execution (unavoidable), and then settle into its normal fuzzing interface. If you tail -f /var/log/quagga/quagga.log you can watch BGPd come up, send an OPEN, get an OPEN back, and either reject the packet (and usually send a NOTIFICATION about why), parse it successfully, or crash.