In this paper, we propose SYSTEM, a novel fuzzer that uses a symbolic virtual machine (SVM) to generate input programs.
We used SYSTEM to search for bugs in binutils version 2.34, and we found 50 new bugs. The search ran for 24 hours on a system with a EPYC 7003 CPU and 1 TB of memory. We found 8 high severity vulnerabilities: CVE-2020-2736, CVE-2020-2737, CVE-2020-25758, CVE-2020-25759, CVE-2020-25757, CVE-2020-25756, CVE-2020-25760 and CVE-2020-25754.
We use SYSTEM to search for bugs in parser generators and programming languages. We found 4 new vulnerabilities in the Racc ruby parser generator.
We use SYSTEM to search for bugs in three Linux distributions: Arch Linux, Fedora and Debian. We found 31 bugs in a one week search. 14 of the bugs are medium or high severity.
Clearly, SYSTEM is able to find bugs in a real-world setting. The question is, how many bugs does it find? To answer this question, we had to change SYSTEM a bit to make it less effective at finding bugs. We ran the fuzzer on a set of Linux distributions and parser generators, and we found that SYSTEM can find approximately 1-2 bugs per day.
The number of bugs found by SYSTEM depends on the number of inputs generated by the fuzzer. We found that approximately 10-100 inputs need to be generated for each new bug found. Therefore, the more inputs SYSTEM generates, the more bugs it will find.
We believe that 1-2 bugs per day is a good rate for a fuzzer. However, we would like to see SYSTEM find more bugs. To do this, we need to generate more inputs.