TheTechromancer/BBOT 2.0 Release Blogpost.md Secret

## BBOT 2.0 Release Blogpost.md

      
    Raw
  

              BBOT 2.0 Release Blogpost.md
            
          
    BBOT 2.0 Release

Today, just in time for DEF CON 32, we are excited to announce the release of BBOT 2.0.
The new features in 2.0 are designed to make BBOT easier to use, and significantly speed up scans.

Above: A chord graph of the relationships between each of BBOT's modules and the data types they produce/consume. Click the image to explore the graph interactively.
How did we get here?

Two years ago we released BBOT (Bighuge BLS OSINT Tool), an open-source scanner inspired by Spiderfoot. Its initial claim to fame was its ability to find more subdomains than any other tool. Since then, it's been steadily gaining users, and as of today, it's been downloaded 400K times. It's always wonderful to hear how people are using it in the bug bounty space. Whenever we hear that BBOT got someone a new payout by finding an outlier subdomain, or a critical RCE, it warms our hearts!
BBOT's success has been thanks to the countless contributions from the community, which include many of the powerful new modules and features in 2.0. Development has been happening at a fast pace. To give you an idea, BBOT has already passed 4,000 commits, surpassing even Spiderfoot (with ~3,700), a tool that has been in active development for ten years! That is how much work has been going into BBOT -- both by us at BLS, and by the community -- and how we've already arrived at version 2.0!
New Features in 2.0

BBOT 2.0 keeps BBOT's original recursive design, while adding some powerful new features and optimizations.
Note: for full release notes, see Upgrading to BBOT 2.0.
Highlights

Here are the three main feature highlights for BBOT 2.0:

Presets: An alternative to command-line flags that let you conveniently store your entire scan config in a single YAML file.
BadDNS: Find subdomain hijacks and other DNS-related vulns
Speed Optimizations

YARA integration by @liquidsec == insane boost in regex performance
New DNS/HTTP Engines by @TheTechromancer == leverage all your CPU cores!


Presets

Presets are one of the biggest features in BBOT 2.0. They were born out of necessity, to save you from having to construct giant BBOT commands. They use a simple YAML format.
Some of BBOT's builtin presets include subdomain-enum, spider, dotnet-audit, and kitchen-sink. The spider preset looks like this:
description: Recursive web spider

modules:
  - httpx

config:
  web:
    # how many links to follow in a row
    spider_distance: 2
    # don't follow links whose directory depth is higher than 4
    spider_depth: 4
    # maximum number of links to follow per page
    spider_links_per_page: 25
In a way, presets are a natural evolution from flags. Flags were convenient in that they let you enable groups of similar modules, but they lacked the ability to add config options. For example, if you wanted to enable the web spider, you had to specify a custom --config:
bbot -t evilcorp.com -m httpx --config web_spider_depth=2 web_spider_distance=2
Something we discovered early on, was that due to BBOT's extreme customizability and the fact that it has over 100 modules, these commands could get out of hand pretty quickly:

Presets solve this by consolidating all your scan settings into a single config. You can create your own, or choose from a list of builtin presets.
You can list them all with -lp:
# list BBOT presets on the command-line
bbot -lp
And enable them with -p:
# enumerate subdomains on evilcorp.com
bbot -t evilcorp.com -p subdomain-enum
You can also mix and match an unlimited number of presets:
# combine subdomains + web spider
bbot -t evilcorp.com -p subdomain-enum spider
You can also create your own custom preset that includes other presets:
target:
  - evilcorp.com
  - 1.2.3.0/24

blacklist:
  - test.evilcorp.com

# include other presets
include:
  - subdomain-enum
  - spider

config:
  web:
    http_proxy: http://127.0.0.1:8080
  modules:
    github:
      api_key: 258e88dcbd3cd44d8e7ab43f6ecb6af0
Run BBOT with your custom preset:
bbot -p ./my_preset.yml
For a full list of built-in presets, see Full List of Presets
For details on Presets, see the Documentation
BadDNS

BadDNS is a slick DNS-hijacking tool written by @liquidsec that's integrated into BBOT 2.0. It replaces BBOT's old subdomain_hijack module, and detects a myriad of vulnerabilities include dangling records.
For details, see the BaddNS Blog Post.
Speed Optimizations

BBOT 2.0 includes several very significant performance improvements, along with numerous small ones. These have combined together to make BBOT 2.0 close to 10x faster than its predecessor.
The two most significant performance-boosting features are YARA integration and new DNS + HTTP engines.
YARA Itegration

Initially, we used Python's builtin regex library to mine useful goodies (emails, URLs, subdomains, etc.) from various sources like HTTP responses. This was effective, but not very efficient. Lots of regexes multiplied against lots of data resulted serious slowdowns for the scan.
In BBOT 2.0, @liquidsec has completely overhauled the excavate module to use YARA. This not only provides an insane speed boost (YARA has some wicked algorithms for this), it allows you to add on your custom YARA rules. Pair this with the work @domwhewell-sage has done to download git repos and docker images, and pair that again with his module that extracts text from practically every file format known to man, what you effectively have is a grep -R for your target's entire web presence. Oh, and he also made a Trufflehog module to search all that for secrets.
Yeah, so @domwhewell-sage has been real busy. Stay tuned for new developments on these features. It's only going to get crazier!
New DNS / HTTP Engines

Early on in BBOT's development, we transitioned to using asyncio. This allowed us to simplify the code, and resulted in better stability and performance.
However, we are constantly looking for new ways to speed up scans, and the next bottleneck we encountered was in asyncio itself. Specifically, BBOT was issuing so many DNS and HTTP requests that it reached the max capacity of the asyncio loop within a single CPU core.
To address this, we've introduced an optimization to the way BBOT interacts with DNS and HTTP, which gives DNS and HTTP each their own dedicated Python process and asyncio event loop. To achieve this without the dreaded overhead of multiprocessing, we use ZeroMQ in a ROUTER/DEALER configuration. ZeroMQ enables extremely fast and efficient communication between the processes.
Before (BBOT v1):


After (BBOT v1):


Community Shoutouts

We want to give special thanks to four specific members of the community, who have been most active in contributing to BBOT:


@domwhewell-sage for continuing to create loads of powerful modules for secrets-looting and more.


@amiremami for testing every new feature ruthlessly, and showering us with awesome ideas!


@colin-stubbs for bringing his DNS expertise to BBOT by adding CAA-record capabilities (and many more to come!)


@nicpenning and @CarsonHrusovsky for integrating BBOT with Elasticsearch.