Visualising Amass Data Over Time

Visualising Amass Data Over Time.livemd

Visualising Amass Data Over Time

Mix.install([
  {:jason, "~> 1.4"},
  {:kino, "~> 0.9.2"},
  {:vega_lite, "~> 0.1.7"},
  {:kino_vega_lite, "~> 0.1.8"}
])

What's This About?

Point this notebook to an Amass data directory to get a historical perspective on the enumeration data. It'll plot a chart like the one below: enumeration index along the x-axis, domain name along the y-axis, and the number of hosts indicated by the dot size and shade.

Introduction

Pick a domain name and the enumeration indices that you're interested in. The cells that need your input are marked with a ➡️.

Of course, you have to have already run a handful of enumerations on the same target, and have the data directory locally and Amass installed to use this notebook. You'll also need Elixir's Livebook. Installation is easy: https://livebook.dev/#install. Start Livebook with the CLI so that the shell and command utilities inherit your environment so that we can find the Amass executable.

If you like it, remember to ⭐️ it. Feedback is very welcome! Write to me at joseph@yiasemides.com.

Set-Up & Configuration

Let's make sure we can find Amass!

{_, 0} = System.shell("which amass")

➡️ Now choose an Amass data directory (perhaps you've pulled one over from a different machine where you're running reconnaissance).

path_in = Kino.Input.text("Path")

path = Kino.Input.read(path_in)

➡️ The domain you want to examine.

domain_in = Kino.Input.text("Domain")

domain = Kino.Input.read(domain_in)

Let's find out how many enumeration entries you've got for that domain.

{out, 0} = System.cmd("amass", ["db", "-d", domain, "-list", "-dir", path])

out
|> String.split("\n\n")
|> Enum.count()

The scans are sorted most recent first. Index 1 being the most recent. Not 0 and not whatever the number above is.

out
|> String.split("\n\n")
|> Enum.take(5)

➡️ Enter a start and stop index. Remember, as per the above, the most recent scan is at index 1. A start of 25 and a stop of 1 will give you the 25 most recent enumerations (so start > stop).

start_ix_handle = Kino.Input.number("Start index")

stop_ix_handle = Kino.Input.number("Stop index")

{start, stop} = {Kino.Input.read(start_ix_handle), Kino.Input.read(stop_ix_handle)}

if start > stop do
  Kino.Text.new("All good!")
end

The Amass DB Enumeration Object

In case you're curious, here's the JSON object for the most recent scan in an friendly-to-browse tree view.

{out, 0} =
  System.cmd("amass", [
    "db",
    "-d",
    domain,
    "-enum",
    Integer.to_string(1),
    "-json",
    "-",
    "-dir",
    path
  ])

obj = Jason.decode!(out)
Kino.Tree.new(obj)

Crunching The Data

The code below works enumeration-by-enumeration, between the indices given, toward the most recent enumeration. It invokes Amass, reads the data in, and extracts the domain -> [host] mapping for each enumeration. Intermediate results are flattened so that there is no nesting in the data. The resulting table has three columns: enumeration index, domain name, and IP address. While there is a lot of redundancy, this is the format that works best with VegaLite, the plotting library we're using.

history =
  for ix <- start..stop//-1, reduce: [] do
    acc ->
      {out, 0} =
        System.cmd("amass", [
          "db",
          "-d",
          domain,
          "-enum",
          Integer.to_string(ix),
          "-json",
          "-",
          "-dir",
          path
        ])

      out
      |> Jason.decode!()
      |> Map.fetch!("domains")
      |> Enum.map(& &1["names"])
      |> List.flatten()
      |> Enum.flat_map(fn %{"name" => name, "addresses" => addresses} ->
        for address <- addresses, do: %{scan: ix, name: name, host: address["ip"]}
      end)
      |> Enum.concat(acc)
  end

Kino.DataTable.new(history)

Visualising Domain -> [Host] Over Time

Magnify the chart below using the 🔍 icon in the cell header! This chart maps domain name to the number of IP address behind those domains scan-by-scan.

alias VegaLite, as: Vl

[title: %{text: "Domain Name → IPs (Over Time)", font: "sans-serif"}]
|> Vl.new()
|> Vl.data_from_values(history)
|> Vl.mark(:circle, tooltip: true)
|> Vl.encode_field(:x, "scan",
  type: :ordinal,
  title: "Enumeration Index",
  sort: :descending
)
|> Vl.encode_field(:y, "name",
  type: :ordinal,
  title: "Domain Name"
)
|> Vl.encode_field(:size, "host",
  title: "Host Count",
  type: :ordinal,
  aggregate: :count,
  legend: [orient: "bottom"]
)
|> Vl.encode_field(:color, "host", title: "Host Count", type: :ordinal, aggregate: :count)