Google Traffic Analysis - Marimo notebook

mo_google_traffic_analysis.py

import marimo

__generated_with = "0.24.0"
app = marimo.App(width="medium")


@app.cell
def _():
    import marimo as mo

    return (mo,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    # How Google Sees Your Site

    ## With Its Many Little Artificial Googly Eyes

    I keep stumbling over different Google bots in my logs. So today I'm looking at each and every Googly eye that takes a look at my site. Because when you stare at me, I'll stare back.

    You have the right and the means to define what Google is allowed to do on your site, whether the Google traffic you face is extensive or you'd like to opt-out of AI training. Sadly, using [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) is not enough, Google admits that some of their bots [will not respect](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#google-safety) the rules.

    Previous articles were about [TLS fingerprinting JA3/JA4](https://blog.miloslavhomer.cz/ja3ja4-tls-client-fingerprinting/) and how to deploy it on [HAProxy](https://blog.miloslavhomer.cz/deploying-ja4/). In this article/[marimo](https://marimo.io/) notebook ([full source](https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688)) I'll use the HAProxy logs containing IP addresses (plus [ASNs](https://blog.miloslavhomer.cz/asn-check/)), user agents and JA4 hashes to examine Google traffic. We'll see a couple of Google bots that are not mentioned in their docs.

    Let's get into it.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ![This is how I see Google in this article. The rest of the article is how Google sees me.](https://blog.miloslavhomer.cz/media/images/googly_eyes.max-3000x5000.png)
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    This article doubles as a [marimo](https://marimo.io/) notebook (of course, it's not modifiable and executable, that's a security nightmare to setup). Full [source here](https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688). So to kick things off, I'll do some arbitrary global imports. There are a couple dependencies like my own asn-check ([source](https://github.com/ArcHound/asn-check), [article](https://blog.miloslavhomer.cz/asn-check/)) and [requests](https://pypi.org/project/requests/).
    """)
    return


@app.cell
def _():
    import csv
    from ipaddress import ip_address, IPv4Network, IPv6Network, IPv4Address, IPv6Address
    from asn_check import ASNChecker
    from asn_check.ip_binary_tree import IPTree
    from typing import Union
    import requests
    import dataclasses
    from enum import Enum
    import re
    from collections import defaultdict

    return (
        ASNChecker,
        IPv4Address,
        IPv6Address,
        Union,
        csv,
        dataclasses,
        ip_address,
    )


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Getting Data: HAProxy Logs of This Blog

    We need logs. First, you have to setup HAProxy (checkout [the previous part](https://blog.miloslavhomer.cz/deploying-ja4/)). Then you have to parse the logs into something more sane, e.g. [like this](https://gist.github.com/ArcHound/cdecf272b3d07e402b168b0ffdf6c331). That "format" optimizes lookups, so I've forced it to CSV for easier processing. [Here's the CSV file](https://blog.miloslavhomer.cz/documents/1/stats.csv).

    Here's a sample of what we're working with.
    """)
    return


@app.cell
def _(csv):
    with open('haproxy_logs/stats.csv') as f:
        cr = csv.DictReader(f)
        data = [row for row in cr]

    data[0]
    return (data,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Generally, the signals to look for are IP addresses, User-Agents and the overall traffic. My new signal here is JA4, which I hope will give us some clarity on classifying these bots.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### ASN Checks - IP Trees

    Another useful signal is ASN ([Autonomous System Number](https://en.wikipedia.org/wiki/Autonomous_system_(Internet))). If IP is an address, then ASN is a city or a ZIP code. I have my [own lib](https://blog.miloslavhomer.cz/asn-check/) to get these numbers, so let's fetch them. Initialization is done in a separate cell, it takes some time (~4min) to build the trees.
    """)
    return


@app.cell
def _(ASNChecker):
    asn_checker = ASNChecker()
    return (asn_checker,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    The trick is organizing IP addresses to binary trees, where the prefix is known and suffix is variable. This plays really well with the [CIDR](https://en.wikipedia.org/wiki/Subnet) notation, read more in [my ASN Check article](https://blog.miloslavhomer.cz/asn-check/#ips-and-binary-trees). Here's a quick "illustration":

    ```
    192.168.50.102    = 11000000.10101000.00110010.01100110
    192.168.50.102/24 = 11000000.10101000.00110010.????????
    ```

    I'll return to this concept once we have some more addresses and networks of importance.

    The benefit is that consequent searches are fast, so we can replay them as we'd like. And it's already paying off, since we're encountering trash even in the IP fields!
    """)
    return


@app.cell
def _(asn_checker, data, ip_address):
    for r in data:
        try:
            asn = asn_checker.search(ip_address(r["ip"]))
            r["asn"] = asn["asn"]
            r["as_name"] = asn["name"]
            r["country_code"] = asn["country_code"]
        except ValueError as e:
            print(r)
            print(e)

    data[0]
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    In the log collection, I'm preferring the `X-Forwarded-For` header for the IP address, but that creates the opportunity to feed trash into this headers in an attempt to break/exploit the parser of these data. Well, here I am, writing my parser around that.

    We see the `2a06:98c0:3600:` entry a couple of times. That's an IPv6 prefix, not a full address, so the lib is complaining rightfully. It belongs to [Cloudflare](https://www.abuseipdb.com/check/2a06:98c0:3600::103) and it should be a reverse proxy. Seems that some CLoudFlare customers are abusing this. I imagine you can order a reverse proxy to point it to a target you don't own. Then if you generate traffic to that proxy, the target gets hit. The key difference is that this IP is usually white-listed and therefore these attacks are hard to block. Internet is fun!
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Log Entry Dataclass

    Let's finish this stage of parsing by defining a log entry:
    """)
    return


@app.cell
def _(IPv4Address, IPv6Address, Union, dataclasses):
    @dataclasses.dataclass(frozen=True)
    class LogEntry:
        ja4: str
        user_agent: str
        ip: Union[IPv4Address, IPv6Address]
        count: int
        asn: str # not all ASNs might be numbers and we are not using any number operations on them.
        as_name: str
        country_code: str

    return (LogEntry,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    And let's also finish up the data clean-up by wrangling our data into this format. I employ some dirty tricks for brevity. The kwargs trick is especially useful with a dictionary -> dataclass transformation, but you have to conform to the format exactly.
    """)
    return


@app.cell
def _(IPv4Address, IPv6Address, LogEntry, data):
    log_entries = list()
    for entry in data:
        try:
            new_entry = dict(entry)
            new_entry["ip"] = IPv4Address(entry["ip"]) if ":" not in entry["ip"] else IPv6Address(entry["ip"])
            new_entry["count"] = int(entry["count"])
            log_entries.append(LogEntry(**new_entry))
        except Exception as e:
            print(f"Malformed entry {entry}, exception {e}")
    log_entries[0]
    return (log_entries,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Google Assets

    In this section, let's map what we know about Google assets to identify traffic from Google.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Declared Google Bots

    Google has [a lot of bots](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers). Let me manually extract the "robots.txt" versions of user-agents. That should be an immutable string that appears in all versions and flavors of that particular bot. Here:
    """)
    return


@app.cell
def _():
    google_bot_ids = [
        # https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
        "Googlebot",
        "Googlebot-Image",
        "Googlebot-Video",
        "Googlebot-News",
        "Storebot-Google",
        "Google-InspectionTool",
        "GoogleOther",
        "GoogleOther-Image",
        "GoogleOther-Video",
        "Google-CloudVertexBot",
        "Google-Extended",
        # https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
        "APIs-Google",
        "AdsBot-Google",
        "AdsBot-Google-Mobile",
        "Mediapartners-Google",
        "Google-Safety",
        "DuplexWeb-Google",
        "Google Favicon",
        "AdsBot-Google-Mobile-Apps",
        "googleweblight",
        # https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
        "Google-CWS",
        "FeedFetcher-Google",
        "Google-GeminiNotebook",
        "Google-NotebookLM",
        "Google-Agent",
        "GoogleMessages",
        "Google-Pinpoint",
        "GoogleProducer",
        "Google-Read-Aloud",
        "google-speakr",
        "Google-Site-Verification"
    ]
    assert all(["google" in x.lower() for x in google_bot_ids])
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Noticed something? Literally all of them have "google" in them in one way shape or form, so we can filter them out based on user-agents like so:
    """)
    return


@app.cell
def _(log_entries):
    google_ua_entries = [x for x in log_entries if 'google' in x.user_agent.lower()]
    google_ja4s = {x.ja4 for x in google_ua_entries}
    google_ja4_uas = {x:{y.user_agent for y in google_ua_entries if y.ja4==x} for x in google_ja4s}
    len(google_ua_entries)
    return (google_ua_entries,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    And if you ask why didn't I use that nice list to check for the google bots, I'll show you in a later section. Spoilers: there are more bots.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google ASNs

    Another view we can try is to take a look at ASNs owned by Google and the traffic coming from those IPs:
    """)
    return


@app.cell
def _(log_entries):
    google_asn_entries = [x for x in log_entries if 'goog' in x.as_name.lower() or 'alphabet' in x.as_name.lower()]
    google_asns = {(x.asn, x.as_name) for x in google_asn_entries}
    google_asns
    return google_asn_entries, google_asns


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Not much info is available online about the purpose of these ASNs. But let's try anyway.

    #### ASN 16591: Google Fiber

    [IPInfo details](https://ipinfo.io/AS16591)

    This one is clear - it's [Google's internet offering](https://gfiber.com/). If you ask me, Google has already way too much control over browsing, this is then the ultimate final step: your internet connection is fully owned by Google. Of course, they need an ASN for their fleet of routers. What's nice that this is an ISP type ASN that is treated differently than hosting type ASNs.

    #### ASN 396982: Google Cloud Platform

    [IPInfo details](https://ipinfo.io/AS396982)

    You can rent a piece of Google in their Google Cloud Platform offering. It's a cloud hyperscaler, but you know this. I expect a lot of trash coming from this ASN as people might abuse the free tiers.

    #### ASN 394089: GCP Enterprise User Traffic

    [IPInfo details](https://ipinfo.io/AS394089)

    This one I suspect is what Google can give you if you have assets in GCP plus a better contract. Otherwise I expect the same as in Google Cloud Platform ASN396982. As they say, if you have to ask, you can't afford it.

    #### ASN 15169: Google

    [IPInfo details](https://ipinfo.io/AS15169)

    That leaves us with this general ASN. I can only presume that this is the traffic from all the rest of Google Internal services. It's explicitly not traffic from GCP workloads and from users hooked up to Google Fiber ISP. I make the conclusion that this is "official Google traffic."

    I've found this [list of prefixes](https://www.gstatic.com/ipranges/cloud.json) by searching [in their docs](https://docs.cloud.google.com/vpc/docs/subnets#subnet-types). To take a break from python, here's the breakdown after I ran a sophisticated bash analytics pipeline:

    ```
    $ wget -O- https://www.gstatic.com/ipranges/cloud.json 2>/dev/null | \
    jq | grep 'ipv4Prefix' | cut -f2 -d':' | grep -o '"[^"]*"' | tr -d '"' |\
    sed 's?./.*$?9?' | asn-check |\
    cut -f3 -d',' | sort | uniq -c | grep -v 'name'

        909 GOOGLE-CLOUD-PLATFORM - Google LLC
         81 GOOGLE - Google LLC
          7 LEVEL3 - Level 3 Parent
    ```
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Google Traffic

    It's at this point when we can count the percentage of traffic coming from Google to see how it stacks against the rest of the internet. Note, that I have counts of requests in the LogEntry, now's the time to use them.

    I'll also exclude traffic from my monitoring tool (Uptime Kuma) as it would probably dominate the conversation.
    """)
    return


@app.cell
def _(google_asns, log_entries):
    no_monitor = [x for x in log_entries if x.user_agent != "Uptime-Kuma/1.23.15"]
    total_sum = sum([x.count for x in no_monitor])
    google_sums = {f"{x[0]}: {x[1]}":sum([y.count for y in log_entries if y.asn==x[0]]) for x in google_asns}
    percentages = {x:f"{google_sums[x]} -> {round(google_sums[x]*100/total_sum,2)}%" for x in google_sums}
    percentages
    return (total_sum,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Not gonna lie, I was hoping for a bigger number here so that I could have a great clickbait headline. But alas, seems that Google has it's bots under control. The 3% at GCP ASN might very well be google customers, not Google and still it isn't that much.

    Now let's take a look at who is trying to impersonate Google:
    """)
    return


@app.cell
def _(google_asns, log_entries, total_sum):
    google_as_numbers = [x[0] for x in google_asns]
    google_ua_non_google_asn = [x for x in log_entries if x.asn not in google_as_numbers and "google" in x.user_agent.lower()]
    imp_count = sum([x.count for x in google_ua_non_google_asn])
    f"{imp_count} -> {round(imp_count*100/total_sum,2)}%"
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Also a nothing burger. I have attempted science today. Sometimes the dramatic story just isn't there and now I have to think of another clickbaity headline.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Official Google Traffic

    Now let's take a look at user agents with "google" in them that also come from Google ASNs. That's the strongest signal we have for official Google traffic.
    """)
    return


@app.cell
def _(google_asn_entries, google_ua_entries):
    google_ua_set = {x for x in google_ua_entries}
    google_asn_set = {x for x in google_asn_entries}
    assert len(google_ua_set) - len(google_ua_entries) == 0 # based on log-collecting methods, this should be true
    assert len(google_asn_set) - len(google_asn_entries) == 0
    google_entries = google_ua_set & google_asn_set
    f"{len(google_ua_set)} entries have 'google' in them, {len(google_asn_set)} entries come from google ASNs, {len(google_entries)} have both."
    return (google_entries,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Let's break it down by user-agent:
    """)
    return


@app.cell
def _(google_entries):
    google_uas = {x.user_agent for x in google_entries}
    google_uas_map = {x:[y for y in google_entries if y.user_agent==x] for x in google_uas}
    google_uas_map
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    Ok, this is quite the list of various Google Bots, so let's try and find what are they up to. The order is, of course, arbitrary. Also note, that some of these bots come from GCP rather than pure Google LLC. While Google might use GCP ranges for official traffic, it's more likely one of GCP customers impersonating Google.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### GoogleBot, GoogleBot Image, GoogleBot Video

    By far the most common bot, this is the one that brings your pages into the google search results ([docs](https://developers.google.com/search/docs/crawling-indexing/googlebot)). Comes in various different flavors (desktop and mobile), I guess it wants to see if there's the same content for various user agents.

    The Image and Video variants are similar, but they feed the multi-media results into the respective sections of Google search.

    So if you want your pages to be searchable by THE search engine, you'll have to allow these bots.

    BEWARE! These are also the most commonly impersonated bots on the internet. Since everyone wants to be searchable by Google, admins generally allow these user agents and the attackers will happily impersonate them. Take a look at these two entries:

    ```
    "LogEntry(ja4='t13d190900_9dc949149365_97f8aa674fd9', user_agent='Googlebot-Image/1.0', ip=IPv4Address('34.181.217.51'), count=1, asn='396982', as_name='GOOGLE-CLOUD-PLATFORM - Google LLC', country_code='US')"
    "LogEntry(ja4='t13d181300_e8a523a41297_43ade6aba3df', user_agent='Googlebot-Image/1.0', ip=IPv4Address('66.249.74.4'), count=23, asn='15169', as_name='GOOGLE - Google LLC', country_code='US')"
    ```

    One comes from GCP, so it might be anyone. The other comes from Google LLC so it seems to be directly from Google. The JA4 hashes are also different, which is a hint that this is weird. We'll meet the `t13d190900_9dc949149365_97f8aa674fd9` hash later and I suspect this one is a spammer. So let's take a look at it:
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google Toolbar? Feed Fetcher? GoogleBot News? AdsBot? MediaPartners? IMPOSTORS!

    For completeness, here's a quick breakdown of the listed bots:

    - [Google Toolbar](https://en.wikipedia.org/wiki/Google_Toolbar): [discontinued](https://arstechnica.com/gadgets/2021/12/happy-21st-birthday-to-google-toolbar-which-inexplicably-still-exists/) (check the [Google graveyard](https://killedbygoogle.com/) fully, it's.. really something if you never seen it).
    - FeedFetcher: [legit google bot](https://developers.google.com/crawling/docs/crawlers-fetchers/feedfetcher) designed to fetch feeds using RSS.
    - GoogleBot News: [legit google bot](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#googlebot-news) similar to googlebot but providing info into the "news" segment of google search.
    - AdsBot: [legit google bot](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#adsbot). Used for ads, specifically checking google ads quality. That's how I found these impostors - I'm not running Google Ads at all.
    - MediaPartners: [legit google bot](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#adsense). Used for google AdSense to provide relevant ads (or something IDK).
    - Google Web Preview: Not mentioned in any google documentation I could find, the best I got is [this archived link](https://web.archive.org/web/20260317004559/https://usehall.com/agents/google-web-preview-bot).

    Also in this category there's a user agent `BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)`. I haven't found any WAP proxy service by google, but I found [one reference](https://user-agents.net/string/blackberry7520-4-0-0-profile-midp-2-0-configuration-cldc-1-1-up-browser-5-0-3-3-up-link-5-1-2-12-google-wap-proxy-1-0) for this user-agent.

    Well actually these all come from Google GCP ASN, so they might be impostors. And if you'd look at the JA4 hashes, this one `t13d190900_9dc949149365_97f8aa674fd9` is different from the usual `t13d181300_e8a523a41297_43ade6aba3df`.

    If I take even closer look, I see it in this batch:
    ```
    ... 35.245.141.163:47184 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; LGMS428) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} "GET /backups/db.sql HTTP/1.1"
    ... 35.245.141.163:47196 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36|} "GET /tmp/backup.sql HTTP/1.1"
    ... 35.245.141.163:47176 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 MicroMessenger/7.0.5(0x17000523) NetType/4G Language/zh_CN|} "GET /backup/dump.sql HTTP/1.1"
    ... 35.245.141.163:47210 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0|} "GET /tmp/dump.sql HTTP/1.1"
    ... 35.245.141.163:47222 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13|} "GET /exports/db.sql HTTP/1.1"
    ... 35.245.141.163:47188 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/4.0 (compatible; GoogleToolbar 4.0.1019.5266-big; Windows XP 5.1; MSIE 6.0.2900.2180)|} "GET /backups/dump.sql HTTP/1.1"
    ... 35.245.141.163:47248 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; EVA-L09) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} "GET /config.js HTTP/1.1"
    ... 35.245.141.163:47232 ... {t13d190900_9dc949149365_97f8aa674fd9|Peach/1.01 (Ubuntu 8.04 LTS; U; en)|} "GET /config.php HTTP/1.1"
    ... 35.245.141.163:47264 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64; en-US; rv:2.0b2pre) Gecko/20100712 Minefield/4.0b2pre|} "GET /config.yml HTTP/1.1"
    ... 35.245.141.163:47256 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36|} "GET /config.json HTTP/1.1"
    ... 35.245.141.163:47276 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 YaBrowser/17.3.0.1785 Yowser/2.5 Safari/537.36|} "GET /config.yaml HTTP/1.1"
    ```

    This is a textbook example of a random spray and pray vulnerability scan. So I conclude this is in fact not Google, but someone sending trash from GCP while rotating the user agents randomly.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google AppsScript

    There are a couple of Google AppsScript user agents, e.g. `Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd_1zdKXtniWEDkKiDKLGH6k2AjErCM)` that are related to the [Google Apps Script offering](https://developers.google.com/apps-script/reference/url-fetch).

    That means these are most likely not directly google initiated, but google-user initiated.

    I appreciate the inclusion of the script ID, but I didn't find a way to report a misbehaving script, so that's a minus from me.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google NotebookLM, Google Agent, Gemini Deep Research

    Google has a concept of "user initiated fetch" (see [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)). So in a sense these are user-initiated but coming from the Google Infra. I've observed these:

    - Google NotebookLM: [renamed to GeminiNotebook](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers#gemini-notebook) in August 2026, it's about fetching URLs provided as sources in those notebooks.
    - GoogleAgent-URLContext: Not mentioned explicitly, but related [to GoogleAgent](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers#google-agent). It's one of the [tools available](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/url-context) for the agents in the [Gemini Enterprise Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform).
    - Gemini-Deep-Research: Seems to me like a [user initiated action for deep research](https://gemini.google/overview/deep-research/). Then there are [the agent docs](https://docs.cloud.google.com/gemini-enterprise-agent-platform/agents/use-deep-research).

    In a sense, these are AI bots. But the stated intent is to gather context to execute agents and not to train new AI.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Googlebot AMPHTML

    So there's this user agent `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Google-AMPHTML)` which can be identified by the `Google-AMPHTML` string. Apparently, this is a bot used to identify Accelerated Mobile Pages (AMP).

    The issue is, I didn't find this one in the Google official documentation, nor did I find it in any google source when googling. Using reverse DNS on the [64.233.172.233](https://ipinfo.io/64.233.172.233?lookup_source=search-bar) IP I get `google-proxy-64-233-172-233.google.com` which also gives me nothing.

    Not cool google, not cool.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google Safety

    While it is recognized in [the docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#google-safety), there is only a general description of

    > The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.

    And I also love to see that:

    >  The Google-Safety user agent ignores robots.txt rules.

    So... if a link to my site somehow gets onto google property, they reserve the right to ignore my wishes for external crawlers to do what they need to do to ensure "safety" (whatever that means). A "Magic The Gathering" player in me recognizes the killer combo with the GoogleBot search spider. Yay.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google (yes, just Google)

    By far my favourite user agent on this list: `Google`. It's even better that it comes from the official Google IP range and not GCP. Like what do you want to know you silly admin? It's Google, that'll have to do.

    So I guess hello John Google, enjoy your stay.

    Looking at the reverse DNS for the [108.177.64.74](https://ipinfo.io/108.177.64.74?lookup_source=search-bar) address I get `rate-limited-proxy-108-177-64-74.google.com`. Surely, Google wouldn't re-route the traffic and anonymize the User Agent when it detects rate-limiting (which I have) to ignore the website wishes. No, Google would never.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google Other

    Well of course, the Other bot traffic. The bot traffic that is other. This other traffic is unlike any traffic that we've seen. There are apparently other use cases that are not covered by the previous bots and product portfolio. There are other things Google wants to do. Don't worry, every other thing ends well.

    The thing is, this one is actually recognized in the official documentation. By far my least favourite because it's so opaque. And keep in mind, most of these come from Google LLC ASN and not the GCP ASN, so it's "official". I think if you're such a big company like Google and you're setting multiple random bots my way, you should have the courtesy to explain your reasons. And not just "yea, we're looking at your site because we can, what are you going to do about it"?

    Somehow, [Cloudflare assigned the AI CRAWLER category](https://radar.cloudflare.com/bots/directory/google-other) to this bot... I don't know why, but I also don't know how to disprove this so... bear that in mind.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Undeclared Google Traffic

    Recall there are these entries that come from Google ASNs and yet don't have "google" mentioned in their user agent. Let's break it down by ASN:
    """)
    return


@app.cell
def _(google_asn_entries, google_asns, google_entries):
    non_google_google_asn_set = [x for x in google_asn_entries if x not in google_entries]
    sorted_non_google_google_asn = {x:[y for y in non_google_google_asn_set if y.asn==x[0]] for x in google_asns}
    sorted_non_google_google_asn
    return (sorted_non_google_google_asn,)


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### GCP Traffic (Both AS396982 and AS394089)

    As expected, most of the trash comes from GCP where anybody can run anything. By trash I mean things like:

    - Automated WordPress vulnerability Scanners (xmlrpc.php is my top 404),
    - Forced browsing attempts (trying things like `/.env`, `/backup.sql` and others),
    - Completely random and obtuse bots (probably a project for scraping that went nowhere)
    - etc..

    Maybe the most fun info here is seeing Claude Code in the Enterprise GCP tier, confirming their claims of partnership. But ask yourself again, why would Claude Code, primarily a desktop application live in a cloud server?

    I plan to do a proper analysis of that traffic, but as I was digging around this I've hit Google so many times that I'm starting here and excluding all of the weird Google quirks from the main article. Sorry for now and TBD.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google

    Not much to see here, it's a small provider and it provides somewhat sane traffic.

    There is one fun user-agent though:

    ```
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
    ```

    As per [this question](https://webmasters.stackexchange.com/questions/137914/spike-in-traffic-from-facebot-twitterbot-user-agent), this is a service of Apple that pre-fetches links for previews. I'd appreciate if Apple docs could confirm this, but it is what it is.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ### Google LLC Traffic

    There is some weird traffic coming also from Google LLC. Recall, these shouldn't be Google customers, but Google itself as we've covered the other cases previously. Let's group it by user-agents:
    """)
    return


@app.cell
def _(sorted_non_google_google_asn):
    uas_15169 = {x.user_agent for x in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")]}
    official_google_traffic_lol = {x:[y for y in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")] if y.user_agent==x] for x in uas_15169}
    {x: len(official_google_traffic_lol[x]) for x in official_google_traffic_lol}
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    For the first time in forever, I can call upon JA4 to group the rest of the entries:
    """)
    return


@app.cell
def _(sorted_non_google_google_asn):
    ja4s_15169 = {x.ja4 for x in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")]}
    official_google_traffic_ja4 = {x:[y for y in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")] if y.ja4==x] for x in ja4s_15169}
    uas_ja4s = {x:list({y.user_agent for y in official_google_traffic_ja4[x]}) for x in official_google_traffic_ja4}
    uas_ja4s_keys_sorted = sorted(list(uas_ja4s.keys()))
    for _k in uas_ja4s_keys_sorted:
        print(_k)
        for _ua in uas_ja4s[_k]:
            print(f"    {_ua}")
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    We can clearly see a couple of groups here. Even though the JA4 hashes are not matching exactly, the similarities are strong enough to predict some trends.

    First is the [Chrome privacy preserving prefetch proxy](https://developer.chrome.com/blog/private-prefetch-proxy) (CPPPP) that has a surprisingly unique JA4 hash of `t13d181300_e8a523a41297_43ade6aba3df`.

    Then we have the `t13d190900_9dc949149365_.*` group that seems to be linked to abuse.xmco.fr, some [French security scanner](https://www.xmco.fr/). While it's slightly surprising that this comes from the Google LLC ASN and not GCP ASN, the addresses are quite similar and I can imagine that this IP range was originally in the GCP. Or they did have some agreement with Google, IDK. It seems highly unlikely to me that Google would decide to impersonate this random French company in particular. Or it's just an impostor, like we saw above.

    We also have the `t13d5212h1_b262b3658495_.*` group which has `BuiltWith/1.4; rb.gy/xprgqj` in the User-Agent. They have a [site describing their crawler](https://builtwith.com/biup) and similarly, I think this is associated with Google LLC only because of some shuffle of IP addresses between Google LLC ASN and GCP ASN.

    The last group seems the most interesting to me. All user agents are covered by the `t13d1514h2_8daaf6152771_.*` fingerprint but you can see quite the variety of browsers and operating systems. While it is possible that all this hardware and software versions produce very similar fingerprints, my bet would be that Google is trying to see how the page behaves under different versions of Chrome for.... unknown reasons. UX? Legibility? General interest? I don't know.

    While I didn't consent to this level of scrutiny, I have to admit that it has literally no impact on the site operations, so it's all good in my book.
    """)
    return


@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    ## Concluding: All Googly Eyes on Me

    There are way too many bots looking at my page from Google. I really don't know why they need a bag of eyes staring at me, at this point it would be easier to just copy my page fully and then dissect it internally. They behave quite well though and it's not a problem for my site.

    While there's only one bot specifically marked for AI training (Google Other), there's nothing really preventing Google from grabbing the other bot outputs for these purposes. And as you can see, fully blocking Google is hard, even if you accept not being searchable by the most prominent search engine.

    As we saw, there are several good marks you can focus on if you want to block Google. These are:

    - Google ASNs: 15169 (Google), 16591 (Fiber ISP), 396982 (GCP), 394089 (GCP Enterprise),
    - User-agents with "google" in them,
    - t13d181300_e8a523a41297_43ade6aba3df JA4 Fingerprint (YMMV),
    - t13d1514h2_8daaf6152771_.*JA4 Fingerprint (YMMV).

    With that out of the way, I'll take a look at the rest of the traffic next time.
    """)
    return


if __name__ == "__main__":
    app.run()