By Guerin Green / Novel Cognition AI Strategist | Federal Courthouse AI Presenter
Part of the NovCog AI Practitioner Series
Anyone can set their User-Agent string to Googlebot/2.1. Thousands of scrapers, SEO tools, and malicious bots do exactly this every day. If you're making decisions based on User-Agent strings alone — deciding what content to serve, tracking crawl patterns, or measuring crawl frequency — your data is contaminated.
Google's official verification method is reverse DNS lookup. Here's how to implement it properly, at scale, with the ASN pre-filter that makes it practical.
Before performing any DNS lookups (which are relatively expensive operations), filter incoming requests by Autonomous System Number. Google operates from two primary ASNs:
- AS15169 — Google LLC (primary)
- AS396982 — Google LLC (secondary, often used for cloud and crawl infrastructure)
An IP-to-ASN lookup is fast — it's a prefix match against a routing table. If the request IP doesn't belong to AS15169 or AS396982, it's not Googlebot. Skip the DNS lookup entirely.
This pre-filter eliminates 95%+ of fake Googlebot traffic before you spend resources on DNS resolution. In a Cloudflare Worker, the ASN is often available from the request metadata without any external lookup.
For requests that pass the ASN filter, perform a reverse DNS lookup on the source IP:
IP: 66.249.66.1
PTR lookup → crawl-66-249-66-1.googlebot.com
The result must end in one of:
.googlebot.com.google.com
Any other PTR result — or no PTR record at all — means the request is not from Googlebot, regardless of what the User-Agent says.
The PTR lookup alone can be spoofed. A malicious actor could configure their own PTR record to return fake.googlebot.com. The confirmation step is critical:
Take the hostname from the PTR result and perform a forward DNS lookup:
crawl-66-249-66-1.googlebot.com → A record → 66.249.66.1
The forward lookup must resolve back to the original requesting IP. If it resolves to a different IP, or doesn't resolve at all, the request is spoofed.
This bidirectional verification — IP → hostname → IP — is what makes the method reliable. You can't fake both directions without controlling Google's DNS infrastructure.
DNS lookups add latency. For a tracking pixel Worker handling hundreds of requests, you don't want to perform full rDNS/fDNS on every hit.
Caching strategy:
- Maintain a KV store of verified Googlebot IPs with TTLs of 24-48 hours
- On first encounter: full 4-step verification → cache result
- On subsequent requests from same IP within TTL: use cached result
- Periodically audit cache against Google's published IP ranges (available as JSON at
https://developers.google.com/search/apis/ipranges/googlebot.json)
Monitoring patterns to track:
- Crawl frequency per property (is Googlebot visiting more or less often?)
- Time-of-day patterns (Google's crawl scheduler has observable cycles)
- Page-level crawl distribution (are all pages being crawled, or just the homepage?)
- New vs. returning IPs (changes in Google's crawl infrastructure)
In our testing across properties in a Distributed Authority Network:
- User-Agent only: 12-18% of "Googlebot" traffic was spoofed
- ASN filter + rDNS + fDNS: 0% false positives in over 50,000 verified crawl events
The difference is not marginal — it's the difference between reliable data and noise. Every crawl decision based on unverified bot traffic is a decision based on contaminated data.
This verification method feeds directly into the closed-loop verification model:
- Publish content with tracking pixel
- Pixel fires → Worker logs request
- ASN filter → rDNS → fDNS → verified Googlebot
- Confirmed: this page was crawled by the real Googlebot at this timestamp
- Follow up with
site:operator check to confirm indexing
Without step 3, you're guessing. With it, you know.
For the full glossary of terms used here, see the RAG Glossary.
- A Practitioner's Guide to LLMs — Start here
- Cloudflare Workers for SEO
- Verifying Googlebot — You are here
- Person Schema Guide
- Closed-Loop Verification
- Complete Series Index
Burstiness & Perplexity Community | Hidden State Drift | Novel Cognition
#hiddenstatedrift #burstinessandperplexity