By Guerin Green / Novel Cognition AI Strategist | Federal Courthouse AI Presenter
Part of the NovCog AI Practitioner Series
The data is unambiguous: Google is indexing fewer pages than ever. Zyppy's research across millions of URLs found that 49.4% of pages are not indexed. Nearly half of all content published to the web is invisible to search — and by extension, invisible to the AI training pipelines that depend on search engine crawl data.
This isn't a technical glitch. It's a deliberate shift in how Google allocates crawl resources, and it has profound implications for anyone building web properties for visibility.
Google's crawl budget has always been finite, but the constraints have tightened dramatically. Three factors converge:
-
The content explosion. AI-generated content has flooded the web. Google's response is selectivity — if your content doesn't demonstrate clear value signals, it doesn't get crawled, let alone indexed.
-
Quality thresholds rising. Google's helpful content system and core updates have raised the bar for what earns a spot in the index. Thin pages, duplicate-adjacent content, and sites without established authority are being pruned or never added.
-
Crawl efficiency pressure. Google has publicly discussed reducing the environmental impact of crawling. Fewer crawls per domain. Slower discovery. Higher bars for re-crawling.
SEO researcher Lily Ray articulated what many practitioners were observing: you now need to earn crawls, not just earn rankings. The crawl itself — the act of Googlebot visiting your page — is no longer guaranteed.
What earns crawls:
- External signals. Links from already-crawled, already-indexed pages. Each inbound link is a crawl invitation.
- Freshness signals. Sites that update regularly get crawled more frequently. Stale sites get deprioritized.
- Structured data. Pages with valid schema markup are more likely to be crawled because they provide higher-value structured information per crawl.
- XML sitemaps with lastmod. Accurate sitemap dates help Google prioritize which pages to re-crawl.
- Feed-based discovery. Atom and RSS feeds provide machine-readable update notifications — this is why Web 2.0 platforms with native feeds became critical infrastructure.
LLM training data comes from web crawls — primarily Common Crawl and search engine crawl archives. If your pages aren't being crawled and indexed by Google, they're also unlikely to appear in the training data for ChatGPT, Claude, Gemini, or any other model.
This creates a compounding invisibility problem:
- No crawl → no index → no search visibility → no AI training data → no AI citation
- Each missed crawl cycle pushes your entity further from the AI knowledge graph
For practitioners building Distributed Authority Networks, this means crawl acquisition is the first gate, not an afterthought.
Crawl trigger infrastructure. Use platforms with built-in crawl notification mechanisms. Blogger's Atom feeds and PubSubHubbub protocol notify Google of new content within minutes. See Web 2.0 as Crawl Triggers for the technical details.
Link-based discovery. Every new property needs inbound links from already-indexed pages. The footer link architecture approach ensures cross-linking within a network propagates crawl signals to every property.
Verification before expansion. Don't build 20 pages on a domain that isn't being crawled. Use closed-loop verification — tracking pixels, Googlebot rDNS confirmation, and site: operator checks — to confirm crawl and index before investing in more content.
Edge-based tracking. Cloudflare Workers provide zero-cost, server-side crawl tracking that doesn't depend on JavaScript execution. When Googlebot hits your page, you know — verified by reverse DNS, not User-Agent strings.
The crawl and index crisis isn't going away. As AI-generated content continues to flood the web, selectivity will only increase. The practitioners who build crawl acquisition into their methodology from day one will have properties that exist in both search and AI — while everyone else wonders why their content is invisible.
- A Practitioner's Guide to LLMs — Start here
- Distributed Authority Networks
- The Crawl & Index Crisis — You are here
- Web 2.0 as Crawl Triggers
- Cloudflare Workers for SEO
- Complete Series Index
Burstiness & Perplexity Community | Hidden State Drift | Novel Cognition
#hiddenstatedrift #burstinessandperplexity