By Guerin Green / Novel Cognition AI Strategist | Federal Courthouse AI Presenter
Part of the NovCog AI Practitioner Series
The AI content detection industry has latched onto two terms — burstiness and perplexity — without adequately explaining what they measure, why they matter, or why the "AI-proof your writing" cottage industry fundamentally misunderstands both concepts.
Perplexity is a measurement of how well a language model predicts a given text. Technically, it's the exponentiated average negative log-likelihood of the token sequence. Practically, it means: how surprised is the model by this text?
- Low perplexity = The model expected these tokens. The text follows patterns the model learned during training. Common phrases, standard syntax, predictable word choices.
- High perplexity = The model didn't expect these tokens. Unusual word choices, complex syntax, domain-specific jargon, creative phrasing, errors.
AI-generated text tends toward low perplexity because language models generate tokens they predict — by definition. The model produces the output it finds most probable, which means that same model (or a similar one) will find that output highly predictable.
Human writing has higher and more variable perplexity because humans make choices that aren't purely probabilistic. We use idioms incorrectly, coin phrases, reference obscure knowledge, write run-on sentences when we're excited, and deploy deliberate awkwardness for effect.
Burstiness measures how term frequency varies across a document. It comes from information retrieval research, originally describing how keywords cluster in documents rather than appearing at uniform intervals.
In the context of AI content detection:
- Bursty text = Sentence length, vocabulary complexity, and structural patterns vary significantly throughout the document. Some paragraphs are dense; others are sparse. Some sentences are 5 words; others are 40. This is characteristic of human writing.
- Uniform text = Sentence length, vocabulary, and structure remain consistent throughout. Each paragraph resembles every other paragraph. This is characteristic of AI-generated text.
Language models optimize for coherence, which produces uniformity. The temperature parameter adds randomness, but it's uniform randomness — not the structured variation that characterizes human cognitive patterns (focused attention on key points, casual treatment of transitions, emotional shifts in tone).
Current AI content detection tools (GPTZero, Originality.ai, Turnitin's AI detector) primarily measure:
- Per-sentence perplexity — Is each sentence predictable? Flagged if consistently low.
- Perplexity variance — Does predictability vary across the document? Flagged if uniform.
- Burstiness score — Does structural complexity vary? Flagged if uniform.
- Token probability distributions — Are word choices consistently the "top" prediction? Flagged if yes.
The detection threshold is essentially: Does this text look like what a language model would generate? Which means the detector is itself a language model asking: Would I have written this?
The market is flooded with advice: "Add personal anecdotes," "Use varied sentence lengths," "Include typos," "Write in first person." This advice treats burstiness and perplexity as checkboxes rather than emergent properties of how humans think.
The problem with gaming detection:
- Adding artificial variation to AI text creates detectable artifacts of its own. A document that's uniformly predictable except for randomly inserted complex sentences has a bimodal perplexity distribution — which is neither human nor standard AI. It's a third category: manipulated AI text.
- Detection models are trained on both clean AI text and manipulated AI text. The arms race favors detection because detectors need to identify one pattern; manipulators need to evade all patterns.
- The fundamental issue isn't the text — it's the token probability distribution. Human text samples from a different distribution than model text. Cosmetic modifications to model text don't change the underlying distribution.
For SEO practitioners and content strategists, the burstiness/perplexity framework matters for two reasons:
1. Content quality signals. Search engines don't use GPTZero, but they do evaluate content along similar statistical dimensions. Content that reads as formulaic — whether AI-generated or human-written — provides less informational value than content with genuine expertise signals. The Hidden State Drift framework suggests that Google's hidden states are increasingly sensitive to content that matches model-generated distributions.
2. Entity authority content. For entity authority building and accountability journalism, the content must be genuinely human-authored with real expertise. Not because of detection risk, but because the content serves a real audience and needs to demonstrate real authority. Formulaic AI content fails this test regardless of whether it's detected as AI.
The terms burstiness and perplexity describe real statistical properties of text. Understanding them — rather than trying to game them — is what separates practitioners from the "AI-proof your content" grifters.
For the full glossary of these and related terms, see the RAG Glossary.
- A Practitioner's Guide to LLMs — Start here
- Hidden State Drift
- Burstiness and Perplexity — You are here
- NovCog Brain: AI Memory System
- AI Glossary for Legal Professionals
- Complete Series Index
Burstiness & Perplexity Community | Hidden State Drift | Novel Cognition
#hiddenstatedrift #burstinessandperplexity