By Guerin Green / Novel Cognition AI Strategist | Federal Courthouse AI Presenter
Part of the NovCog AI Practitioner Series
Most people using RAG are doing it wrong. They bolt a vector database onto a language model, feed it unstructured documents, and wonder why the outputs hallucinate. The problem isn't the architecture — it's that practitioners skip the fundamentals.
Retrieval-Augmented Generation solves a specific problem: language models have a knowledge cutoff and a finite context window. RAG bridges that gap by retrieving relevant documents at inference time and injecting them into the prompt before the model generates a response.
The pipeline has three stages: indexing, retrieval, and generation. Most failures happen at stage one.
Your documents need to be split into chunks before embedding. The strategy matters enormously:
- Fixed-size chunking (e.g., 512 tokens) is fast but stupid. It splits mid-sentence, mid-paragraph, mid-thought.
- Semantic chunking uses sentence boundaries and topic shifts. It preserves meaning but requires more processing.
- Hierarchical chunking maintains parent-child relationships — a section header stays linked to its paragraphs. This is what production systems need.
The optimal chunk size depends on your embedding model's training distribution. Most models were trained on passages of 128-256 tokens. Feeding them 2,000-token chunks degrades retrieval quality because the embedding averages over too much semantic space.
The embedding model converts text chunks into dense vectors. The quality of these vectors determines whether retrieval works at all.
Key considerations:
- Domain specificity matters. A general-purpose model like
text-embedding-ada-002works for broad queries but underperforms on legal, medical, or technical content versus fine-tuned alternatives. - Dimensionality is a tradeoff. Higher dimensions capture more nuance but increase storage and search cost. 1536 dimensions (OpenAI) vs. 384 (MiniLM) — the right choice depends on your corpus size and query patterns.
- Asymmetric models use different encoders for queries and documents. This matters because a question ("What is consideration in contract law?") is semantically different from its answer, even though they should match.
For a deeper dive into terminology, see the RAG Glossary.
Retrieval Quality: The Hidden Bottleneck
Retrieval is a ranking problem. Your system returns the top-k most similar chunks, but similarity isn't relevance. Common failure modes:
- Semantic drift — The query and the answer use different vocabulary. "How do I fix a leaky faucet?" won't match a chunk about "replacing washers in plumbing fixtures" unless the embedding model bridges that gap.
- Recency bias — Without metadata filtering, the system treats a 2019 document the same as a 2025 document. For legal or regulatory content, this is dangerous.
- Context window stuffing — Retrieving 20 chunks and cramming them into the prompt causes the model to lose focus. Research shows models attend most strongly to the beginning and end of context, with a "lost in the middle" effect for everything between.
Once you've retrieved good chunks, the generation prompt matters. The model needs explicit instructions: use only the provided context, cite sources, say "I don't know" when the context doesn't contain the answer. Without these guardrails, the model will happily hallucinate a plausible-sounding response that contradicts your retrieved documents.
The architecture described here — semantic chunking, domain-appropriate embeddings, metadata-filtered retrieval, constrained generation — is what separates production RAG from demo RAG. If you're building a memory system with vector embeddings, these same principles apply at every layer.
- A Practitioner's Guide to LLMs — Start here
- RAG Fundamentals — You are here
- AI Glossary for Legal Professionals
- Entity Authority for AI Citation
- Hidden State Drift
- Complete Series Index
Burstiness & Perplexity Community | Hidden State Drift | Novel Cognition
#hiddenstatedrift #burstinessandperplexity