If your RAG system hallucinates or returns irrelevant context, the problem is almost always chunking — not the embedding model, not the LLM. Here's what I've found works in practice.
- Too small → chunks lack context, retrieval returns fragments the model can't reason over.
- Too big → retrieval is imprecise, you blow through the context window, and the model gets lost in noise.
- Sweet spot for most prose: 300–500 tokens per chunk, with 50–100 tokens of overlap.
Use a tokenizer to count, not len(text.split()). Tokens ≠ words.
Split every N tokens. Overlap ~15–20% so sentences aren't cut in half.
Use when: the corpus is uniform prose (articles, docs, transcripts) and you need something that just works.
Don't use when: the source has strong structure (code, markdown, tables) — you'll destroy it.
Split on paragraph → sentence → word, only going deeper when the chunk is still too large. LangChain's RecursiveCharacterTextSplitter is the canonical implementation.
Use when: you want decent results on mixed-format text without thinking hard.
Watch for: it still breaks code blocks and tables. Strip or special-case those first.
Embed every sentence. Walk through them and start a new chunk whenever the cosine similarity between consecutive sentences drops below a threshold (a topic shift).
Use when: retrieval quality is mission-critical and you have the compute budget to pre-embed every sentence.
Tradeoff: 5–20× more expensive to build the index. Marginal wins on long, topically-coherent documents; big wins on mixed-topic documents like meeting transcripts.
Respect the document's own boundaries:
- Markdown → split on
#,##,###headers. Keep the heading path in the chunk metadata. - Code → split by function or class using a tree-sitter or AST parser, never by line count.
- HTML → split on
<section>,<article>,<h1–h6>. - PDFs with headings → parse TOC first, use sections as chunk boundaries.
Use when: your source has real structure. This is the single biggest win for technical docs.
Index small chunks (for precise retrieval) but return the parent chunk (for context) to the LLM.
- Split into small chunks (~150 tokens) → embed these.
- Also store a mapping to the larger parent chunk (~1000 tokens).
- At query time, retrieve using the small embeddings.
- Fetch the parents and pass those to the LLM.
This decouples "what matches the query" from "what the LLM needs to answer."
Use when: precision and context are both important (most real-world RAG).
Before embedding each chunk, prepend a short model-generated description of what the chunk is about in the context of the whole document.
Chunk: "Revenue grew 12% year over year."
Contextualized: "This chunk is from Acme Corp's Q3 2024 earnings report.
Revenue grew 12% year over year."
Then embed the contextualized version. Retrieval quality jumps significantly because chunks now carry their own context.
Use when: you have a large, diverse corpus and can afford a one-time preprocessing pass.
- Start with recursive character splitting + 400 tokens + 60 overlap. It's the 80/20.
- Evaluate before tuning. Build a 20-question eval set. Without it, you're guessing.
- Inspect failing retrievals. When the system fails, look at what came back. The answer is almost always visible.
- Preserve metadata. Source URL, section heading, and document date should ride with every chunk. Your LLM will use them.
- Hybrid search beats pure vector. Combine BM25 (keyword) with dense retrieval using Reciprocal Rank Fusion. Often +10–20% recall for free.
- Rerank the top-K. Retrieve 20, rerank with a cross-encoder (e.g.
bge-reranker-large), keep top 5. Huge precision boost.
- Chunks that start mid-sentence
- Tables split across chunks
- Code blocks broken at random lines
- Headers separated from their content
- Any chunk longer than your model's effective attention window (~8k tokens is a practical ceiling even on 200k-context models)
Fix these before touching anything else.