Chunking Strategies in RAG: How to Split Documents for Better Retrieval

1. Introduction

Ask any seasoned RAG engineer what silently makes or breaks a system, and they'll point to chunking. It's the seemingly mundane step of splitting long documents into smaller pieces before embedding and indexing them. Yet this single design decision ripples through everything downstream—embedding quality, retrieval precision, LLM comprehension, and even your infrastructure costs.

Get chunking right, and your RAG system surfaces exactly the right paragraph to answer a question. Get it wrong, and your vector database returns fragments that are either too small to make sense or too large to be precise, leaving the LLM to guess or hallucinate.

There is no universal "best" chunk size. The right strategy depends on your documents, your queries, and your users' expectations. This article explains the trade-offs behind every major chunking approach and gives you the mental models to choose intelligently for your production system.

2. Why Chunking Is Necessary

You might wonder: "Why can't I just embed the entire document as one vector?"

Consider a 50-page company handbook. Embedding it as a single 1536-dimensional vector collapses everything—the vacation policy, the code of conduct, the expense reporting rules—into one numerical fingerprint. When a user asks, "How many vacation days do I get?", the similarity search compares that query vector to the handbook vector. The result might be moderately similar because the word "vacation" appears somewhere, but the vector is so diluted by all the unrelated content that retrieval confidence is low. Even if it's retrieved, passing 50 pages into the LLM's context window is expensive and often exceeds token limits.

Chunking solves this by:

• Creating vectors that represent specific, coherent topics.
• Allowing the retrieval system to pinpoint the exact section relevant to the query.
• Keeping context windows manageable and cost-effective.

Without chunking, retrieval is approximate at best and useless at worst.

3. Where Chunking Fits in the RAG Pipeline

Chunking sits early in the offline ingestion phase, after cleaning and before embedding:

Decisions made at the chunking stage—chunk size, overlap, structure awareness—propagate through the rest of the pipeline. The embedding model sees only the chunks it's given; the vector database indexes only those embeddings; the LLM reads only the retrieved chunks. Chunking is the foundation on which the rest of the retrieval edifice stands.

4. What Makes a Good Chunk?

A good chunk has several properties:

• Self-contained meaning: The chunk should be understandable on its own, without requiring surrounding paragraphs for basic comprehension.
• Sufficient context: It should contain enough information to answer potential questions, not just a single sentence fragment.
• Limited size: It must fit within the embedding model's maximum token limit (often 512 or 8192 tokens) and be small enough for precise retrieval.
• Coherent topic: The chunk should discuss one main idea, not mix unrelated concepts.

Think of a well-written index card: it captures one concept clearly, with enough detail to be useful, but not so much that it becomes a book chapter.

5. Fixed-Size Chunking

The simplest approach: split every N tokens, regardless of content.

How it works: A text splitter counts tokens (or characters) and cuts at exactly the Nth token. If you set N=512, every chunk is 512 tokens long (except the last one, which may be shorter).

Advantages:

• Trivially simple to implement.
• Predictable chunk sizes simplify token budgeting.
• Fast, with minimal processing overhead.

Disadvantages:

• Sentence boundaries are ignored—a chunk may cut a sentence in half.
• Topics can be fragmented: the end of one section and the start of the next get mushed together.
• Retrieval may surface chunks that start or end mid-thought, confusing the LLM.

Fixed-size chunking is a reasonable baseline for simple, homogeneous documents (like a blog post feed) but quickly shows its limits with structured or technical content.

6. Recursive Chunking

Recursive chunking is the workhorse of many production RAG systems. Instead of blindly cutting at a fixed token count, it tries to split on natural boundaries first, only resorting to smaller separators when necessary.

How it works: You define a hierarchy of separators—for example:

1. Double newline (\n\n) — paragraph breaks.
2. Single newline (\n) — line breaks.
3. Period (。) — sentence endings.
4. Space ( ) — word boundaries.
5. Character — last resort.

The splitter recursively attempts to divide the text using the most meaningful separator, checking if the resulting chunk fits within the size limit. If a chunk is still too large, it moves to the next separator down the hierarchy.

Advantages:

• Respects natural document structure (paragraphs, sentences).
• Chunks feel more coherent and readable.
• Widely supported in libraries like LangChain and LlamaIndex.

Disadvantages:

• Slightly more complex than fixed-size.
• Still doesn't understand semantic topic boundaries; a new paragraph might continue the same thought.

Recursive chunking is a solid default for most text documents and provides a noticeable quality improvement over fixed-size splitting with minimal extra complexity.

7. Semantic Chunking

Semantic chunking uses an embedding model (often a smaller, faster one) to decide where to split. Instead of relying on token counts or newlines, it looks for shifts in meaning.

How it works: The text is split into sentences or small segments. The embedder computes vectors for each segment. The chunker then walks through the segments sequentially, and when the cosine similarity between consecutive segments drops below a threshold, it inserts a chunk boundary. The idea: a big semantic shift means a new topic has started.

Advantages:

• Chunks align with actual topic changes.
• Retrieval becomes more precise because each chunk covers one coherent idea.
• Works well for prose-heavy documents like reports, articles, and books.

Disadvantages:

• Significantly slower preprocessing (requires many embedding calls).
• More complex to implement and tune (the similarity threshold is a hyperparameter).
• Can be sensitive to the embedding model used for chunking—it may not match the one used for retrieval.

Semantic chunking is powerful but not always necessary. It shines when document quality and retrieval precision are paramount, such as in legal or medical applications.

8. Structure-Aware Chunking

Many documents carry explicit structure: Markdown headings, HTML tags, PDF bookmarks, or JSON schemas. Structure-aware chunking exploits this.

How it works: The parser understands the document format and splits on structural elements. For example:

• Markdown: Split on # headings, preserving the hierarchy.
• HTML: Split on <section>, <article>, or heading tags.
• PDF: Use the built-in table of contents or detect whitespace patterns.
• Code: Split on function or class boundaries using AST parsers.

Advantages:

• Chunks naturally align with how the document was written.
• Headings and sections provide built-in context—you can prepend the section title to each chunk.
• Ideal for technical documentation, API docs, and knowledge bases.

Disadvantages:

• Requires format-specific parsers.
• Some documents have poor or inconsistent structure (e.g., a PDF with no headings).
• Very long sections still need sub-chunking.

For structured content, this is often the best approach. A common pattern combines structure-aware splitting with recursive chunking: split on headings, then recursively sub-chunk any section that exceeds the token limit.

9. Sliding Window Chunking (Chunk Overlap)

No chunking strategy is perfect. A vital sentence can still land right on a boundary. Chunk overlap mitigates this by having consecutive chunks share a portion of their content.

Example:

• Chunk 1: tokens 1–512
• Chunk 2: tokens 462–974 (10% overlap)

The overlapping region appears in both chunks, so that a question targeting the boundary content can still match at least one chunk fully containing the relevant information.

Advantages:

• Prevents information loss at boundaries.
• Increases retrieval robustness, especially for fixed-size chunking.

Disadvantages:

• Duplicated content across chunks inflates the vector database size.
• Embedding and storage costs increase proportionally to overlap.
• The same fact may appear in multiple retrieved chunks, wasting context window tokens.

Typical overlap values range from 10% to 20%. Too little overlap doesn't solve the boundary problem; too much wastes resources.

10. Chunk Size Trade-offs

Factor	Small Chunk (128–256 tokens)	Large Chunk (1024–2048 tokens)
Precision	High—exact passage matched	Lower—more content, less targeted
Recall	May miss surrounding context	Better at capturing complete answers
Context richness	Low—may lack necessary background	High—includes surrounding details
Embedding cost	More chunks → more embeddings	Fewer chunks → lower cost
Retrieval quality	Can be too narrow, missing the full answer	Can be too broad, diluting relevance
Context window usage	More efficient—less irrelevant text	May waste tokens on irrelevant portions

The sweet spot for most use cases is between 256 and 1024 tokens, with 512 tokens being a common starting point. But this depends heavily on the query patterns. If users ask short, factoid-style questions ("What is the return policy?"), smaller chunks shine. If they ask complex, multi-part questions ("Explain the evolution of our pricing model over the last three years"), larger chunks with more context are better.

11. Choosing Chunk Overlap

Overlap is your insurance policy against borderline cuts.

Overlap	Pros	Cons
0%	No duplication, lowest cost	High risk of boundary information loss
10%	Good balance for most use cases	Modest increase in storage
20%	Very robust retrieval	Noticeable duplication, higher cost
30%+	Maximum continuity	Significant bloat, diminishing returns

A 10–15% overlap is a pragmatic default. Monitor retrieval quality over time; if you see queries that consistently fail because the answer sits on a boundary, increase overlap for those document types specifically.

12. Chunking for Different Document Types

Technical Documentation

Use structure-aware + recursive chunking. Split on headers (##, ###), preserve the heading hierarchy by prepending section titles, and sub-chunk large sections recursively. This keeps procedures and API references coherent.

APIs

For OpenAPI/Swagger specs, chunk per endpoint: the path, method, parameters, and response schema form one chunk. This allows developers to ask "how do I authenticate?" and retrieve exactly the auth endpoint chunk.

Source Code

Use AST-aware chunking. Split on functions, classes, or methods. Never break inside a function body. Include the function signature and docstring in each chunk. A 50-line function with a clear docstring makes a perfect retrieval unit.

PDFs

PDFs are challenging because their structure is often lost. Extract text with a tool that preserves reading order. Use recursive chunking as a baseline; apply semantic chunking if the document is prose-heavy. Be especially careful with multi-column layouts and tables.

Knowledge Bases / FAQs

These are often already structured as Q&A pairs or short articles. Treat each entry as its own chunk. Don't over-split—a FAQ answer of 200 words should remain one chunk.

13. Chunking and Embeddings

The quality of your embeddings is a direct function of the quality of your chunks. Embedding models are trained to encode the meaning of a passage. If you feed them a chunk that mixes three unrelated topics, the resulting vector is a confused average—not clearly similar to anything.

Better chunks → Better embeddings → Better similarity search → Better RAG responses.

If you upgrade your embedding model but don't fix your chunking, you'll see marginal gains at best. The two must be tuned together. Experiment with chunking strategies before concluding an embedding model is underperforming.

Embedding model selection: See our Embedding Models for RAG Systems article for guidance on choosing the right model for your chunks.

14. Chunking and Context Window

Every retrieved chunk consumes precious context window tokens. Poor chunking wastes this budget:

• Chunks too large: You pass thousands of irrelevant tokens to the LLM, increasing cost and latency while potentially burying the answer.
• Chunks too small: You need to retrieve many chunks to assemble a complete answer, each with its own overhead.

Good chunking maximizes information density—the amount of useful content per token. This lets you retrieve fewer chunks with higher confidence, staying comfortably within context limits.

Context management: See our Context Window article for strategies on handling token limits.

15. Common Chunking Mistakes

• Chunks too large: "I'll just make 4000-token chunks so I don't miss anything." Result: retrieval is no better than embedding the whole document.
• Chunks too small: 50-token chunks. Retrieval returns sentence fragments that the LLM can't interpret without surrounding context.
• Splitting tables: A table split in half is unreadable. Detect tables and keep them intact.
• Breaking code blocks: Never split a function in the middle. The LLM will receive syntactically broken code.
• Ignoring headings: A chunk about "Return Policy" without the context that it belongs to "International Orders" loses critical scoping information. Prepend headings.
• Excessive overlap: 50% overlap doubles your vector database size and cost with minimal retrieval benefit.

16. Best Practices for Production RAG

• Start with recursive chunking at 512 tokens with 10% overlap. It's a proven, robust baseline.
• Preserve document structure wherever possible. Use heading-aware splitters for structured docs.
• Evaluate chunk size empirically. Run retrieval quality benchmarks (e.g., recall@k) on a labeled test set with different chunk sizes.
• Use metadata. Tag each chunk with its source document, section, page number, and heading. This enables metadata filtering and citation.
• Monitor retrieval quality in production. Track metrics like average number of chunks retrieved, context relevance scores (using an LLM as a judge), and user feedback.
• Benchmark different strategies per document type. Technical docs, code, and prose all benefit from different approaches. Don't force one strategy on everything.

17. Relationship to Other RAG Components

Chunking doesn't exist in isolation. It's deeply connected:

• Embeddings quality depends on chunk coherence.
• Vector Database storage size and index performance are affected by chunk count and size.
• Retrieval precision is directly tied to chunk granularity.
• Context Retrieval and Reranking work with the chunks provided.
• LLM Inference sees only the retrieved chunks—if they're poor, the answer will be too.

Chunking is the architectural decision that echoes through the entire system.

18. Key Takeaways

• Chunking is foundational to RAG quality. It's not a preprocessing afterthought; it's a core design decision.
• There is no universal chunk size. The right size depends on document type, query patterns, and the embedding model. Start with 512 tokens and iterate.
• Document structure matters. Exploit headings, sections, and formatting whenever possible.
• Recursive chunking is a strong default. It balances simplicity with structural awareness.
• Semantic chunking offers precision at a cost. Use it when retrieval accuracy is paramount.
• Chunk overlap (10–20%) prevents information loss at boundaries at a modest storage cost.
• Test and monitor. Run benchmarks with your actual queries and documents. Chunking is an empirical choice, not a theoretical one.
• Good chunking leads to better embeddings, better retrieval, and better LLM responses. It's the quiet foundation of every successful RAG system.