Chunking

Phase 9 · Document 02 · RAG Prev: 01 — Document Ingestion · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Chunking — how you split documents into retrievable pieces — is the highest-leverage, most-underestimated knob in RAG. The chunk is the unit of retrieval: it's what gets embedded, searched, ranked, and injected. Chunk badly and you cap quality before retrieval even runs — too small and a chunk lacks the context to answer; too large and it dilutes the embedding and stuffs irrelevant text into the prompt; split mid-table or mid-sentence and the meaning is destroyed. Teams routinely chase better embedding models or rerankers when the real fix was better chunks. Because chunking is cheap to change and re-ingest, it's where you get the biggest quality gains for the least effort — if you understand the trade-offs.

2. Core Concept

Plain-English primer: the unit of retrieval

After ingestion produces clean text (01), you split it into chunks — passages of a few hundred tokens — because (a) you retrieve pieces, not whole documents (a 50-page PDF can't go in the prompt), and (b) embeddings represent a bounded span of meaning well but blur if a chunk spans many topics (03). Each chunk becomes one embedded, searchable record (carrying its document's metadata, 01).

The chunk must be:

• small enough to be retrieved precisely (mostly relevant text, sharp embedding),
• large enough to be self-contained (enough context to answer), and
• semantically coherent (don't split mid-sentence, mid-table, mid-thought).

The size/overlap trade-off (the core tension)

chunk too SMALL → sharp retrieval but FRAGMENTS context → partial/missing answers (recall ↓)
chunk too LARGE → full context but DILUTED embedding + irrelevant text in prompt (precision ↓, cost ↑)

There's no universal size; it depends on content and the embedding model's sweet spot. Common starting point: ~256–512 tokens with ~10–20% overlap. Overlap (repeating the last N tokens of one chunk at the start of the next) prevents an answer that straddles a boundary from being lost — at the cost of some duplication. Tune size/overlap empirically against your eval (09); it's the cheapest lever you have.

Measure in tokens, not characters

Chunk size should be in tokens (what the embedding model and context budget actually count, Phase 1.01), not characters or words — and respect the embedding model's max input (e.g., 512 or 8192 tokens, 03). Splitting by raw word_count is a rough proxy; use the model's tokenizer for correctness.

Strategies (worst → best for typical docs)

Recursive character/token splitting (try the largest natural boundary that fits, then fall back) is the pragmatic default; structure-aware (split on headers, keep a section together) is better when your docs have clear structure (which good ingestion preserved, 01). Never naïvely fixed-split structured content like tables or code.

# Recursive splitting: prefer big natural boundaries, fall back to smaller
SEPARATORS = ["\n## ", "\n### ", "\n\n", "\n", ". ", " "]   # headings → paras → sentences → words
def recursive_split(text, max_tokens, overlap, count=count_tokens):
    if count(text) <= max_tokens:
        return [text]
    for sep in SEPARATORS:
        if sep in text:
            parts, chunks, cur = text.split(sep), [], ""
            for p in parts:
                if count(cur + sep + p) > max_tokens and cur:
                    chunks.append(cur)
                    cur = cur[-overlap_chars(overlap):] + sep + p   # carry overlap
                else:
                    cur = (cur + sep + p) if cur else p
            if cur: chunks.append(cur)
            # recurse on any still-too-big chunk
            return [c for ch in chunks for c in recursive_split(ch, max_tokens, overlap, count)]
    return [text[:]]   # last resort: hard cut

Advanced patterns (when basic chunking plateaus)

• Metadata-enriched chunks — prepend the doc title / section heading to each chunk so an isolated passage stays self-contained ("In Refund Policy: …"). Cheap, often a big win.
• Parent–child / small-to-big — embed small chunks (precise retrieval) but return the larger parent passage to the generator (full context). Best of both (07).
• Contextual retrieval — prepend a short LLM-generated summary of the chunk's place in the document (an Anthropic-popularized technique) to boost retrieval of ambiguous chunks.
• Per-content-type chunking — chunk code by function/class, tables by row-group, prose by section. One size does not fit all.

Chunks carry metadata

Every chunk inherits its document's metadata (01) — source, page/section (for citations, 08), acl (for ACL-retrieval, 10), doc_id (to group/return parents). The chunk is a record, not just a string.

3. Mental Model

   clean doc [01] → SPLIT into chunks (the UNIT OF RETRIEVAL: embedded·searched·ranked·injected)
        size↓  precise but fragmented (recall↓)      size↑  full but diluted + noisy (precision↓, cost↑)
        → start ~256–512 tokens, ~10–20% OVERLAP; measure in TOKENS; tune vs eval [09]

   STRATEGY (worst→best): fixed < sentence < RECURSIVE < structure-aware < semantic
        never fixed-split tables/code; respect structure ingestion preserved [01]

   ADVANCED: enrich chunk w/ title/section · parent–child (embed small, return big) [07]
             · contextual prefix · per-content-type chunking
   chunks carry METADATA (source/page→citations[08], acl→ACL-retrieval[10], doc_id→parent)

Mnemonic: the chunk is the unit of retrieval — small=precise/fragmented, large=full/noisy. Default to recursive/structure-aware at ~256–512 tokens with overlap; enrich with title; tune against eval.

4. Hitchhiker's Guide

What to look for first: your chunk size + overlap and whether splitting respects structure (no mid-table/mid-sentence cuts). Then enrich chunks with their title/section.

What to ignore at first: semantic/LLM-based chunking and exotic hierarchies. Start with recursive token splitting at ~400 tokens, ~15% overlap, structure-aware where you have headers; optimize against eval.

What misleads beginners:

• Blaming the embedding model. Poor retrieval is often bad chunks (too big/small, structure-destroyed) — fix chunking first.
• Fixed-splitting everything. Cuts tables, code, and sentences — destroys meaning.
• Measuring in characters. Use tokens (the model's unit) and respect the embedder's max input (03).
• One size for all content. Code, tables, and prose want different chunking.
• Isolated chunks with no context. A passage stripped of its heading is hard to retrieve and to ground — enrich it.

How experts reason: they pick recursive/structure-aware splitting, set size to the embedder's sweet spot with overlap, enrich chunks with title/section, use parent–child when chunks need both precision and context, chunk per content type, and A/B chunking against a retrieval eval (09) because it's the cheapest high-impact lever.

What matters in production: retrieval recall/precision as a function of chunk strategy (measure!), coherent boundaries (no split tables/code), self-contained chunks (enrichment), and re-chunking being cheap to iterate (01 incremental).

How to debug/verify: when retrieval misses, read the chunks — are answers split across boundaries (increase size/overlap)? are chunks topic-soup (decrease size / semantic split)? are tables mangled (structure-aware)? Then re-ingest and re-measure.

Questions to ask: what chunk size/overlap, in tokens? does splitting respect structure? are chunks enriched with title/section? different strategy per content type? measured against eval?

What silently gets expensive/unreliable: structure-destroying fixed splits (mangled tables/code), oversized chunks (cost + diluted embeddings + lost-in-the-middle, 07), and chunks too small to answer (recall collapse).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The chunk is the unit of retrieval — it's what's embedded, searched, ranked, and injected.
• Size/overlap is the core trade-off: small = precise but fragmented (recall↓); large = full but diluted + noisy (precision↓, cost↑).
• Start ~256–512 tokens, ~10–20% overlap; measure in tokens and respect the embedder's max input (03).
• Recursive / structure-aware splitting beats fixed-size for most docs; never fixed-split tables/code.
• Enrich chunks with title/section (and optionally a contextual summary) to keep them self-contained.
• Parent–child (small-to-big) gives precise retrieval + full context to the generator (07).
• Chunk per content type — code, tables, prose differ.
• Chunking is the cheapest high-impact lever — A/B it against retrieval eval (09).

9. Observations from Real Systems

• The #1 cheap RAG win in practice is fixing chunking (size/overlap/structure/enrichment) before touching models (00).
• LangChain's RecursiveCharacterTextSplitter is the de-facto default; Unstructured's by_title and LlamaIndex node parsers add structure/semantic options.
• Anthropic's "contextual retrieval" (prepend an LLM-written context blurb per chunk) measurably improved retrieval on ambiguous chunks — a popular enrichment.
• Parent–child retrieval is widely used so embeddings stay sharp while the generator still gets full context (07).
• Code RAG chunks by function/class (AST), not fixed size — a per-content-type example (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

CHUNK a corpus:
 1. DEFAULT: recursive token splitting, ~256–512 tokens, ~10–20% overlap (in TOKENS, ≤ embedder max). [03]
 2. RESPECT STRUCTURE: split on headers/sections; NEVER fixed-split tables or code.
 3. ENRICH: prepend doc title/section (and optionally a contextual summary) to each chunk.
 4. PER CONTENT TYPE: code → by function/class; tables → by row-group; prose → by section.
 5. NEED PRECISION + CONTEXT? → parent–child (embed small, return parent passage). [07]
 6. CARRY METADATA: source/page (citations[08]), acl (ACL[10]), doc_id (parent grouping).
 7. TUNE: A/B chunk size/overlap/strategy against RETRIEVAL EVAL; re-ingest (incremental [01]).

12. Hands-On Lab

Goal

Show that chunk strategy and size change retrieval quality by A/B-testing chunking against a fixed query set on the same corpus.

Prerequisites

• The ingested docs from 01; an embedding model + a vector store (03/04); a handful of questions with known answer locations.

Steps

1. Build 3 chunkings of the same corpus: (a) fixed-size 128 tokens no overlap, (b) recursive 400 tokens 15% overlap, (c) structure-aware (split on headers) — each stored in a separate collection.
2. Fixed query set: ~10 questions where you know which passage answers each (label the gold chunk/section).
3. Retrieve top-k for each question under each chunking; record whether the answering passage was retrieved (recall@k) and how much irrelevant text came with it.
4. Compare: tabulate recall@k and average retrieved-token count per chunking. Expect fixed-128 to fragment answers (low recall), 400-recursive to balance, structure-aware to shine on the structured docs.
5. Enrichment test: prepend each chunk's section heading; re-embed; re-measure recall on the ambiguous questions.
6. (Optional) parent–child: embed 128-token children but return their 400-token parents; show retrieval stays sharp while the generator gets fuller context (07).

Expected output

A table: chunk strategy/size → recall@k, avg retrieved tokens — demonstrating the size/overlap trade-off and that structure-aware + enrichment beat naïve fixed splitting.

Debugging tips

• Recall low everywhere → questions may need hybrid search (05) not just chunking; or chunks are topic-soup.
• Structure-aware no better → docs lack clear headers (ingestion didn't preserve structure, 01).

Extension task

Add semantic chunking (split where consecutive-sentence embedding similarity drops) and compare to recursive.

Production extension

Pick the winning strategy, make chunk size/overlap config, and re-chunk incrementally on ingestion (01); track recall as a guardrail metric (09).

What to measure

recall@k and avg retrieved tokens per chunk strategy/size; enrichment effect; parent–child precision/context.

Deliverables

• 3+ chunkings of one corpus with a recall@k comparison.
• An enrichment before/after on ambiguous queries.
• A chosen strategy + size/overlap, justified by the data.

13. Verification Questions

Basic

1. Why is the chunk called "the unit of retrieval"?
2. What happens if chunks are too small? Too large?
3. Why measure chunk size in tokens, not characters?

Applied 4. Choose a chunking strategy + size for (a) a wiki, (b) a codebase, (c) PDFs with tables. Justify. 5. What is parent–child (small-to-big) retrieval and what problem does it solve?

Debugging 6. Answers are partial, missing context that spans two chunks. Fix? 7. Retrieved context is topic-soup. Fix?

System design 8. Design a per-content-type chunking pipeline with enrichment and metadata for citations/ACL.

Startup / product 9. Why is chunking often the cheapest, highest-ROI improvement to a struggling RAG product?

14. Takeaways

1. The chunk is the unit of retrieval — chunking caps quality before retrieval runs.
2. Size/overlap trade-off: small=precise/fragmented, large=full/noisy; start ~256–512 tokens, ~10–20% overlap, in tokens.
3. Recursive/structure-aware beats fixed-size; never fixed-split tables/code.
4. Enrich chunks (title/section/context) and use parent–child for precision + context.
5. It's the cheapest high-impact lever — A/B chunking against retrieval eval (09).

15. Artifact Checklist

• 3+ chunkings of one corpus + a recall@k comparison.
• A chosen size/overlap + strategy justified by data.
• An enrichment (title/section) before/after.
• Per-content-type chunking for any code/tables.
• Chunk metadata (source/page, acl, doc_id) carried through.

Up: Phase 9 Index · Next: 03 — Embeddings