Context Packing

Phase 9 · Document 07 · RAG Prev: 06 — Reranking · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

You've retrieved (05) and reranked (06) — now you must assemble the final prompt: which chunks, in what order, within what token budget, formatted how. This is context packing, and it's where good retrieval can still be wasted. Dump too many chunks and you pay more, slow down, and trigger "lost in the middle" (models attend best to the start/end of long context, ignoring the middle). Order them badly and the most relevant chunk sits where the model ignores it. Pack without source markers and you can't get citations (08). Packing is also where query rewriting (HyDE, multi-query) and context compression live — techniques that improve what reaches the generator. Get packing right and a modest model gives grounded, cited answers; get it wrong and great retrieval underperforms.

2. Core Concept

Plain-English primer: assemble the prompt deliberately

After reranking you have a ranked top-k of chunks. Context packing is the step that turns those into the actual text block in the prompt, deciding:

1. How many chunks fit (token budget),
2. In what order (to beat lost-in-the-middle),
3. Formatted how (source markers for citations, dedup), and
4. (Upstream) what query you even retrieved with (query rewriting).

reranked top-k → BUDGET (fit window, reserve output) → DEDUP → ORDER (edges) → FORMAT (source markers)
              → [system prompt + packed context + question] → generate [08]

It's the same idea as the agent's context assembly (what-happens §0–§1) and reduction (§3), specialized for RAG.

The token budget

The prompt must fit system + packed_context + question + reserved_output ≤ context_window (Phase 1.01). So you can't just paste all top-k — you budget the context portion and select the highest-ranked chunks that fit, reserving room for the answer (and any thinking tokens, Phase 2.09). More context also costs more per call and adds latency (Phase 7.09) — so fewer, better chunks beat more, noisier ones.

Lost in the middle (why order matters)

A robust empirical finding: LLMs use information at the beginning and end of a long context far better than the middle — the "lost in the middle" effect. So don't bury your best chunk in the middle. Practical orderings:

• Put the most relevant chunks at the edges (first and last), weaker ones in the middle.
• Or put the most relevant last (closest to the question), since recency is strong.

This is cheap and real: reorder the same chunks and answer quality changes. (Connects to Phase 2.01 and what-happens §5.)

Dedup, formatting, and parent expansion

• Dedup near-identical chunks (overlap from chunking 02, or the same passage from multiple docs) so you don't waste budget on repeats.
• Format with source markers — wrap each chunk with [SOURCE n] (path/title) so the generator can cite it (08) and you keep provenance.
• Parent expansion (small-to-big): if you embedded small chunks for precise retrieval (02), pack the larger parent passage so the generator gets full context — precision in retrieval, completeness in generation.

def pack(reranked, question, budget_tokens, count=count_tokens):
    seen, packed, used = set(), [], 0
    for c in reranked:                      # already in relevance order [06]
        h = chunk_hash(c["content"])
        if h in seen: continue              # dedup
        t = count(c["content"])
        if used + t > budget_tokens: break  # budget
        packed.append(c); used += t; seen.add(h)
    packed = reorder_edges(packed)          # best at start/end (lost-in-the-middle)
    blocks = [f"[SOURCE {i+1}] ({c['source']})\n{c['content']}" for i, c in enumerate(packed)]
    return "\n\n".join(blocks)              # → into the prompt with the question [08]

Query rewriting (improve what you retrieve before you pack)

Packing quality is bounded by retrieval quality, and the query drives retrieval. Rewriting the query often helps:

• Multi-query: generate several paraphrases of the question, retrieve for each, and union/fuse — boosts recall for under-specified queries.
• HyDE (Hypothetical Document Embeddings): ask an LLM to write a hypothetical answer, embed that, and retrieve with it — the hypothetical answer is often closer in embedding space to real answer passages than the bare question.
• Query expansion / decomposition: add synonyms/terms, or split a multi-part question into sub-queries retrieved separately.
• Conversational rewrite: resolve "what about its pricing?" into a standalone query using chat history.

These cost an extra LLM call (latency/$) but can materially lift recall (09) — measure the trade.

Context compression (when chunks are long)

If kept chunks are long or redundant, compress before generation: extractive (keep only the sentences relevant to the query — e.g., LLMLingua-style or an LLM filter) or abstractive (summarize chunks). This packs more signal per token, mitigating lost-in-the-middle and cost — at the risk of dropping a needed detail, so use carefully and evaluate.

3. Mental Model

   reranked top-k [06] → CONTEXT PACKING → final prompt → generate [08]
     ① BUDGET: system + context + question + RESERVED OUTPUT ≤ window (fewer/better > more/noisy) [1.01,7.09]
     ② DEDUP near-identical chunks (overlap [02])
     ③ ORDER for LOST-IN-THE-MIDDLE: best chunks at the EDGES (or most-relevant LAST)
     ④ FORMAT with [SOURCE n] markers → enables citations [08]
     ⑤ PARENT EXPANSION: embed small (precise), pack the bigger parent (complete) [02]

   UPSTREAM (improve retrieval before packing): query rewriting — multi-query · HyDE · expansion · convo-rewrite
   long/redundant chunks → COMPRESS (extractive/abstractive) for more signal per token

Mnemonic: pack deliberately — budget (reserve output), dedup, order for the edges, mark sources. Rewrite the query to retrieve better; compress long chunks. Fewer better chunks beat more noisy ones.

4. Hitchhiker's Guide

What to look for first: are you (a) reserving output budget, (b) ordering for lost-in-the-middle, and (c) adding source markers? Those three are cheap and high-impact.

What to ignore at first: HyDE, multi-query, and compression — add them only if eval shows a recall/length problem. Start with simple budgeted packing + edge ordering + source markers.

What misleads beginners:

• Stuffing all top-k. More chunks = more cost, more latency, and lost in the middle — fewer, better chunks usually win (00).
• Ignoring order. Burying the best chunk in the middle wastes good retrieval — put it at an edge.
• No output reservation. Packing to the window edge truncates the answer (what-happens §10).
• No source markers. Then you can't produce citations (08).
• Forgetting dedup. Overlapping chunks (02) waste budget on repeats.

How experts reason: they budget the context (reserving output), dedup, order best-at-edges, format with source markers, and use parent expansion for completeness; they add query rewriting (HyDE/multi-query) when recall is weak and compression when chunks are long — always validating the trade against eval (09). Their guiding rule: maximize signal-per-token, not chunk count.

What matters in production: answer quality vs number/length of chunks (find the sweet spot), output truncation avoidance, citation-ready formatting, and the latency/cost of rewriting/compression vs their recall/quality gain.

How to debug/verify: if answers are incomplete, check (1) was the needed chunk packed (budget/dedup)? (2) was it in the middle (reorder)? (3) was output truncated (reserve more)? Sweep k and ordering against eval (09); A/B query rewriting on under-specified queries.

Questions to ask: am I reserving output tokens? what ordering? source markers present? would HyDE/multi-query help my under-specified queries? are chunks long enough to need compression?

What silently gets expensive/unreliable: stuffing too much (cost + lost-in-the-middle), middle-buried best chunks, truncated answers (no output reservation), missing citations, and unmeasured rewriting/compression latency.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Context packing = select + dedup + order + format the reranked chunks within a token budget, reserving room for the answer (Phase 1.01).
• Fewer, better chunks usually beat more, noisier ones — more context costs more and triggers lost-in-the-middle.
• Lost in the middle is real: put the best chunks at the edges (or most-relevant last) — reordering the same chunks changes quality.
• Reserve output (and thinking) budget or the answer truncates (what-happens §10).
• Format with [SOURCE n] markers to enable citations (08) and dedup overlapping chunks (02).
• Parent expansion (small-to-big) gives precise retrieval + complete context (02).
• Query rewriting (multi-query / HyDE / expansion) improves recall before packing — at an extra LLM call's cost.
• Context compression (extractive/abstractive) packs more signal per token for long/redundant chunks — measure the risk of dropping detail.

9. Observations from Real Systems

• Reorder-for-edges and dedup are cheap, common production tweaks that meaningfully lift answer quality (Lost-in-the-Middle).
• HyDE and multi-query are widely used for under-specified or conversational queries (09).
• Parent–child / small-to-big is a standard pattern (embed small for precision, pack the parent for context) (02).
• Frameworks (LlamaIndex node postprocessors, LangChain compressors) ship reorder/dedup/compression building blocks.
• The recurring failure is "good retrieval, bad answer" caused by stuffing too many chunks or burying the best one — fixed in packing, not retrieval.

10. Common Misconceptions

11. Engineering Decision Framework

PACK the context (after rerank [06]):
 1. BUDGET: context_tokens = window − system − question − RESERVED_OUTPUT(+thinking). [1.01,2.09]
 2. SELECT highest-ranked chunks that fit; DEDUP near-identical [02]. Fewer/better > more/noisy.
 3. ORDER for lost-in-the-middle: best chunks at the EDGES (or most-relevant last). [2.01]
 4. FORMAT with [SOURCE n] (path/title) markers → citations [08]. Parent-expand if embedded small [02].
 5. UPSTREAM (if recall weak): query rewriting — multi-query / HyDE / expansion / convo-rewrite (extra LLM call).
 6. LONG/redundant chunks → COMPRESS (extractive/abstractive); validate no detail lost.
 7. TUNE k, order, rewriting, compression against EVAL (faithfulness + answer quality). [09]

12. Hands-On Lab

Goal

Show that packing choices change answer quality at fixed retrieval: test k, ordering, output reservation, and a query rewrite.

Prerequisites

• The reranked top-k from 06; a generation model; ~10 labeled questions; an answer-quality/faithfulness check (09).

Steps

1. Baseline pack: take reranked top-8, format with [SOURCE n] markers, put context then question; generate; score answer quality + faithfulness.
2. k sweep: repeat with k=3/5/8/12; find where quality peaks then declines (more chunks → noise/lost-in-the-middle). Note cost/latency rising with k.
3. Order test: with fixed k, compare (a) relevance order, (b) best-at-edges, (c) reversed (best last). Show ordering changes quality on multi-chunk questions (Lost in the Middle).
4. Output reservation: set max_tokens low enough that a long answer truncates; show packing to the window edge causes cutoff; fix by reserving output.
5. Dedup: include overlapping chunks (from 02); add dedup and show freed budget → an extra useful chunk fits.
6. Query rewrite: for 2–3 under-specified questions, apply HyDE (LLM writes a hypothetical answer → embed that → retrieve) and compare recall/answer quality to the bare query.

Expected output

A table: packing variant (k / order / dedup / rewrite) → answer quality + faithfulness + tokens — demonstrating fewer-better-ordered chunks and rewriting beat naïve stuffing.

Debugging tips

• Quality drops as k rises → lost-in-the-middle/noise; reduce k or reorder.
• Answers cut off → not reserving output budget.

Extension task

Add context compression (extractive: keep query-relevant sentences) and show more chunks' signal fits in the same budget without quality loss.

Production extension

Make k, ordering, and rewriting config; reserve output dynamically; track answer faithfulness vs context tokens to find the cost/quality sweet spot (Phase 7.09, 09).

What to measure

Answer quality + faithfulness vs k; ordering effect; truncation vs output reservation; dedup budget savings; HyDE recall uplift; context tokens (cost).

Deliverables

• A packing-variant comparison (k / order / dedup) on quality + faithfulness.
• An ordering before/after (lost-in-the-middle).
• A query-rewrite (HyDE) before/after on under-specified queries.

13. Verification Questions

Basic

1. What four decisions does context packing make?
2. What is "lost in the middle," and how do you mitigate it?
3. Why reserve output budget when packing?

Applied 4. Why do fewer, better chunks often beat more chunks? 5. What is HyDE, and when does it help retrieval?

Debugging 6. Retrieval is good but answers are weak/incomplete. Walk through your packing checks. 7. Answers get cut off mid-sentence. Cause and fix.

System design 8. Design a context packer with budgeting, dedup, edge ordering, source markers, and optional rewriting/compression.

Startup / product 9. How do packing choices (k, order, compression) affect both answer quality and per-query cost?

14. Takeaways

1. Packing assembles the prompt: budget (reserve output) → dedup → order (edges) → format (source markers) → generate.
2. Fewer, better, well-ordered chunks beat more, noisier ones — and lost in the middle makes order matter.
3. Reserve output/thinking budget to avoid truncation; source markers enable citations (08).
4. Parent expansion gives precision + completeness; query rewriting (HyDE/multi-query) lifts recall upstream.
5. Compress long chunks for more signal-per-token; tune k/order/rewriting against eval (09).

15. Artifact Checklist

• A packing-variant comparison (k / order / dedup) on quality + faithfulness.
• An edge-ordering before/after (lost-in-the-middle).
• Output reservation preventing truncation.
• Source markers + dedup in the packed context.
• A query-rewrite (HyDE/multi-query) before/after on weak queries.

Up: Phase 9 Index · Next: 08 — Citations and Grounding