« Phase 04 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 04 — Deep Dive: Context Engineering

The load-bearing idea of this phase is that a prompt is not a string you concatenate; it is the output of three separable operations that a naive builder fuses and gets wrong. Selection decides which candidate sections survive a token budget. Arrangement decides where each survivor lands in the final byte sequence. Constraint decides which sections are non-negotiable regardless of budget pressure. assemble_context and order_for_recall are deliberately split along exactly these axes: assemble_context owns selection and constraint, order_for_recall owns arrangement, and the caller composes them. Fusing the three — the naive "\n".join(everything[:fits]) — is where every context bug lives.

Data structures and why they are shaped that way

ContextSection(name, text, priority, pinned, truncatable) is a frozen=True dataclass, so it is hashable and immutable. The immutability matters because the same section object flows through two passes — order_for_recall reorders the list, then assemble_context iterates it twice — and nothing may mutate a section between passes.

The subtle choice is inside assemble_context: the survivor set is chosen: dict[int, str] keyed by id(section), not by the section itself. A frozen dataclass is hashable, so keying by value is possible — but two value-equal sections (same name, same text, same priority) would collapse to one dict entry and one would silently lose its budget slot. Keying by id() keys by object identity, so duplicates each get an independent decision. truncated_ids: set[int] uses the same identity key. This is the kind of decision that looks like a stylistic preference and is actually a correctness invariant.

The working buffer is a deque[Turn] (not a list) because eviction is popleft() — O(1) at the head, where a list is O(n). _evicted: list[Turn] is a separate append-only log holding everything the summary was ever built from, in order; the rolling summary is a pure function of that log.

assemble_context: control flow, invariants, complexity

The algorithm is a knapsack with a hard mandatory set, executed in three phases:

1. Constraint check. Sum count_tokens over every pinned section. If that exceeds budget, raise — there is no valid prompt. This is fail-closed: the system prompt and the task cannot be evicted to make room for a retrieved document. remaining = budget - pinned_tokens.
2. Priority fill. Sort the non-pinned sections by -priority with Python's stable sorted, so equal priorities keep caller order. Walk them descending: if cost <= remaining, take whole; else if truncatable and remaining > count_tokens(ELISION), take the head via truncate_to_tokens; else append the name to dropped. Decrement remaining by whatever was actually consumed.
3. Render in caller order. Iterate the original sections list (the arrangement handed in), and for each section whose id() is in chosen, emit ### {name}\n{body}, joined by \n\n.

Phase 3 is the mechanism that separates selection from arrangement: selection happened in priority order, but rendering walks the caller's order. So order_for_recall can put the system prompt first and the task last, while assemble_context still selects the task before a low-priority doc. Complexity is O(n log n) for the sort plus O(n) fill — cheap, which matters because this runs on every turn (Principal Deep Dive develops why the hot path must stay cheap).

One efficiency wrinkle worth naming: truncate_to_tokens rebuilds " ".join(kept + [word]) + ELISION and calls count_tokens on the growing candidate once per word, so trimming a w-word section is O(w²) character work. Fine at lab scale; a production version binary-searches the cut point or counts incrementally.

The invariant caveat. token_count is sum(count_tokens(body)) over the chosen bodies. The rendered text also carries the ### {name}\n headers and \n\n joins, which are real tokens the sum does not count. So token_count <= budget holds for section content, but the true rendered prompt is slightly larger. This is an intentional lab simplification; a production assembler counts the fully rendered string against the budget. Know this is there — it is exactly the class of off-by-a-header bug that silently pushes a caller over the model's hard limit.

A worked assembly pass with token math

Take the main() example: budget=60, four sections — system (priority 100, pinned), task (95, pinned), policy (60, truncatable), chit_chat (10, not truncatable).

Token counts (ceil(len/4)): system = "You are a careful billing assistant." = 36 chars → 9 tokens; task = "Explain the duplicate charge on my card." = 40 chars → 10 tokens; policy = a 48-char clause ×6 = 288 chars → 72 tokens; chit_chat = a 38-char clause ×8 = 304 chars → 76 tokens.

First the caller runs order_for_recall. Ranked by priority: [system, task, policy, chit_chat]. Deal alternately — index 0→front, 1→back, 2→front, 3→back — giving front=[system, policy], back=[task, chit_chat]; reverse back to [chit_chat, task]; concatenate: [system, policy, chit_chat, task]. The two highest-priority sections now sit at the two ends.

Now assemble_context on that arrangement:

• Pinned = {system, task} = 9 + 10 = 19 tokens ≤ 60. remaining = 41.
• Non-pinned by descending priority: policy (60), then chit_chat (10).
• policy: cost 72 > 41, but truncatable and 41 > count_tokens(ELISION) = 1. Truncate to 41 tokens: truncate_to_tokens adds whole words until the next word plus " […]" would exceed 41, guaranteeing count_tokens(result) <= 41. remaining drops to roughly 0.
• chit_chat: cost 76 > remaining, and not truncatable → dropped.

Render in caller (recall) order, keeping only chosen: included = [system, policy, task], truncated = [policy], dropped = [chit_chat]. The pins survived, the budget held, the low-priority non-truncatable block was cut first, and system/task bracket the prompt. Every one of those outcomes is a consequence of the three-phase mechanism, not a heuristic.

A routing decision, to the arithmetic

route("cancel and refund charge") against billing = {refund, charge, invoice, payment, money} and cancel = {cancel, charge, subscription, stop, account}.

qvec = {cancel:1, and:1, refund:1, charge:1}, norm √4 = 2. Each intent prototype has norm √5 ≈ 2.236. Dot with billing: shares refund, charge → 2. Dot with cancel: shares cancel, charge → 2. So cos = 2 / (2 × 2.236) = 0.447 for both. support shares nothing → 0.

The scored list sorts by (-score, name); the name tie-break is deterministic, so billing (alphabetically first) is top, cancel is second, both 0.447. Gate one: 0.447 >= threshold 0.3, pass. Gate two: second_score 0.447 > 0 and top - second = 0.0 < margin 0.05 → ambiguous_tie, fallback=True. The router refuses. Change the query to "I need a refund for my invoice payment": billing dot = refund + invoice + payment = 3, cos = 3/(√8·√5) ≈ 0.474; cancel shares nothing → 0. Gate one passes, gate two is skipped because second_score == 0.0 is not > 0 → clean match to billing. Same code, opposite decision, driven entirely by the two gates. The margin gate is what encodes "a workflow collision is a tie, not a low score" — a confident tie (0.447 vs 0.447) is caught precisely because both gates exist.

Compaction: the eviction mechanism

add_turn appends to the buffer, then while len(buffer) > buffer_size it popleft()s the oldest turn, appends it to _evicted, and recomputes summary = summarizer(_evicted). The rolling summary is therefore a pure function of the full eviction history, recomputed from scratch on each eviction — O(E) per eviction, O(E²) over a session. That is deliberate: it makes the summary a deterministic function of what fell out, so a test can add buffer_size + 1 turns and assert the first turn's content now appears in the summary. A production system folds incrementally (summarize the summary plus the newly evicted turn) to avoid the quadratic, trading exact reproducibility for cost.

Why the naive approach fails at the mechanism level

Concatenate-and-truncate fails because it has no representation of the three axes. It has no constraint axis, so a large retrieved doc can push the system prompt past the cut — deleting safety rules to quote a web page. It has no arrangement axis distinct from selection, so the most important sections land wherever priority order happens to put them, usually the middle, where recall is weakest (the U-shaped curve). It truncates on byte offsets, producing garbage sub-word tokens and a list that looks complete because there is no elision marker. And its routing analog — argmax(scores) — always dispatches, so a two-way tie is resolved by a coin flip that silently mishandles half the request. Splitting selection, arrangement, and constraint into separate, individually testable operations is the entire mechanical contribution of this phase.