« Phase 04 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 04 — Principal Deep Dive: Context Engineering

Zoom out from the functions. In a production agent platform, build_agent_context is one stage of a per-turn pipeline: orchestrate → route → gather (memory + retrieval + tools) → assemble → call model → parse → persist. Context engineering is the "gather + assemble" stage, and it runs on the request hot path for every turn of every session of every tenant. That single fact — it is on the hot path, at fan-out — dictates most of the architecture below.

The hot path and the cold path

The assembler must be cheap: O(n log n) over a handful of sections, no I/O, no model calls. That is why assemble_context, order_for_recall, and route (bag-of-words cosine) are pure CPU. The expensive operations are pushed off the synchronous assembly path:

• Embedding happens on write (remember embeds when a fact is stored), not on assembly. Recall at assembly time is a cosine sweep over already-embedded vectors.
• Compaction happens on eviction, amortized across the turns that pushed a turn out of the buffer — not as a blocking "summarize the whole history" call before each model invocation.
• Retrieval (Phase 05/06) is the one genuinely expensive gather step; it is a vector-store round trip, and it is why retrieval latency, not assembly latency, dominates the gather stage.

The design rule: the thing that runs every turn is arithmetic; the things that touch a model or a database run on writes and evictions and are cached. Invert that — call the summarizer LLM synchronously on every turn — and you have doubled your per-turn latency and cost to maintain a summary that changed on one turn in ten.

The token and cost envelope

Concretely: suppose an 8K-token budget, output priced 3–5× input, and a ReAct loop that resends its scratchpad every step so input grows like (t+o)·n²/2 in n steps (Phase 00). Assembly is what keeps n from mattering — compaction caps the history contribution at one bounded summary string instead of a linearly growing transcript, so a 40-turn conversation still assembles into the same 8K budget as turn 3. Without compaction, the transcript alone exhausts the budget around the turn where cumulative history crosses 8K, and every turn after that either overflows (hard error) or silently truncates the oldest — which is the amnesia failure mode. The budget is the invariant; compaction and recall are how you hold it as the conversation grows without bound.

Where the bodies are buried: RAG vs long context

The seductive architecture is "the model has a 1M window, paste the corpus, skip retrieval." It loses on all three budget axes at once. Cost: the pasted corpus is re-billed every turn (and a ReAct agent re-sends it every step). Latency: prefill scales with input length, so a 1M-token prefix is a multi-second first token. Quality: context rot and lost-in-the-middle mean accuracy falls as you dilute the signal across a huge window. Retrieval — fetch the few most relevant chunks, place them at the edges via order_for_recall — is usually cheaper and more accurate. The honest nuance a principal owns: for a small, stable corpus that fits comfortably and rarely changes, long-context wins on operational simplicity (no index to build, embed, or keep fresh); for a large or volatile corpus, retrieval wins on both cost and quality. This is a per-corpus decision, not a doctrine.

The tension a principal must resolve: recall order vs cache prefix

This is the single most important architectural subtlety in the phase, and it is a genuine conflict between two things the labs teach.

order_for_recall arranges sections by this turn's priority and which sections happen to exist this turn — it deals highest-priority to the edges. Prompt caching (Anthropic prompt caching, provider KV caching; Phase 14) wants the opposite: a byte-identical prefix across turns so the processed prefix is billed at a steep discount and skips prefill. These pull against each other. Recall ordering reshuffles the prompt whenever priorities or section membership change — recalled facts differ per query, retrieved docs differ per query — so the prefix is not stable, and the cache never warms.

The resolution, which is the thing that separates someone who read the lab from someone who has run this: partition the prompt into an immovable cacheable head and a recall-ordered volatile tail. The system prompt, tool schemas, and few-shot examples are identical across turns; freeze them as a fixed prefix outside the recall permutation and mark the cache breakpoint there. Only the volatile sections — recalled facts, retrieved docs, recent turns, the task — flow through order_for_recall. You lose the ability to bracket the task at the very front (it is already after the cached head), but you keep it at the very end, which is the position that matters most, and you keep a warm cache. The lab's order_for_recall permutes everything including the pinned system prompt; a production assembler would exempt the cacheable prefix from the permutation. That "looks wrong but is intentional" for a teaching miniature — it demonstrates the recall mechanism in isolation — and would be wrong in production against a cache.

Failure modes and blast radius

Rank failures by blast radius, because that is how you prioritize the guardrails:

• Pinned overflow → raise. Fail-closed, blast radius one turn, loudly. Correct by construction.
• A retrieved doc dropped by the budget. Wrong or incomplete answer, blast radius one turn, silent unless you log the accounting. This is why AssembledContext carries included / dropped / truncated: the eviction is invisible without the trace.
• Compaction drops a fact into the lossy summary. Blast radius the whole session — the fact is gone for every subsequent turn, not one. This is why you keep recent turns verbatim and only summarize what evicted, and why the summarizer prompt (in production) is instructed to preserve entities, IDs, and decisions.
• Cross-tenant memory leak. Blast radius catastrophic — see below.

Cross-cutting: multi-tenancy and the privacy of memory

Memory is the stateful, cross-turn, cross-session tier, which makes it the highest-risk surface in the phase. The lab's TieredMemory is a single in-process object; a real platform has one logical memory per (tenant, user, thread). Two non-negotiables:

1. Recall must filter by identity before cosine, not after. recall in the lab sweeps every SemanticItem. In production the semantic store is partitioned by tenant/user and the query is scoped to that partition — a cosine match that surfaces tenant A's account ID into tenant B's prompt is a data-exfiltration incident, not a relevance bug. Scoping is a pre-filter (which rows are even candidates), never a post-filter you might forget.
2. The summary and the semantic store are durable PII. Compaction folds raw turns — which may contain names, card numbers, health details — into a persisted summary; remember persists facts indefinitely. That means retention, TTL/decay, right-to-erasure, and redaction all apply to memory, and an append-only store (like the lab's) that never forgets is a compliance liability. The lab's "extensions" note — memory decay, dedup, pin_fact — is where those requirements land.

Observability and the token math you actually emit

The debugging question in production is always "why did the agent forget / ignore / mishandle X on turn N?" and the answer must be in a trace. Emit per turn: budget, token_count, the dropped and truncated section names, the route intent + score + reason (match / low_confidence / ambiguous_tie), and cache hit/miss on the prefix. With that, "it forgot the account ID" resolves in one query — the ID was dropped by the budget, or truncated off the end of a section, or never recalled, or fell into the lossy summary. Without it, you are guessing at a non-deterministic model. The included / dropped / truncated accounting is not lab decoration; it is the single most useful telemetry an agent emits, and almost nobody instruments it.

The intentional-looking-wrong decisions, catalogued

• Recompute the whole summary on every eviction (O(E²) per session): intentional for determinism in the lab; incremental folding in production.
• token_count excludes section headers: intentional simplification; count the rendered string in production.
• Global threshold/margin on the router: intentional; production routers set a per-intent threshold because "billing" and "smalltalk" have different natural score distributions.
• order_for_recall permutes the pinned prefix: intentional to isolate the recall mechanism; exempt the cacheable head in production.

Every one of these is a deliberate teaching simplification with a known production upgrade path — which is precisely the set of decisions an architecture review will ask you to defend.