« Phase 04 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 04 — Staff Engineer Notes: Context Engineering

Anyone can call an assembler. The line between someone who uses context engineering and someone trusted to own it is whether they treat the context window as a resource with a budget, a policy, and a failure catalog — or as a text box they stuff. This is the judgment layer: the decisions a staff engineer owns here, the framework for making them, the review reflexes, and the exact signal an architecture review is listening for.

The decisions you own

A staff engineer does not own "which tokenizer" — that is a swappable seam. They own the policy:

  • The budget allocation. How many tokens each source (system, memory, retrieval, task) gets, and the priority order that resolves contention. This is a product decision disguised as a config value: a support bot that must never lose the account context allocates memory differently than a coding agent that must never lose the current file.
  • The route-or-clarify boundary. The threshold and margin, tuned against a labeled set, with the false-route rate measured and watched — not guessed once and forgotten. Owning routing means owning the tradeoff between "clarify too often (annoying)" and "route wrong (silent workflow collision)."
  • The compaction policy. What stays verbatim, what the summarizer is instructed to preserve (IDs, decisions, open threads), and when the summary itself gets re-summarized. Compaction is lossy on purpose; owning it means owning what you are willing to lose.
  • Memory scoping and retention. Tenant/user partitioning, TTL/decay, redaction of PII before it is persisted into a summary or semantic store. This is the highest-blast-radius decision in the phase and it is not optional.
  • Whether it should be an agent at all. The Phase 00 reflex still applies: the most senior move is sometimes deleting the context-engineering problem by not building the agent.

The decision framework: RAG vs long-context vs cache

When someone hands you "the model has a big window, why are we retrieving," you should answer from a framework, not a preference:

  1. Is the same prefix (system, tools, few-shots) sent on most turns? Yes → cache it: mark a stable prefix, keep it byte-identical, order volatile content after the breakpoint. This is the cheapest win and it is orthogonal to the next two.
  2. Does the knowledge needed vary per query, or is it a fixed small corpus? Varies, or the corpus is large/volatile → RAG: retrieve the few most relevant chunks, place them at the edges. Small, stable, fits comfortably → long-context can win on operational simplicity (no index to keep fresh).
  3. Does the conversation outgrow the budget over time? Yes → compaction: verbatim recent window plus a rolling summary, durable facts in a semantic store recalled by similarity.

These compose — a real agent caches the prefix, retrieves per query, and compacts history. The signal of seniority is naming all three as budget decisions on the same axis (dollars, milliseconds, correctness), not treating them as unrelated features.

Code-review red flags

The things that make you stop a PR:

  • The system prompt is truncatable or not pinned. A big retrieved doc can now evict your safety rules. Pins are non-negotiable; pinned overflow must raise, never silently drop.
  • The router does argmax with no fallback. Ships a silent workflow-collision bug on day one. Ask: "what does it do when the top two intents tie?" If the answer is "picks one," it is broken.
  • Memory is an unbounded list of turns. No compaction (overflow at turn N), no semantic store (amnesia across sessions). "Memory" that is just chat history is not memory.
  • Truncation on a byte offset with no elision marker. Garbage sub-word tokens, and a truncated list the model reads as complete — the "3 open tickets" bug from a cut-off 30.
  • The budget counts bodies but not the rendered prompt. Headers, delimiters, and role scaffolding are real tokens; a budget check that ignores them pushes callers over the hard limit.
  • Recall with no tenant filter, or a post-filter. Scope must be a pre-filter on which rows are candidates. A cosine match that surfaces another tenant's data is an incident, not a bug.
  • A synchronous summarizer LLM call on every turn. Doubles per-turn latency and cost to maintain a summary that rarely changed. Compaction belongs on eviction, embedding on write.
  • Recall-ordering the whole prompt including the cacheable prefix. Reshuffles the prefix, defeats prompt caching. The cache head must be exempt from the recall permutation.

Production war stories, read as mechanism

  • The 1M-window agent that got dumber. A team pasted the whole handbook in "because it fits." Accuracy fell, latency tripled. The cause was context rot plus lost-in-the-middle: the relevant paragraph was diluted across a huge window and usually landed in the weak middle. Retrieve-top-five-and-put-them-at-the-edges beat the giant context on quality and cost. The lesson a staff engineer draws: a bigger window is a bigger budget, not a strategy.
  • The refund that never happened. "Cancel my plan and refund the last charge" routed to the cancel pipeline; the refund evaporated with no error. It was an ambiguous tie (two intents at equal score) that an argmax router resolved by coin flip. The margin gate — refuse and clarify on a tie — made the whole class of bug disappear. The most valuable output of a router is often the refusal.
  • Amnesia at turn 12. A fixed buffer with no compaction dropped everything older off a cliff. A rolling summary on eviction plus a semantic store for durable facts held 40-turn conversations together. The demo always worked because the demo was short; the truth showed up at scale.
  • The leaked account ID. A shared semantic store with no tenant partition surfaced one customer's account number into another's prompt via a cosine match. Blast radius was every session, not one turn. Scoping recall by identity is the fix, and it is a pre-filter.

The signal an interviewer or review is listening for

When the prompt is "your agent has the right document in context but answers wrong, what do you check," the junior answer is "improve the prompt" and the staff answer is "position — recall is U-shaped, so I pull the actual assembled context for that turn, check whether the fact was present, at an edge or the middle, dropped by the budget, or lost to compaction, then reorder and retrieve fewer-better-ranked chunks." The reviewer is listening for three things: (1) you reach for the trace (included/dropped/truncated), not the wording; (2) you frame every choice — RAG, cache, compaction — as spending a finite billed budget; and (3) you name a tension unprompted. The strongest single signal at the staff/principal line is raising the recall-order vs cache-prefix conflict before you are asked — it proves you have run this against a real cache, because that tension only shows up in production. "Context engineering, not prompt engineering" said as a level claim, then backed by the budget/priority/ordering/routing/memory mechanics, is the move.

Closing takeaways

  1. The window is a finite, billed, quality-bounded budget you spend deliberately — dollars, milliseconds, and correctness bound it, and every context question (RAG, cache, compaction) is a spend decision on that budget.
  2. Selection, arrangement, and constraint are three different operations. Pins are a hard constraint that must fail closed; priority is soft selection; recall order is arrangement. Fusing them is where the bugs live.
  3. The router's most valuable output is the refusal. A confident wrong route is worse than an honest "which did you mean?"; the tie gate, not the classifier, prevents workflow collisions.
  4. Memory is a hierarchy and a liability. Buffer, compaction summary, semantic recall — and because it is durable, cross-session, and cross-tenant, it is the highest-blast-radius surface in the phase. Scope it, decay it, redact it.
  5. Instrument the assembler or you are debugging blind. The included/dropped/truncated/route accounting is the single most useful trace an agent emits; "why did it forget" is a query, not a guess.
  6. Get the decisions right and the components are commodity. The tokenizer, embedder, and summarizer are swappable seams. The judgment — spend the budget, order for recall, refuse to collide, tier the memory — is the thing you are hired to own.