« Phase 04 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 04 — Core Contributor Notes: Context Engineering

Our four primitives — assembler, order_for_recall, IntentRouter, TieredMemory — are each a stripped-down copy of a real system with a real commit history. This is what those systems actually do under the hood, where their designs turned, and what our miniature deliberately leaves out.

LangChain memory: the abstraction that got deprecated

The tightest mirror of TieredMemory is the family LangChain shipped early and then walked back. The originals were ConversationBufferMemory (keep everything verbatim — our buffer with no bound), ConversationBufferWindowMemory (keep the last k turns — our deque(maxlen) working tier), ConversationSummaryMemory (replace history with a running LLM summary — our compaction with no verbatim window), and ConversationSummaryBufferMemory (a verbatim window plus a rolling summary of what falls out of it — exactly our two-tier buffer + summary). They all subclassed a BaseMemory with load_memory_variables() and save_context().

The maintainers deprecated this whole family, and the why is the interesting part. BaseMemory was implicit, mutable, per-chain state hidden inside the chain object. You could not easily inspect what was in it, compose two chains that shared it, checkpoint it, or reason about it as data — it was a side effect. The migration went two hops: first to LCEL with RunnableWithMessageHistory (history passed explicitly as a runnable input), then to LangGraph, where conversation state is an explicit typed graph state persisted by a checkpointer keyed by a thread_id. That evolution is the same lesson our lab opens with: memory is your data structure, not the model's, and it should be explicit state you own, inspect, and test — which is why TieredMemory is a plain object with visible buffer, summary, and semantic fields rather than a hook buried in a chain. A committer's summary: LangChain moved memory from an implicit chain side effect to explicit checkpointed graph state, and our miniature is the explicit-state version by design.

semantic-router: prototypes, encoders, and per-route thresholds

IntentRouter is a from-scratch semantic-router (Aurelio Labs). The real library's shape: you define Route objects, each carrying a name and a list of example utterances; an Encoder (OpenAI, HuggingFace, FastEmbed, and others) embeds those utterances once; a RouteLayer / router object embeds the incoming query and scores it against the precomputed route vectors, picking the best above a threshold. Two source-level facts a contributor knows:

• Route embeddings are precomputed and cached, exactly as we precompute a bag-of-words prototype per intent in register. The hot path is query-embed plus a similarity sweep, not re-embedding the routes.
• Thresholds are typically per-route, not global, because different routes have different natural score distributions — a well-separated intent can afford a high bar, an overlapping one cannot. Our single threshold is the simplification; the pattern in the real thing is a threshold you can set (and tune/fit) per route.

The thing our router adds that vanilla similarity routing does not emphasize is the ambiguous_tie margin gate — refusing when the top two are within margin. Pure threshold routing catches "matches nothing" but happily returns a confident coin-flip when two routes tie. The "refuse on a tie" discipline is closer to a dialogue manager's disambiguation step (NeMo Guardrails flows, Rasa's fallback/two-stage classifiers) than to a bare encoder router — and it is the branch Citi's "avoid workflow collisions" language is really asking for.

MemGPT / Letta: paging driven by the model itself

TieredMemory's eviction-and-summarize is the deterministic cousin of MemGPT (the paper; Letta is the company and OSS implementation around it). MemGPT frames the context window as an OS's main memory and the vector store / recall storage as disk, with the model paging between them. The non-obvious design decision: MemGPT lets the LLM itself manage the boundary via function calls — it exposes memory-edit and memory-search functions, and when a "memory pressure" warning fires as the context fills, the model chooses what to evict, summarize, and page out. Recursive summarization compresses evicted content. So where our add_turn evicts deterministically at a fixed buffer_size and folds via an injected pure function, MemGPT evicts on a pressure signal and folds via model-issued tool calls. Our version is testable and reproducible; theirs is adaptive and self-directed. The tiering — main/working context vs external/recall context — is identical; the control policy is the difference.

Mem0: extraction and reconciliation, not append-only

Our remember appends a (text, embedding) pair; recall returns cosine top-k. Mem0's insight is that this is not enough at scale, and the gap is instructive. Mem0 does not store raw turns — it runs an extraction pass (an LLM pulls salient facts out of the conversation) and then a reconciliation pass against existing memories: does this new fact add, update, supersede, or contradict something already stored? It issues add/update/delete operations accordingly, over a vector store (and optionally a graph store for relationships). The sharp edge our miniature hides: an append-only semantic store accumulates duplicates and contradictions — store "user prefers email" on turn 5 and "user prefers SMS" on turn 40 and both sit in memory forever, and recall may surface the stale one. Reconciliation (and dedup/decay, which our extensions note calls out) is what keeps a long-lived memory coherent rather than a growing pile of half-true facts.

Claude Code and Anthropic prompt caching: compaction and the prefix

Claude Code is the shipped three-tier hierarchy end to end: live turns in context (working), CLAUDE.md as durable pinned project facts (long-term), and auto-compaction that summarizes the conversation as it approaches the context limit — plus the manual /compact. That is our buffer + semantic/pinned facts + rolling summary, in a real product, and it is worth studying as the reference for what compaction preserves (it is instructed to keep decisions, file paths, and open threads, not to summarize uniformly).

Prompt caching is the other real system our order_for_recall collides with (see the Principal Deep Dive for the tension). The prefix rules a contributor must respect, described as patterns rather than exact figures since they are model- and version-dependent: you mark a cache breakpoint (cache_control) after a prefix; the prefix must be byte-identical across calls to hit; there is a minimum cacheable prefix length (on the order of a thousand tokens for larger models, model-dependent — check current docs, do not hard-code it); and the cache entry has a short TTL (a few minutes, refreshed on hit). The design consequence is the ordering rule our warmup states: stable content (system, tools, few-shots) first so it forms a reusable prefix, volatile content (the task) last. Anything that reshuffles the prefix — like naively recall-ordering the whole prompt — defeats the cache.

What our miniature deliberately simplifies

Held against the real systems, the honest ledger of what the lab trades away:

• count_tokens is ceil(len/4), not a BPE tokenizer. Real systems call the model's own tokenizer (tiktoken, the HF tokenizer) so the budget check matches billing exactly. Our estimate can be off by tens of percent on code or non-English text.
• embed is the hashing trick (feature hashing), not a learned model. It captures lexical overlap only — "car" and "automobile" are orthogonal. A real encoder captures meaning; the mechanism (text → vector, compare by cosine, top-k) is unchanged, which is why Phase 05 swaps the seam without touching recall.
• The router is bag-of-words cosine, not a trained classifier or an encoder. semantic-router uses real embeddings; the threshold/margin/fallback logic is identical.
• The summarizer is an injected pure function, not an LLM. This is the testability seam the whole track uses — eviction→summary is deterministic so a unit test can assert it, and the real LLM summarizer drops in at one call site.
• The memory is append-only, single-tenant, in-process. No reconciliation (Mem0), no per-tenant partition, no persistence, no decay.

The point of the miniature is that every decision — pin, drop, truncate, order, route-or-clarify, evict-and-summarize, recall-by-similarity — is the production decision; only the components behind each seam are stubbed. Get the decisions right and you can read any of these libraries' source in an afternoon, because you already know the shape they implement.