« Phase 01 · Warmup · Track Overview

Principal Deep Dive — Architecture, Tradeoffs & Blast Radius

Table of Contents

• 1. The three tradeoffs of a kernel
• 2. Where to put the loop
• 3. Scaling envelope
• 4. Failure modes and blast radius
• 5. The state-store choice
• 6. Memory as a governance surface
• 7. Decisions that look wrong but are intentional
• 8. What changes at 10×

1. The three tradeoffs of a kernel

Tradeoff 1 — enforcement vs adoption. Every invariant the kernel imposes is a thing an agent team cannot do. Impose too little and you have a library nobody's SLO depends on; impose too much and teams route around you, which is worse than not having a platform because now you have a platform and shadow agents.

The resolution is to be absolutist about a small set and permissive about everything else. The small set: lifecycle, budgets, state location, evidence emission. Everything else — loop shape, prompt strategy, tool composition, memory usage — is the team's. The test for whether something belongs in the small set: would its absence in one agent become the platform's incident? A team that writes bad prompts owns its own quality problem. A team without a step budget owns your capacity problem.

Tradeoff 2 — checkpoint frequency. Checkpoint every step and you get fine resumability at the cost of a store write per step (latency, and a write-throughput ceiling at fleet scale). Checkpoint every N steps and you re-execute up to N steps on resume — which, for non-idempotent tools, is not a performance question but a correctness one.

The resolution is side-effect-aware checkpointing: always checkpoint immediately before and after a mutating tool call; batch checkpoints for read-only steps. The kernel knows the tool's side-effect class from the registry (Phase 09), so this is a policy, not a guess. It also buys back most of the write throughput, since read steps dominate.

Tradeoff 3 — memory richness vs governability. Long-term memory makes agents dramatically better and creates a data store nobody classified, with contents derived from customer conversations, that persists across sessions and can surface in another user's context. In a bank that is a data problem before it is a quality feature.

The resolution: memory writes are typed, scoped, attributed, and expiring. A fact carries its scope, its owner, its provenance (which session wrote it), and a TTL. Facts derived from customer data inherit that classification. Nothing is written to app scope by an agent, ever — that requires a human. This costs some capability and is the difference between a memory system you can put in front of Internal Audit and one you cannot.

2. Where to put the loop

Three viable placements, and the choice determines your operational model:

For a bank platform serving both interactive and long-running work, the honest answer is two execution modes over one kernel: synchronous for interactive runs under a latency budget, queue-driven for anything that can suspend. The kernel's interface — snapshot in, decision out, snapshot out — is identical in both, which is precisely why the state model must be externalized from day one. Retrofitting a second execution mode onto an in-memory kernel is a rewrite.

The durable-engine option is worth taking when the action half dominates: multi-step money movement with compensation. That is Phase 10 territory, and the sane architecture is often a synchronous kernel that hands a saga to a durable engine, rather than one engine running everything.

3. Scaling envelope

Two non-obvious ones:

Session-store writes are the real throughput ceiling. At 1 000 concurrent runs averaging one step per 2 seconds, that is 500 writes/second of a document that grows with the run. A naive "store the whole snapshot every step" design writes the entire step history each time — \( O(n^2) \) bytes over a run, the same quadratic that bit the token cost. The fix is an append-only step log plus a small mutable header: steps are appended once, the header (state, version, counters) is updated per step. Same resumability, linear bytes.

Session-id shape matters. Sequential ids concentrate writes on one partition in most stores and create a hot spot on the ring. Use a random or hashed prefix — and note this is a case where the identifier scheme is a scaling decision, made once, cheap at design time, expensive later.

4. Failure modes and blast radius

Memory poisoning deserves the attention. An agent that writes to long-term memory can be induced — by an injected instruction in a retrieved document — to write a false fact that persists and influences later sessions, possibly for other users of the same tenant. Unlike a prompt injection that affects one run, this one is durable. Controls: memory writes go through the same guardrails as actions (Phase 11), every fact carries the session that wrote it, and a fact written during a run that touched untrusted content is quarantined until reviewed. Most platforms discover this after the fact.

5. The state-store choice

The decision usually goes to Postgres in a bank, for a non-technical reason that is nonetheless correct: the operating model, backup, DR, and audit story already exist. A platform that introduces a novel datastore also introduces a novel set of conversations with four other teams, and the technical advantage rarely pays for that.

The genuinely important part is the schema shape, not the engine: immutable append-only steps plus a small mutable header with a version column. That shape is portable across all four.

6. Memory as a governance surface

The JD asks for memory architecture and, separately, for auditability and data residency. Those requirements meet inside the memory system, and most designs miss it.

Questions the design must answer, before the first fact is written:

1. Classification. A fact derived from a customer conversation carries the conversation's data classification. Does the store know that? Can it answer "show me every fact derived from restricted data"?
2. Residency. If the tenant's data may not leave a jurisdiction, the memory store is in that jurisdiction — including its backups and its replicas.
3. Right to erasure. A customer exercises a deletion right. Facts derived from their data must be findable and deletable. That requires provenance from day one; it cannot be reconstructed.
4. Information barriers. Two desks that may not share information must not share a memory scope. tenant may be too coarse — the partition may need to be the desk.
5. Retention. Episodic memory of an investigation is a business record with a retention period, which may be longer and shorter than you want (delete-by is as binding as keep-for).

The architectural consequence: memory is a first-class data store with an owner, a classification, a retention policy and a DSAR path — not a cache the agent team manages. Teams that treat it as a cache get an audit finding on their first review.

7. Decisions that look wrong but are intentional

acting → completed is illegal. Looks like needless ceremony and one extra model call. It prevents an agent returning a raw tool result as an answer with no recorded reasoning, which is both a quality bug and an evidence gap — the chain would show a result and no explanation of why it answered the question.

The kernel does not retry tools. Looks like a missing feature. Retry policy depends on the side-effect class and the idempotency key, both of which live at the action gateway. A kernel that retries independently will one day retry a payment. The kernel's job is to observe the failure and let the model or the gateway decide.

Compaction can leave the pad over budget. Looks like the function does not do its job. Keeping the recent window is more important than the budget; the token budget will stop the run if it truly cannot proceed. An amnesiac agent produces confidently wrong output, which is worse than a failed run.

Two suspension states (waiting_input and suspended). Looks redundant — both mean "not running." They have different causes (agent-initiated vs kernel-initiated), different resume paths (needs an answer vs does not), and different SLO treatment (time in waiting_input is not platform latency; time in suspended is). Collapsing them makes your latency metrics lie.

Episode tags are sorted. Looks cosmetic. Without it, set iteration order makes two identical runs produce different episodes, and a "deterministic" test fails intermittently on another machine.

The kernel owns HITL rather than the channel. Looks like a UI concern. Putting the pause in the kernel means the human's answer lands in the execution chain, attributable and timestamped, and the run's identity is continuous across the pause. Channel-owned approvals produce a chain with a hole in it exactly where the interesting question is.

8. What changes at 10×

At 20 agents and 100 concurrent runs, the lab's design is close to what you would ship. At 200 agents and 5 000 concurrent runs:

• Append-only step storage becomes mandatory (§3), not an optimization.
• Checkpointing becomes side-effect-aware (§1), or the store is your bottleneck.
• Two execution modes (§2) — synchronous and queue-driven — because interactive latency and hours-long approvals cannot share one path.
• Memory needs lifecycle management: TTLs, compaction of the memory store itself, and a process for retiring facts. Unbounded semantic memory degrades retrieval quality long before it degrades storage cost.
• Budgets become per-tenant and dynamic, sourced from the control plane rather than per-kernel constants, and enforced against a shared pool.
• The execution chain outgrows the session store. Steps go to the trace backend and object storage; the session store keeps the header and a pointer. The join key discipline from Phase 00 is what makes this survivable.
• Ring weighting appears, because the fleet becomes heterogeneous (GPU-adjacent workers, memory-heavy workers).

The seams to build early, all cheap now and expensive later: Step as an append-only record, side_effect_class on every dispatch, tenant and session_id on every emitted artifact, provenance on every memory write, and budgets read from a config object rather than hard-coded.