« Phase 01 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy: should you write a kernel at all?
• 2. A decision framework for "does this belong in the kernel?"
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy: should you write a kernel at all?

The honest answer for most banks is neither pure build nor pure buy: wrap.

The wrap is where the value is, and it is a specific value: your interface is the thing thirty teams code against, so your interface is where invariants can live. Budgets, session identity, memory scoping, evidence emission and tenant propagation go in your layer; graph execution, streaming and checkpoint storage come from the framework.

Two rules keep the wrap from becoming a second framework:

1. The wrapper adds invariants, never features. The moment it grows a nicer way to define tools, you own a framework.
2. Escape hatches are explicit and logged. A team that needs the raw runtime gets it, with a recorded exception, so you can see how often your abstraction is wrong.

Regardless of which you pick, write the transition table by hand once. It takes an hour, it becomes the diagram in your design doc, and it is the thing that makes "an agent cannot act after it completed" a statement you can defend.

2. A decision framework for "does this belong in the kernel?"

Someone proposes a feature. Four questions, in order:

1. Would its absence in one agent become the platform's incident? Step budgets: yes (capacity). Prompt quality: no (that team's problem). This is the primary test.
2. Does it need to be true across all agents to be true at all? Tenant propagation, evidence emission, identity. A control that thirty teams implement thirty ways is not a control.
3. Is it a policy or a capability? Policies (what is allowed, what is bounded, what is recorded) belong in the kernel. Capabilities (a nicer retrieval helper, a prompt library) belong in a library teams may ignore.
4. Can it be verified from the outside? If you cannot write a test at the kernel boundary proving the invariant holds, it is not an invariant — it is a convention, and it will drift.

Applied to real requests:

3. Review red flags

In a design document

• No statement of where session state lives. Ask it first, every time.
• A lifecycle drawn as a diagram with no enumeration of illegal transitions.
• "Memory" as a single component with no scope model.
• HITL described as a UI flow with no mention of how the approval enters the execution chain.
• Sticky sessions described as a requirement rather than an optimization.
• No answer to "what happens if the worker dies during a tool call?"
• Budgets listed as max_tokens only.
• A memory write path with no classification, provenance or TTL.

In code

# Red flag: booleans as a state machine
if session.is_running and not session.is_done: ...

# Red flag: state in the worker
SESSIONS: dict[str, Session] = {}          # gone on the next deploy

# Red flag: last-write-wins
store[session_id] = snapshot                # two runs interleave into one chain

# Red flag: salted hash in a ring
replica = replicas[hash(session_id) % len(replicas)]   # twice wrong

# Red flag: budget checked after the call
decision = model(pad.render())
if steps > MAX: break                       # already paid for it

# Red flag: unknown tool as an exception
result = tools[decision.tool](args)         # KeyError kills a conversation

# Red flag: compaction on the persisted record
snapshot.steps = snapshot.steps[-2:]        # the audit trail just lost eight steps

# Red flag: rank-then-filter in memory
facts = rank_all(tags)[:10]
return [f for f in facts if f.owner == caller]   # leaks existence; one refactor from leaking content

In an incident review

• "We lost the conversations" → where was state?
• "We didn't know it was looping" → is there a budget-breach rate alert, not just a breach log?
• "We can't reproduce it" → is the chain complete enough to replay?

4. Production war stories

The upgrade that could not happen. Every agent team imported the framework directly. A major version bump changed the state schema. Twelve teams, twelve migrations, six months, two of which never migrated and were frozen on an unsupported version. The wrap exists for this. Your interface is the thing you can version.

The approval with no record. HITL was implemented in the Teams bot: the bot showed a card, the human clicked Approve, the bot called the agent's resume endpoint. Audit asked who approved a release and found the approval in the channel's logs, the execution in the agent's logs, and no join key between them. Put the pause in the kernel; the answer belongs in the chain.

Memory that remembered a lie. An agent summarized a retrieved document into long-term memory. The document contained an injected instruction that produced a false "fact" about a counterparty's limit. It persisted, surfaced in three later sessions for other users of the same tenant, and influenced advice. Memory writes need the same guardrails as actions, plus provenance, plus a quarantine for facts written during runs that touched untrusted content.

The quadratic bill. A run that averaged 22 steps. Nobody had done the arithmetic. Compaction plus consolidating tools to shorten runs cut token spend 70% with no quality loss on the eval set. The arithmetic is one line; do it before you optimize anything else.

Two workers, one session. A queue redelivered a message. Both workers ran the same step; the store was last-write-wins. The resulting chain showed a sequence of actions that no single execution ever performed — which was discovered during, of all things, a model-risk validation. OCC is four lines.

The deploy that halved the cache hit rate. Replicas were removed from the ring at SIGTERM instead of drained. Every rolling deploy reassigned a third of sessions, and the retrieval cache hit rate visibly stepped down for twenty minutes after each release. Nobody connected the two for a quarter.

5. The interview signal

Signal 1 — you say "kernel" and mean it. The candidate who explains the runtime as an OS for probabilistic programs, and can map process→run, memory limit→token budget, syscall→tool dispatch, process table→session store, is signalling that they have thought about why the responsibilities divide the way they do, not just which library they used.

Signal 2 — you volunteer the checkpointed-vs-durable distinction. Unprompted. "Checkpointing gives me resumability, not exactly-once; effects are the action gateway's problem." This single sentence separates people who have run agents in production from people who have built demos.

Signal 3 — you treat memory as a data store. Classification, residency, erasure, information barriers, retention. In a regulated JD this is the highest-value unprompted observation available, because it is the thing that turns a memory feature into an audit finding.

Signal 4 — you distinguish what the kernel imposes from what it supports. "Lifecycle, budgets, state, evidence — imposed. Loop shape — supported." It shows you have thought about adoption, not only correctness, which is the actual failure mode of internal platforms.

Signal 5 — you reach for arithmetic. nb + a·n(n−1)/2, 1/n versus (n−1)/n. Numbers end arguments.

Anti-signals, worst first:

• Session state in process memory, unremarked.
• "We use LangGraph" as an answer to "how does your runtime work?"
• Treating sticky sessions as a correctness requirement.
• No answer to "what happens if the worker dies mid-tool-call?"
• One Memory abstraction, no scopes.
• Describing HITL purely as a UI concern.
• Believing checkpointing implies exactly-once.

The question to ask them: "Where does session state live today, and what happens to in-flight runs during a deploy?" The answer tells you the maturity of the platform in one sentence, and asking it signals that you know which sentence to ask for.

6. Mentoring notes

Three exercises that build the judgment faster than reading:

1. Make them draw the state machine before writing code. Then ask for three illegal transitions and what bug each prevents. Engineers who cannot name the bug will write the boolean version.
2. Have them kill the worker mid-run. Literally: kill -9 during a tool call, then resume. Whatever they believed about durability, they will now believe something more accurate.
3. Give them a run that costs $4 and ask why. They will look at the model. Walk them to the scratchpad arithmetic. This is the fastest way to install the habit of computing before optimizing.

And one framing worth repeating to a team: the kernel is the only place where "we do this for every agent" is cheap. Every invariant you decline to put there is one you will later ask thirty teams to implement, during an audit, under time pressure. That is the argument that wins the prioritization conversation, and it is worth having early.