« Phase 16 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 16 — Deep Dive: Real-Time & Voice Agents

The load-bearing idea of this phase is that a voice agent is not an AI system with a timing problem bolted on; it is a soft-real-time streaming state machine whose controlling variable is time, and whose AI happens to live in one of its stages. Everything hard here — endpointing, barge-in, the latency budget — falls out of that reframing. This deep dive walks the actual data structures, the algorithms that drive them, the invariants they must hold, and a step-by-step trace of one conversation.

The frame clock and why determinism is possible

The entire pipeline is driven by a stream of Frame values, each carrying is_speech: bool and a monotonic timestamp (a tick, not wall-clock). This is the single most important design decision in the lab: by making the clock an explicit sequence of integer ticks rather than reading time.time(), the whole system becomes a pure function of the frame sequence. Same frames in, same events out, every run. Real voice stacks cannot do this — audio arrives on a wall clock over a lossy network — but the decision logic they run is exactly this pure function, which is why interviews probe the logic and not the DSP. The tick clock is the seam that lets you unit-test a real-time system without a microphone.

The endpointer: a fold over frames

Endpointer.feed(frame) is a left fold with O(1) state and O(1) work per frame. Its state is small: whether we are currently inside an utterance, and the timestamp of the last observed speech frame. The rule:

• First is_speech=True frame while not in an utterance → emit speech_start, mark in-utterance, record last_speech_ts.
• Each subsequent speech frame → update last_speech_ts.
• A silence frame whose timestamp - last_speech_ts >= silence_ms while in-utterance → emit utterance_end, leave the utterance.

The critical invariant is that utterance_end fires exactly once per utterance, and only after silence_ms of trailing silence. A common miniature bug is emitting on the first silence frame (clips every mid-word gap) or re-emitting because the in-utterance flag was never cleared. Note what the endpointer is not: it has no notion of words or meaning. It is a timer over a boolean stream, which is precisely why the fixed-threshold tradeoff (too-short clips, too-long lags) is unavoidable at this layer and why semantic turn detection is a different, higher layer, not a tuning of this one.

The turn-taking machine as an explicit transition table

TurnManager is a four-state machine — LISTENING, THINKING, SPEAKING — driven by typed events through dispatch(event). The forward table is total and explicit:

Any event that is not in the table for the current state is an InvalidTransition — surfaced loudly, never silently absorbed. This is the same discipline as the durable-workflow engine (Phase 08): the control flow is data (a table), not scattered if branches, so an illegal sequence such as playback_done arriving while still THINKING is a caught bug rather than a corrupted "whose turn is it" that makes the agent talk over itself. Modeling whose-turn-it-is as an enum with a guarded transition function is the mechanism that makes the alternation of a conversation provable instead of hoped for.

Barge-in: one frame doing two jobs

Barge-in is the only backward edge, and its mechanism is where the timing math bites. While SPEAKING, a Playback is in flight: it has total tokens, a fixed rate (ticks per token), and a start_ts. It would finish at start_ts + total * rate. When a user speech frame arrives at cancel_ts strictly before that finish time, it is a barge-in. The engine computes how many tokens actually reached the speaker:

spoken = min(total, floor((cancel_ts - start_ts) / rate))

and the load-bearing assertion is spoken < total — the arithmetic proof that the agent stopped mid-sentence. That same frame then does double duty: it drives speech_start (SPEAKING → LISTENING) and begins the user's new utterance in the endpointer. The half-spoken response is discarded outright; there is no resume, because a user who interrupts has changed the state of the world and the buffered tokens are now stale. Ordering matters here: cancel the playback, transition the machine, then let the endpointer consume the frame — reverse that order and you either lose the frame or transition on a playback that is still "playing."

The latency measurement and why it is first-audio

run_pipeline measures each turn's end-of-speech → first-audio latency: the delta from the utterance_end timestamp to the timestamp at which the first audio chunk is emitted (the THINKING → SPEAKING edge). It decomposes as STT_decode + LLM_TTFT + TTS_TTFB — a sum across the cascade because the stages are sequential per turn. The subtlety is first-token/byte, not total: the agent starts speaking as soon as the first chunk exists and streams the rest, so total generation time is irrelevant to felt latency. within_budget then checks the worst turn against the threshold, not the mean — the tail discipline from Phase 00, because one 1.2 s turn in a call of forty good ones is the turn the caller remembers and talks over.

Worked trace: a two-turn call with a barge-in

Frames [S S S _ _ _ (gap≥silence_ms) ...] with silence_ms threshold:

1. Frame 0 is_speech: endpointer emits speech_start; TurnManager stays LISTENING (it only cares about utterance_end).
2. Frames 1–2 speech: last_speech_ts advances to tick 2.
3. Silence accrues; at the frame where t - 2 >= silence_ms, endpointer emits utterance_end → dispatch(utterance_end): LISTENING → THINKING.
4. Cascade runs; at first audio, dispatch(response_ready): THINKING → SPEAKING; latency recorded = first_audio_ts − utterance_end_ts.
5. Playback of 10 tokens at rate 3 begins at start_ts; would finish at start_ts + 30.
6. Turn two: a user speech frame lands at cancel_ts = start_ts + 12. spoken = floor(12/3) = 4 < 10 → barge-in proven. Playback cancelled, dispatch(speech_start): SPEAKING → LISTENING, and that frame opens the new utterance. The unspoken 6 tokens are dropped.

Every number here is reproducible because nothing sampled a clock.

Why the naive approach fails at the mechanism level

The naive voice agent is "STT → LLM → TTS, mutate a dict of flags, sleep on a real clock." It fails structurally, not cosmetically: mutable boolean flags with no total transition function let an out-of-order event (a playback_done racing a barge-in) silently flip two flags and produce an agent that is both listening and speaking; measuring total latency hides that the felt number is first-audio; and a wall-clock endpointer is untestable and non-reproducible, so the too-eager/too-patient bug can never be pinned in CI. The explicit clock, the fold-based endpointer, the guarded transition table, and the spoken < total proof are not stylistic choices — each one closes a specific class of real-time bug that the naive version ships to production and only discovers when a customer is talked over.