« Phase 16 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 16 — Core Contributor Notes: Real-Time & Voice Agents

Our lab is a deterministic, single-threaded simulation of turn-taking. The real systems it mirrors — LiveKit Agents, Silero VAD, the OpenAI Realtime and Gemini Live APIs, and WebRTC underneath all of them — are concurrent, networked, and lossy. This is the maintainer's-eye view of how the production stack actually implements these mechanisms, where our miniature deliberately simplifies, and the sharp edges a committer on such a system knows.

VAD: what Silero actually returns, and where endpointing really lives

Our Frame.is_speech is a boolean. A real VAD like Silero VAD is a small neural network that runs on short windows (commonly 30 ms of 16 kHz audio) and returns a speech probability, not a boolean. That difference matters at the source level: the consumer thresholds the probability (with separate on/off thresholds — hysteresis — to stop flapping around the boundary) and often applies a minimum-speech-duration and minimum-silence-duration filter before it will declare a segment. Crucially, VAD alone is not endpointing. In LiveKit's agent stack the end-of-turn decision is a separate concern layered on top of VAD, STT partials, and increasingly a dedicated turn-detector model that reads the words and prosody to judge whether the utterance is semantically complete — the thing our warmup calls "the frontier." Our lab collapses VAD-output and the silence-timer endpointer into one deterministic rule; the real system splits them and adds a learned model on top, precisely because a fixed silence threshold has no good universal value.

LiveKit Agents: the framework shape

LiveKit Agents organizes a voice agent around a session that owns the media tracks and a pipeline of pluggable STT, LLM, and TTS components (the cascaded shape) or a single realtime model (the speech-to-speech shape). The non-obvious source-level decisions are all about cancellation and streaming: every stage is an async stream, the framework has an explicit interruption/interrupt path, and when the user barges in it must tear down the in-flight LLM generation and the in-flight TTS synthesis and stop the outbound audio track — the same "one frame does two jobs" logic our lab models, but spread across async tasks and a network. Our miniature computes spoken < total synchronously; the real framework has to reason about how many audio frames were actually flushed to the track before the cancel landed, which is genuinely harder because the answer depends on jitter-buffer and playout state it does not fully control.

WebRTC: the transport our lab does not build

We never touch the transport; production lives and dies by it. WebRTC carries audio as RTP over UDP, encrypted as SRTP, with signaling (SDP offer/answer, ICE for NAT traversal) to establish the session. The parts that make it real-time and that a systems engineer owns: the jitter buffer (packets arrive out of order and unevenly, so a small adaptive buffer smooths playout at the cost of a few milliseconds of latency — a direct latency-vs-smoothness knob), packet-loss concealment (synthesizing plausible audio for a dropped packet), echo cancellation and noise suppression (so the agent's own TTS does not get transcribed as user speech — get this wrong and the agent barges in on itself), and clock synchronization between endpoints. This is why the LiveKit role is a Rust SDK role: it is buffers, codecs (Opus), loss handling, and clock sync, with an agent on top. Our lab abstracts all of it into a clean monotonic tick, which is honest about what we are teaching (the decision logic) and honest about what we are not (the media systems work).

OpenAI Realtime and Gemini Live: the speech-to-speech side

The Realtime-class APIs collapse STT+LLM+TTS into one model reached over a persistent bidirectional connection (WebSocket, or WebRTC for the browser). The design decisions worth knowing: they emit server-side VAD and turn events so the model itself signals end-of-turn and can be interrupted; they support function/tool calling mid-conversation; and they stream audio out in chunks you must be ready to cancel when the user speaks. The evolution here is instructive — these APIs moved toward server-managed turn detection and explicit interruption events precisely because client-side-only endpointing and uncancellable generation were the two things that made early voice demos feel broken. The tradeoff our warmup names is real at the API level: you gain latency and prosody, you lose the inspectable transcript boundary where cascaded systems insert guardrails, RAG, and their own tool validation.

The turn-taking research this rests on

The mechanism is not arbitrary engineering; it encodes decades of conversation-analysis research (Sacks, Schegloff, and Jefferson's work on turn-taking established that human conversation minimizes gap and overlap, and that sub-second timing is a hard perceptual expectation, not a preference). That is why the 800 ms budget and the intolerance of talk-over are non-negotiable: they are properties of human listeners, not of the software. A contributor who knows this treats the latency budget and barge-in as fixed requirements handed down by human perception, and treats every engineering decision as serving them.

What our miniature deliberately simplifies

To keep the lab deterministic and dependency-free, we cut, in order of significance: (1) concurrency — real STT/LLM/TTS run as overlapping async streams; we sequence them on a tick; (2) the network — no jitter, loss, NAT, or codecs; a clean frame stream instead of SRTP; (3) probabilistic VAD and the semantic turn detector — a boolean and a silence timer instead of a neural VAD plus a learned end-of-turn model; (4) partial-hypothesis STT — real STT emits revising partial transcripts that endpointing and barge-in both consume; ours transcribes an already-segmented utterance; (5) the media plane entirely — we model the agent's decisions, not the transport that a production role would spend most of its time in. None of these omissions change the control logic — the endpointer fold, the guarded transition table, the spoken < total barge-in proof, the first-audio-worst-turn budget — which is exactly the part that transfers to a real LiveKit or Realtime deployment. Describe the pattern accurately, do not invent version-specific API names or exact model thresholds; those drift, and the shape is what endures.