« Phase 16 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 16 — Hitchhiker's Guide
30-second mental model
A voice agent is a chat agent with a stopwatch strapped to it. Latency is the product: a spoken turn can't be regenerated. Three mechanisms: endpointing (is the user done? — silence timer, frontier is a semantic turn-detector), a turn-taking state machine (LISTENING → THINKING → SPEAKING → LISTENING), and barge-in (user talks over agent → cancel TTS → LISTENING). The budget metric is end-of-speech → first-audio (~<800 ms), measured at the worst turn. WebRTC is the transport under it all.
The numbers to tattoo on your arm
| Number | Meaning |
|---|---|
| < ~500 ms first-audio | feels snappy |
| < ~800 ms first-audio | feels natural (the budget) |
| > ~1 s | user thinks you didn't hear them, starts over |
| latency = STT + LLM TTFT + TTS TTFB | cascade is a sum; use first-token/byte |
barge-in: spoken < total | proof the agent stopped mid-sentence |
| check the worst turn, not the mean | one bad turn is remembered |
Framework one-liners
- LiveKit Agents — the real-time agent framework (rooms/tracks/turn-detector); the JD is a Rust SDK role.
- Silero VAD — the standard voice-activity detector.
- OpenAI Realtime / Gemini Live — speech-to-speech models (audio in, audio out).
- WebRTC — the low-latency lossy-media transport (UDP, jitter buffers).
- Cascaded (STT→LLM→TTS) vs speech-to-speech — control vs latency/naturalness.
War stories
- The agent that talked over the customer. No barge-in; it finished its 20-second policy recital while the caller shouted "stop." Unusable.
- The clipper. 200 ms endpointing cut people off mid-pause; orders came in half-complete. Bumped the timer and added semantic turn-detection.
- The p50 that felt fine and the p95 that didn't. One slow turn per call is one bad turn the user remembers; the budget is worst-case.
Vocabulary
VAD · endpointing / turn-detection · utterance_end / speech_start · turn-taking state machine · barge-in · TTFT / TTFB · end-of-speech → first-audio · cascaded vs speech-to-speech · WebRTC / jitter buffer.
Beginner mistakes
- Treating voice as chat + TTS bolted on.
- No barge-in (agent talks over the user).
- A fixed endpointing silence that's too short (clips) or too long (sluggish).
- Measuring total latency instead of first-audio; measuring the mean instead of the worst turn.
- Trying to run real-time audio over HTTP instead of WebRTC.
- Assuming speech-to-speech is strictly better (you lose control + tool use).