« Phase 16 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 16 Warmup — Real-Time & Voice Agents

Who this is for: you've built text agents; you've never built one that talks and listens in real time. By the end you'll have built the endpointer, the turn-taking state machine, barge-in, and the latency-measuring pipeline — the mechanisms LiveKit and the Realtime APIs hide.

Table of Contents

  1. Why voice changes the contract
  2. The voice pipeline
  3. VAD: detecting speech
  4. Endpointing: is the user done?
  5. The turn-taking state machine
  6. Barge-in: the reverse gear
  7. The latency budget
  8. Cascaded vs speech-to-speech
  9. WebRTC: the real-time transport
  10. Common misconceptions
  11. Lab walkthrough
  12. Success criteria
  13. Interview Q&A
  14. References

1. Why voice changes the contract

In a text agent, latency is a quality-of-service concern: if the answer takes three seconds, the user waits and reads it. In a voice agent, latency is the product, for two reasons. First, a spoken turn cannot be regenerated — once the agent starts talking, the audio is in the air; the only "undo" is to stop, which the user experiences as being interrupted. Second, humans have deep, subconscious expectations about conversational timing: a gap over ~1 second reads as "they didn't hear me" and the user starts talking again. So voice engineering is dominated by time: deciding when the user finished, getting the first audio out fast, and stopping instantly when interrupted. Everything else is plumbing.

2. The voice pipeline

The classic cascaded voice agent is a pipeline: microphone → VAD → STT → LLM → TTS → speaker.

  • VAD (voice activity detection) decides, frame by frame, whether the user is speaking.
  • Endpointing decides when a whole utterance is finished.
  • STT (speech-to-text) transcribes the utterance.
  • LLM generates the response (streaming tokens).
  • TTS (text-to-speech) turns tokens into audio (streaming, cancellable).

Each stage adds latency, and they're sequential, so the budget is a sum (Phase 00). The lab simulates this whole pipeline with injected STT/LLM/TTS stand-ins and a tick clock, so the timing is exact and reproducible — the same "inject the model" discipline as every other phase, applied to three models at once.

3. VAD: detecting speech

Voice activity detection runs on tiny slices of audio (~10–30 ms) and returns a speech / no- speech verdict (a real one like Silero VAD returns a probability). It's the first filter: it tells the rest of the pipeline when there is speech to process and, crucially, feeds the endpointer. The lab models VAD's output — a boolean is_speech per Frame — rather than the DSP, because the endpointing logic that consumes it is what interviews probe, not the signal processing. The important property: VAD is per-frame and local; deciding an utterance is over is a separate, harder problem (§4).

4. Endpointing: is the user done?

Endpointing — deciding the user has finished their turn — is the hardest UX problem in voice, and the lab's Endpointer is where you feel it. The simple, deterministic rule: the user is done once you've seen silence_ms of trailing silence after their last speech frame. Emit speech_start on the first speech frame, utterance_end when the silence gap reaches the threshold.

The reason it's hard is that a fixed silence threshold trades one failure for another:

  • Too short (say 200 ms) and you clip people who pause mid-thought — "I'd like to order… [500 ms pause] …a large pizza" becomes two utterances, and the agent answers the first half.
  • Too long (say 1200 ms) and every reply feels sluggish — the user finished, and the agent just sits there.

There's no fixed value that's right for everyone, which is why the frontier is semantic / model-based turn detection: a small model reads the words and the prosody and asks "was that a complete thought?" (LiveKit ships a turn-detector model for exactly this). But the silence timer is the floor everything builds on, and it's what you implement. Note the endpointer is a pure function of the frame sequence — no clock, no randomness — so it's fully reproducible and testable.

5. The turn-taking state machine

A conversation is a strict alternation of turns, and the agent must always know whose turn it is. The lab's TurnManager is a small state machine: LISTENING → THINKING → SPEAKING → LISTENING.

  • utterance_end in LISTENING → THINKING (the user finished; go generate).
  • response_ready (first audio) in THINKING → SPEAKING (start talking).
  • playback_done in SPEAKING → LISTENING (finished; listen again).

An event that doesn't fit the current state (e.g. playback_done while still THINKING) is an InvalidTransition — surfaced loudly rather than silently corrupting whose-turn-it-is, because a turn-tracking bug is the kind of thing that makes an agent talk over itself. This is the same "model the control flow as an explicit state machine" discipline as the durable engine (Phase 08).

6. Barge-in: the reverse gear

Barge-in is the one place the turn machine runs backwards: while the agent is SPEAKING, if the user starts talking, the agent must cancel its own TTS playback mid-stream and snap back to LISTENING. An agent that keeps talking over you is unusable — barge-in is not a nice-to-have, it's table stakes for natural conversation.

The mechanism in the lab: during SPEAKING, a Playback is in flight (tokens played at a fixed rate); a user speech frame whose timestamp lands before the playback would finish is a barge-in — cancel the playback at that instant (recording how many tokens actually reached the speaker, the proof it stopped mid-sentence: spoken < total), dispatch speech_start (SPEAKING → LISTENING), and let that same frame begin the user's new utterance. The subtlety interviewers probe: what happens to the half-spoken response? It's discarded; the agent doesn't "resume" — it listens, because the user interrupted for a reason.

7. The latency budget

The number voice engineers live and die by is end-of-speech → first-audio: from the instant the user stops talking to the instant they hear the agent. That's the clock the user actually feels, and the budget is roughly:

  • < ~500 ms feels snappy;
  • < ~800 ms feels natural;
  • > ~1 s and the user thinks you didn't hear them and starts talking again.

It decomposes across the cascade: STT decode + LLM time-to-first-token (TTFT) + TTS time-to- first-byte (TTFB). Note it's first-token/byte, not total — you start speaking as soon as the first chunk is ready and stream the rest, exactly like text streaming (Phase 12) but now load-bearing. And you check the worst turn, not the mean (Phase 00's tail): one slow turn is a bad turn the user remembers. The lab's run_pipeline measures each turn's first-audio latency and within_budget checks them all against the budget.

8. Cascaded vs speech-to-speech

The pipeline in §2 is cascaded — separate STT, LLM, TTS models. Its advantages: you can swap each component, use any LLM, and inspect the transcript. Its cost: latency accumulates across three models, and information is lost at each text boundary (tone, emotion, overlap).

The alternative is a speech-to-speech / Realtime model (OpenAI Realtime, Gemini Live) that takes audio in and emits audio out directly, end to end. Advantages: lower latency (one model, no text round-trips) and it preserves prosody and can handle overlap more naturally. Costs: less control/inspectability, harder to plug in your own tools/RAG mid-turn, and a newer, less mature stack. The senior take: cascaded when you need control, tool use, and model choice (most enterprise voice agents today); speech-to-speech when latency and naturalness dominate and you can live with a more closed pipeline. LiveKit Agents supports both shapes.

9. WebRTC: the real-time transport

You can't ship real-time audio over plain HTTP request/response — you need a transport built for continuous, low-latency, lossy media. That's WebRTC: it does the media transport (over UDP with its own reliability), NAT traversal, jitter buffering (smoothing out packets that arrive unevenly), and echo cancellation. This is why LiveKit exists and why their role is a Rust SDK engineer: building a fast, correct real-time media SDK is systems work — buffers, codecs, packet loss, clock synchronization — with an AI agent on top. You don't implement WebRTC in the lab (it's a deep systems topic of its own), but you should know that it's the layer under the agent, that it's why voice is real-time and not request/response, and that the SDK's job is to hide its complexity while keeping latency low.

10. Common misconceptions

  • "Voice is just chat with STT and TTS bolted on." The real-time constraints — endpointing, latency budget, barge-in — reshape the whole design.
  • "Longer endpointing silence is safer." It trades clipping for sluggishness; there's no universally-right value, which is why semantic turn-detection exists.
  • "Barge-in is optional." An agent that talks over the user is unusable; it's table stakes.
  • "Total latency is the metric." It's first-audio latency (you stream), measured at the worst turn.
  • "HTTP is fine for audio." Real-time media needs WebRTC (UDP, jitter buffers, loss handling).
  • "Speech-to-speech is strictly better." It's lower-latency and more natural but less controllable and harder to combine with tools/RAG.

11. Lab walkthrough

Open lab-01-voice-turn-taking/. The scaffolding (Clock, Frame, Playback, Fake STT/LLM/TTS, latency dataclasses) is given; you implement Endpointer.feed (the silence rule), TurnManager.dispatch (the forward table + the barge-in reverse gear), within_budget (worst-case check), and run_pipeline (drive the frames through the cascade, measure end-of-speech → first-audio, handle barge-in). Run LAB_MODULE=solution pytest -v first, then match it. solution.py's main() scripts a two-turn customer-support conversation with a barge-in and prints the state trace + latency report.

12. Success criteria

  • Your endpointer fires speech_start / utterance_end at the right frames and not early.
  • Your turn machine transitions correctly and rejects invalid events; barge-in reverses SPEAKING → LISTENING and cancels playback.
  • You measure end-of-speech → first-audio and check the worst turn against the budget.
  • You can explain cascaded vs speech-to-speech and where WebRTC sits.
  • All 30 tests pass under lab and solution.

13. Interview Q&A

Q: Why is latency different in voice than in text? A: In text, a slow answer just makes the user wait. In voice, a spoken turn can't be regenerated (it's committed to the air), and humans have subconscious timing expectations — a gap over 1 s reads as "you didn't hear me." So latency is the product: you design to end-of-speech → first-audio (<800 ms), stream the first audio out ASAP, and measure the worst turn, not the mean.

Q: What is endpointing and why is it hard? A: Deciding when the user finished their turn. A fixed silence threshold trades failures: too short clips people who pause mid-thought, too long feels sluggish. There's no universally-right value, so the floor is a silence timer and the frontier is a semantic turn-detector model that reads the words and prosody to judge "was that a complete thought?"

Q: How does barge-in work? A: While the agent is SPEAKING, a user speech frame that arrives before playback finishes is a barge-in: cancel the TTS mid-stream (the proof is tokens-spoken < total), transition SPEAKING → LISTENING, and start the user's new utterance. The half-spoken response is discarded — the user interrupted on purpose. An agent without barge-in is unusable.

Q: Cascaded vs speech-to-speech? A: Cascaded (STT → LLM → TTS) gives control, model choice, tool use, and an inspectable transcript, at the cost of accumulated latency and lost prosody. Speech-to-speech / Realtime models are lower-latency and more natural but less controllable and harder to combine with tools/RAG. Most enterprise voice agents are cascaded today; pick speech-to-speech when latency and naturalness dominate.

Q: Why does LiveKit need a Rust SDK, and what's WebRTC doing? A: Real-time audio can't ride plain HTTP; WebRTC handles low-latency lossy media over UDP — jitter buffers, NAT traversal, echo cancellation, clock sync. Building that SDK fast and correct is systems programming (buffers, codecs, packet loss), which is why it's a Rust role; the SDK hides that complexity so an agent can run on top with tight latency.

14. References

  • LiveKit Agents — docs & turn detection. https://docs.livekit.io/agents/
  • Silero VAD. https://github.com/snakers4/silero-vad
  • OpenAI Realtime API. https://platform.openai.com/docs/guides/realtime · Google Gemini Live API.
  • WebRTC. https://webrtc.org/ · High Performance Browser Networking (Grigorik), the WebRTC chapter.
  • Conversational turn-taking research (Sacks, Schegloff & Jefferson, 1974) — the linguistics of turn-taking timing.
  • Phase 00 (latency tails) and Phase 12 (streaming) of this track.