« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 16 — Real-Time & Voice Agents (LiveKit-class)

Answers these JD lines: LiveKit's "Staff Rust SDK Engineer" — real-time systems, voice agents, WebRTC, production agents (jd.md). Also directly relevant to the OpenAI/Gemini Realtime APIs and any product moving agents into voice.

Why this phase exists

Every phase so far assumed a text agent where latency is a nice-to-have: if the answer takes three seconds, the user waits. Voice changes the contract. A spoken turn can't be regenerated — once the agent starts talking it's committed to the air — and latency stops being a metric and becomes the product. A voice agent that's 500 ms slow feels broken; one that talks over you is unusable. So the discipline is different: decide when the user finished (endpointing), get first audio out fast (the latency budget), and stop instantly when the user cuts in (barge-in). This phase builds those mechanisms as a deterministic sim — no microphone, no WebRTC, no wall-clock.

Concept map

  • The voice pipeline: mic → VAD → STT → LLM → TTS → speaker (cascaded), vs speech-to-speech / Realtime models.
  • VAD & endpointing: turning voice-activity frames into speech_start / utterance_end — deciding "is the user done?", the hardest UX problem in voice.
  • Turn-taking state machine: LISTENING → THINKING → SPEAKING → LISTENING.
  • Barge-in: the user talks over the agent → cancel TTS mid-stream → back to LISTENING.
  • Latency budget: end-of-speech → first-audio ≈ <800 ms to feel natural; TTFT, streaming, and the tail (Phase 00).
  • WebRTC: the real-time transport (jitter buffers, packet loss) — why not HTTP.

The lab

LabYou buildProves you understand
01 — Voice-Agent Turn-Taking State Machinea silence-based endpointer, the turn-taking state machine with barge-in, and a streaming STT→LLM→TTS pipeline that measures end-of-speech→first-audio latencythat in voice, endpointing + latency + barge-in are the product, and how each mechanism works

Integrated scenario

A customer-support voice agent. The caller asks a question; your endpointer waits just long enough (too eager cuts them off mid-pause, too patient feels slow) to fire utterance_end; the cascade (STT → LLM → TTS) gets first audio out in ~700 ms so the answer feels immediate; and when the caller interrupts with "actually, never mind," the agent stops mid-sentence and listens — because an agent that keeps talking over a customer is a support disaster. Your latency report proves every turn hit the budget, and the state trace shows the barge-in. That responsiveness is the product, and it's what LiveKit builds the SDK for.

Deliverables checklist

  • Lab 01 green under LAB_MODULE=solution pytest and your own lab.py.
  • You can explain endpointing and the too-eager/too-patient tradeoff.
  • You can trace the turn-taking state machine and the barge-in reverse gear.
  • You can compute end-of-speech → first-audio latency and why it's the budget metric.

Key takeaways

  • In voice, latency is the product; you design to end-of-speech → first-audio, at the tail.
  • Endpointing (is the user done?) is the hardest UX problem; the silence timer is the floor, semantic turn-detection is the frontier.
  • Barge-in is non-negotiable — the agent must stop instantly when the user speaks.
  • A voice agent is not chat + TTS bolted on; the real-time constraints reshape the whole design.