Lab 01 — Voice-Agent Turn-Taking State Machine

Phase 16 · Lab 01 · Phase README · Warmup

The problem

A voice agent lives or dies on timing. Build the three mechanisms that make one feel like a conversation — as a deterministic, offline sim (no microphone, no WebRTC, no wall-clock):

  1. Endpointing — turn voice-activity frames into speech_start / utterance_end (is the user done?).
  2. Turn-taking state machine — LISTENING → THINKING → SPEAKING → LISTENING.
  3. Barge-in — the user talks over the agent → cancel TTS mid-stream → back to LISTENING.

…and a streaming STT→LLM→TTS pipeline that measures the metric voice engineers live by: end-of-speech → first-audio latency.

What you build

PieceWhat it does
Endpointer.feedsilence-based endpointing: speech_start + utterance_end after silence_ms
TurnManager.dispatchthe forward turn table + the barge-in reverse gear (SPEAKING → LISTENING)
Playback (given)a cancellable TTS token stream; spoken < total proves an interruption
within_budgetworst-case first-audio latency check
run_pipelinedrive frames through endpointer → STT → LLM → TTS with barge-in; measure latency

The scaffolding (Clock, Frame, Playback, Fake STT/LLM/TTS, latency dataclasses) is provided; you implement the algorithms.

File map

FileRole
lab.pyyour implementation (fill the # TODOs)
solution.pyreference + a scripted two-turn support conversation with a barge-in (main())
test_lab.py30 tests: endpointing, state transitions, barge-in cancel, latency, budget, determinism
requirements.txtpytest only

Run it

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

  • Your endpointer fires speech_start on the first speech frame and utterance_end exactly when trailing silence reaches silence_ms — not early.
  • Your turn machine follows LISTENING → THINKING → SPEAKING → LISTENING and raises on invalid events; barge-in reverses SPEAKING → LISTENING and cancels the playback.
  • You measure end-of-speech → first-audio and check the worst turn against the budget.
  • All 30 tests pass under both lab and solution.

How this maps to the real stack

  • The endpointer is the silence-timer floor under LiveKit's turn-detector and the Realtime APIs' VAD; real VAD is Silero, and the frontier is a semantic turn-detection model.
  • The turn state machine + barge-in are what LiveKit Agents implements around your STT/LLM/TTS (or a speech-to-speech model); the cancellable Playback is real TTS interruption.
  • The latency accounting (STT + LLM TTFT + TTS TTFB, measured from end-of-speech) is the exact budget a voice team tracks; WebRTC is the transport under all of it (not modeled here).

Limits. Real VAD is DSP on audio; real STT/LLM/TTS stream continuously with partials; real transport is WebRTC with jitter/loss. The lab models the control and timing, which is the part interviews probe.

Extensions (your own machine)

  • Add semantic endpointing: a stand-in "is this a complete thought?" check that can override the silence timer.
  • Model overlapping speech / backchannels ("mm-hmm") that shouldn't trigger a barge-in.
  • Wire real components behind the injected STT/LLM/TTS (e.g. LiveKit Agents) and compare the latency report to your sim.

Interview / resume signal

"Built a voice-agent turn-taking engine — silence-based endpointing, a LISTENING/THINKING/SPEAKING state machine with barge-in (cancel TTS mid-stream), and a streaming STT→LLM→TTS pipeline that measures end-of-speech→first-audio latency against a budget — the real-time mechanisms LiveKit and the Realtime APIs hide."