« Phase 16 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor
Phase 16 — Staff Engineer Notes: Real-Time & Voice Agents
The gap between someone who uses a voice framework and someone trusted to own a voice product is not knowledge of the API. It is knowing that in voice, latency is the product, that barge-in is table stakes, and that the interesting engineering is in the milliseconds and the turn-taking, not the model. This is the judgment view: what a staff engineer owns here, how they decide, what they flag in review, and the exact signal an interviewer is listening for.
What a staff engineer actually owns
You own the latency budget as a contract, not a hope — end-of-speech to first-audio, SLO'd at the tail, with per-turn tracing so a regression is caught in a dashboard and not a customer complaint. You own the cancellation path end to end, because a system that cannot reliably stop talking is worse than one that is slightly slow. You own the build-vs-buy call: whether to run a cascaded pipeline you control or a speech-to-speech Realtime API, and you own being honest about which kind of "voice engineer" your team is — an applied-AI team that orchestrates vendors, or a systems team that can own the WebRTC media plane. Knowing which one you are is itself a staff-level signal; over-claiming real-time media systems depth you do not have is a fast way to lose credibility.
The decision framework: when to reach for what
- Cascaded (STT→LLM→TTS) when you need tool use mid-turn, RAG grounding, an auditable transcript, model choice, or independent scaling — i.e. almost all enterprise and regulated voice. The price is accumulated latency; you pay it with stage overlap and warm connections.
- Speech-to-speech / Realtime when latency and naturalness are the whole product and you can accept an opaque, harder-to-gate pipeline — consumer companions, low-friction assistants.
- Fixed silence endpointing as the floor everywhere; add a semantic turn detector only once you have measured that clipping or sluggishness is hurting real conversations. Do not reach for the model first; the timer is 80% of the value.
- Do not build WebRTC yourself. Use LiveKit or an equivalent SFU unless media transport is your product. Reinventing jitter buffers and echo cancellation is a multi-quarter systems project most teams should buy.
Code-review red flags
- A voice turn implemented as mutating boolean flags with no explicit state machine — a talk-over bug waiting to ship. Demand a guarded transition function.
- A TTS or LLM call with no cancellation token, or a barge-in path that stops generation but not the outbound audio track — the agent will talk over the user.
- Endpointing on a single fixed threshold with no hysteresis — it will flap and clip.
- Latency measured as total generation time or as a mean — the felt metric is first-audio at the worst turn; a p50 dashboard hides the calls that feel broken.
- Wall-clock time in the decision logic, making it untestable — the clock should be injectable so turn-taking can be unit-tested deterministically.
- Echo/self-transcription not handled — the agent barges in on its own voice.
Production war stories
The agent that talked over the customer. No barge-in; it recited a 20-second policy while the caller shouted "stop." The fix was not a better model — it was a cancellation path that kills the audio track the instant speech_start fires. The clipper. A 200 ms endpoint cut people off mid-pause; orders came in half-complete because "a large… pizza" became two utterances. The fix was a longer, hysteretic timer plus a semantic turn detector for the ambiguous cases. The p50 that lied. A dashboard showed a healthy 600 ms mean while every call had one 1.4 s turn that made the caller start over; moving the SLO to p95-per-turn made the real problem visible and fixable.
The interview signal
When an interviewer asks about voice-agent performance, the junior answer talks about model quality. The staff answer reframes to time: "latency is the product; I design to end-of-speech-to-first-audio, stream the first chunk out immediately, and SLO the worst turn, not the mean." Then, unprompted, you raise barge-in — "the agent must cancel its own TTS the instant the user speaks, discard the half-spoken response, and listen" — which signals you have used a voice agent that lacked it and hated it. Then you show you know the boundary of your own expertise — that the real-time media plane is systems work you either own or buy. That trio (latency-as-product, barge-in-as-table-stakes, honest scope) is the exact signal that separates someone who read the Realtime docs once from someone who has shipped voice.
Closing takeaways
- Latency is the product. Design to end-of-speech → first-audio, stream immediately, SLO the tail. Everything else is plumbing.
- Barge-in is non-negotiable and it is a cancellation problem that must propagate all the way to the audio track. A voice agent that cannot stop is unusable.
- Endpointing has no universally-right threshold. The silence timer is the floor; semantic turn detection is the frontier you add when you have measured the need.
- Model whose-turn-it-is explicitly. A guarded state machine turns talk-over bugs from silent corruption into caught errors.
- Cascaded vs speech-to-speech is a real architecture decision driven by control/auditability vs latency/naturalness — have the tradeoff ready, not a favorite.
- Know which voice engineer you are. Owning the media plane and orchestrating vendors are different jobs; claiming the wrong one is a credibility risk, and knowing the difference is seniority.