« Phase 16 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 16 — Principal Deep Dive: Real-Time & Voice Agents

At the mechanism level a voice agent is an endpointer, a state machine, and a latency budget. At the platform level it is a distributed, media-plane system whose entire economics and reliability story are shaped by one fact: you are holding a live bidirectional media session open for the full duration of every call. This changes the architecture, the capacity math, the failure modes, and the cost model away from anything a request/response text-agent platform looks like. This is the view a principal is expected to hold.

The architecture split: media plane vs agent plane

A production voice platform is two planes with very different characteristics. The media plane carries RTP audio over WebRTC/SRTP: it is stateful, sticky, latency-critical, and typically written in a systems language (this is exactly why LiveKit hires Rust SDK engineers, not prompt engineers). The agent plane runs the endpointer, the STT/LLM/TTS orchestration, and tool calls. The interface between them is a stream of audio frames one way and a cancellable stream of audio chunks the other. The single most important architectural property is that cancellation must propagate from the media plane back through the agent plane in tens of milliseconds — barge-in is a distributed cancellation problem, and if your TTS vendor call cannot be aborted mid-stream, no amount of clever endpointing saves you. Design the whole system around the abort path, not the happy path.

Capacity and the latency budget as a hard SLO

The budget — end-of-speech to first-audio under ~800 ms to feel natural, under ~500 ms to feel snappy, over ~1 s and the user assumes you did not hear them — is not a nice-to-have metric; it is a tail SLO that you provision against. The math is unforgiving because the budget is a sum over a cascade: if STT endpoint-confirm takes 150 ms, LLM time-to-first-token is 400 ms at p95, and TTS time-to-first-byte is 200 ms, you are already at 750 ms with zero network jitter, and you have spent your whole budget before the packet leaves the datacenter. The principal moves are structural: overlap stages (start TTS on the first LLM sentence, not the last), keep a warm model connection so TTFT is not paying a cold start, colocate STT/LLM/TTS in one region to avoid inter-service RTT, and treat the worst concurrent-call turn as the number you provision for, because GPU contention makes p99 TTFT balloon exactly when you are busiest.

Where the bodies are buried

Three decisions look wrong until you have operated voice at scale. First, endpointing is deliberately conservative-then-corrected: you fire a fast endpoint to start generating, but keep listening, and if more speech arrives you discard the in-flight generation — you spend tokens you throw away to buy latency. That waste is intentional and correct. Second, you stream partial audio you may have to retract: barge-in means some fraction of every generated response is spoken into the void, which looks like burning money and compute until you accept it as the cost of natural turn-taking. Third, the mean latency is a vanity metric; you SLO the p95/p99 per-turn, because conversational quality is destroyed by the single bad turn, not the average. A platform tuned to a good mean and a bad tail feels broken while looking healthy on a dashboard.

Failure modes and blast radius

The failure taxonomy is media-plane-shaped. A stuck TTS stream that will not cancel produces the worst user-visible failure — the agent talking over the customer — so the mitigation is a hard client-side kill of the audio track independent of whether the vendor acknowledges the abort. STT endpoint flapping (the confirmed end-of-speech oscillating) produces clipped or doubled utterances; you damp it with hysteresis, not a single threshold. Regional GPU exhaustion does not throttle gracefully like a text API — it stretches TTFT past the budget on every live call at once, so the blast radius is "every caller in that region simultaneously hears dead air," which argues for per-region capacity headroom and fast failover rather than global pooling. And because sessions are long-lived and sticky, a rolling deploy that kills a media node drops live calls; you need connection draining and session migration, which is far harder than draining stateless HTTP.

Cross-cutting concerns

Cost is priced per minute of open session across three models plus a media SFU, not per request — a 6-minute call is 6 minutes of STT streaming, dozens of LLM turns, TTS for every agent utterance, and a held media port; your unit economics are dollars-per-call-minute and your biggest lever is the barge-in waste and the model tier. Observability must be turn-scoped: you trace each turn with its end-of-speech timestamp, first-audio timestamp, per-stage latencies, and a barge-in flag, because "the call felt laggy" is only debuggable if every turn's budget breakdown is recorded (the same OpenTelemetry-GenAI discipline as Phase 14, extended with media timestamps). Security and privacy are heightened: live audio is PII, often under recording-consent law, so retention, redaction, and per-tenant isolation of transcripts are first-class, not afterthoughts. Multi-tenancy on the media plane means noisy-neighbor is a latency problem — one tenant's traffic spike steals GPU headroom and blows another tenant's turn budget — so isolation here is about capacity reservation, not just data scoping.

Cascaded vs speech-to-speech as an architecture decision

The choice between a cascaded pipeline (separate STT/LLM/TTS) and a speech-to-speech Realtime model is a genuine architectural fork with platform consequences. Cascaded gives you an inspectable transcript (essential for audit, eval, and injecting tool calls or RAG mid-turn), model choice, and independent scaling of each stage — at the cost of accumulated latency and lost prosody. Speech-to-speech collapses the cascade into one model: lower latency, preserved tone and overlap, but an opaque pipeline that is hard to gate with your own guardrails, hard to inject tools into, and less mature operationally. The principal answer is not a preference; it is: cascaded for enterprise voice where control, tool use, auditability, and eval dominate; speech-to-speech where latency and naturalness are the product and you can accept a more closed loop. Most regulated-industry voice agents are cascaded today for exactly the auditability reason, and a platform that wants both must abstract the pipeline behind a common turn interface so it can swap the implementation per use case without rewriting the media plane.