Lab 01 — Batching Server Simulator

Phase: 09 — Model Serving Platform | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Triton's dynamic batcher has two knobs: max_batch_size and max_queue_delay. This lab builds the mechanism behind them as a discrete-event simulator — so the latency-throughput tradeoff stops being folklore and becomes a table you computed.

What you build

percentile / LatencyRecorder — honest tail measurement (linear-interpolated order statistics; the p50/p99 vocabulary every SLO is written in)
DynamicBatcher — the canonical flush policy: seal on size-full OR oldest-waits- max_wait; FIFO single-server simulation with full latency accounting
sweep — the latency-throughput frontier across knob settings
CanaryRouter — sticky hash-based traffic splitting + guardrail decision logic (continue / promote / rollback)

The two regimes (the tests make you produce both)

Regime	What happens	Measured in tests
Saturation (arrivals ≥ capacity)	batches fill instantly; throughput multiplies as `c_fixed` amortizes	batch-32 > 5× batch-1 throughput
Low traffic (sparse arrivals)	batches never fill; every request pays `max_wait` for nothing	mean batch = 1.0; p50 = batch-1 p50 + `max_wait`, exactly

Reference frontier (2 000 requests at ~1 req/ms, model cost 8 + n ms):

        config  thr (req/ms)     p50     p99  avg batch
      (1, 0.0)         0.111  8020.2 15811.3        1.0
    (32, 20.0)         0.715   425.1   769.1       20.6

(Saturated: the queue itself dominates latency — batching here reduces p99 6.4× while multiplying throughput, because the alternative is an exploding queue.)

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (16 tests)
python solution.py                          # the frontier table

Suggested TODO order

percentile — pin against the hand-computed test
DynamicBatcher.run — get the invariants first (size cap, wait cap), then the hand-computed latency test, then conservation
sweep — both regime tests
CanaryRouter — routing stickiness, then the three-way decision

Success criteria

All 16 tests pass
You can read the frontier table aloud: which regime each row is in and why
You can derive max_wait from a latency budget
You can explain why canary routing must be sticky and deterministic (same user on both arms contaminates the comparison; non-reproducible routing is undebuggable)

Extensions

Multiple replicas (k servers) + the queue-depth autoscaling signal
Priority lanes: interactive vs batch traffic sharing one server
Coordinated-omission demonstration: closed-loop vs open-loop load generation on the same system, compare measured p99
Shadow mode: mirror every request to a second model; divergence report

Q: Batching raised your p99 at low traffic. Why, and what do you do? Underfull batches wait the full max_wait — pure latency cost with no throughput benefit (the low-traffic regime test). Fixes: adaptive max_wait (shrink when queue empty), flush-on-idle policies, or traffic-aware config (batch only above a QPS threshold). The general lesson: batching knobs are workload-dependent; tune against the production arrival process, not a synthetic saturated one.

Q: Your canary shows 0.3% higher error rate. Promote or rollback? First: is it significant? 0.3% on a few hundred samples is noise — compute the two-proportion test (Phase 11) or keep soaking (continue). Second: is it the model or the infrastructure (cold cache, new container image)? Check error types. Third: pre-agreed thresholds decide — if 0.3% exceeds the agreed max_error_delta with significance, rollback without negotiation; renegotiating thresholds mid-canary is how bad models ship.

References

Triton dynamic batching — the productionized version of this lab
Dean & Barroso, The Tail at Scale (CACM 2013)
Tene, How NOT to Measure Latency — coordinated omission
Phase 11 — Online Experimentation — the statistics the canary decision borrows

Senior ML Engineer