Warmup Guide — Model Serving Platform

Zero-to-expert primer for Phase 09: the system around the model — serving patterns, latency distributions, dynamic batching, caching tiers, progressive delivery, and the cost economics of inference fleets.

Table of Contents

Chapter 1: The Four Serving Patterns

Choose the pattern before the framework — most "serving problems" are pattern mismatches:

PatternShapeRight whenWrong when
Online (request/response)sync API, ms budgetsuser-facing decisions (ranking, fraud-at-checkout)results aren't needed now
Batch scoringscore the whole table nightlychurn scores, embeddings refresh, anything pre-computableinputs only exist at request time
Streamingscore events as they flow (Phase 02's consumer)alerting, real-time features, near-real-time personalizationstrict request/response coupling needed
Async / queueaccept → enqueue → callback/pollexpensive inferences (LLM generations, video), spiky loadsub-second UX requirements

The senior move the JD calls "cost efficiency": precompute everything precomputable. A recommender that batch-scores candidates nightly and serves from a key-value lookup beats an online-scoring design by 10–100× in cost — and the hybrid (precomputed candidates + light online re-rank, Phase 06's architecture) is the industry default for exactly this reason.

Chapter 2: Latency Is a Distribution

  • Report p50 / p95 / p99, never means: latency is right-skewed, and the mean is dominated by the tail you're trying to manage. SLOs bind on percentiles ("p99 < 100 ms") because users experience the tail.
  • Tail amplification under fan-out: a request touching N services each with p99 = 100 ms has P(any one is slow) = 1 − 0.99ᴺ — at N = 10, ~9.6% of requests hit a 100 ms+ dependency. Microservice-heavy ML stacks (feature fetch + model + ranker + filters) live or die by this arithmetic; it's why feature-store reads (Phase 03) get single-digit-ms budgets.
  • Measurement honesty (the Phase 07 model-accuracy lessons apply verbatim): warm up before measuring, percentiles need enough samples (p99 of 50 requests is noise), watch for coordinated omission (a load generator that waits for slow responses under-samples the tail — fixed-rate arrival schedules fix this; the lab's simulator uses arrival schedules for exactly this reason).
  • The latency budget exercise: decompose the SLO end to end — network 5 ms + feature fetch 10 ms + inference 25 ms + post-processing 5 ms + headroom — and assign the budget per component. Components without budgets grow until the SLO breaks.

Chapter 3: Dynamic Batching

Why batching wins: every inference call pays fixed overhead (framework dispatch, memory movement, kernel launches — and on GPUs, underutilized parallelism). A model whose cost is c_fixed + n · c_item per call has per-item cost c_fixed/n + c_item — batching amortizes the fixed term. On GPUs the effect is dramatic (the hardware wants 10³+ parallel items — the Phase 07/08 model-accuracy bandwidth story); on CPUs it's still real via vectorization (Phase 01's numpy lesson).

The mechanism (the lab's core): requests arrive asynchronously; a batcher queues them and flushes when either:

  • the batch reaches max_batch_size (the throughput knob), or
  • the oldest queued request has waited max_wait (the latency knob).

The tradeoff, quantified: batching adds up to max_wait of queueing delay to buy throughput. The frontier — p99 vs sustainable QPS as you sweep the knobs — is the deliverable graph of both the lab and any real tuning exercise. Two regimes to recognize: at low traffic, batches rarely fill → latency cost ≈ max_wait, little throughput gain (consider batch size 1); at high traffic, batches fill instantly → throughput gain ≈ full, queueing delay dominated by capacity, and max_wait barely binds.

Setting max_wait: from the latency budget (Ch. 2) — if inference has a 40 ms budget and the model takes 25 ms at your batch size, max_wait ≤ ~10 ms. It's a derived number, not a vibe.

(LLM serving replaces this with continuous batching at iteration granularity — the llm-inference track Phase 09 and model-accuracy Phase 08 cover it; the queueing logic you build here is its foundation.)

Chapter 4: Caching Tiers

Work avoidance beats work optimization. The tiers, outermost first:

  1. Exact-match response cache: hash(model version, canonicalized input) → response. Hit rates are workload-dependent: high for popular-item recsys and repeated searches, ~zero for unique-per-user inputs. Always key on model version — serving stale predictions from the old model after a deploy is the classic cache incident.
  2. Feature cache / online store (Phase 03): the most load-bearing tier — it's why p99 feature reads stay single-digit ms.
  3. Embedding cache: user/item embeddings change slowly; cache with TTL matched to the refresh cadence; pairs with the ANN index (Phase 04).
  4. Intermediate caches (KV cache in LLMs, candidate lists in recsys): inside the model server itself.

The discipline for every tier: an explicit staleness contract (TTL chosen against the upstream change rate), an invalidation story (event-driven beats TTL where possible — Phase 02's streams), and hit-rate monitoring (a silently degraded cache looks exactly like a capacity problem).

Chapter 5: Progressive Delivery — Shadow, Canary, Rollback

Models are statistically validated, so deployment must be too:

  • Shadow (mirroring): the new model receives a copy of live traffic; its outputs are logged, not served. Compares: output distributions, latency, error rate, divergence-vs-current on identical inputs. Zero user risk; catches engineering-level breakage and gross prediction shifts. Cannot measure outcome impact (nobody acted on its predictions) — that needs the canary/A-B.
  • Canary: route a small fraction (1% → 5% → 25% → 100%) of live traffic; monitor guardrails — error rate, latency, and the business guardrail metrics — with statistical care (a 1% canary needs hours-to-days for significance on subtle metrics; Phase 11's power math applies). The lab's CanaryRouter implements the decision: promote when healthy after the soak, auto-rollback on guardrail breach — rollback must be a pointer move (Phase 08's registry), not a build.
  • Blue/green: full parallel fleet, instant cutover/cutback — simpler, costlier, no gradual exposure.
  • The composition that mature teams run: shadow first (engineering safety), then canary (outcome safety), then the A/B test (outcome measurement — Phase 11). Three different questions; three mechanisms.

Chapter 6: Autoscaling and Cost

  • The GPU economics fact: an idle GPU costs the same as a busy one. Fleet cost ≈ replicas × instance price; replicas ≈ peak QPS / per-replica throughput — which is why batching (throughput ↑) and caching (QPS ↓) are cost levers first and latency levers second. Utilization below ~30% on a GPU fleet is a redesign signal (batch more, consolidate models, or move to CPU).
  • Autoscaling signals, in order of usefulness for ML serving: queue depth / concurrency (leading, direct), latency (lagging — you scale after users already hurt), CPU/GPU utilization (misleading for memory-bound or bursty workloads), QPS (only with stable per-request cost).
  • Cold starts: model loading (seconds–minutes for large models) makes scale-from-zero brutal — mitigations: min-replicas ≥ 1, model pre-warming, weights on fast local storage, smaller models (the whole model-accuracy track's reason for existing, from the platform side).
  • Right-sizing heuristic (Extension B): compute per-replica sustainable QPS at the SLO from the measured frontier (Ch. 3), then fleet = peak-QPS / that, +30% headroom, then compare instance types by $/sustained-QPS — a one-afternoon exercise that routinely halves bills.

Chapter 7: The Serving Stack in Practice

Mapping the concepts to the tools you'll meet:

  • Model servers: TorchServe / Triton Inference Server (dynamic batching built in — the knobs are literally max_batch_size/max_queue_delay, i.e., the lab), vLLM for LLMs (continuous batching), ONNX Runtime for portability.
  • The two-service shape: a thin business-logic API (auth, feature fetch, post-processing) in front of a model server — keeps GPU fleets dense and independently scalable from CPU logic.
  • Interfaces: REST for compatibility, gRPC for internal hops (latency + streaming), batched endpoints for bulk.
  • What "reliability" means here: timeouts + fallbacks at every hop (a ranking timeout falls back to popularity order, not an error page — degrade, don't fail), load shedding before collapse, and the observability of Phase 12 wired in from day one.

Lab Walkthrough Guidance

Lab 01 — Batching Server Simulator, suggested order:

  1. percentile + LatencyRecorder — exact interpolation method per the docstring; the hand-computed test pins it.
  2. SimulatedClock + request/arrival plumbing (provided) — read it; the simulator is discrete-event, so tests are deterministic and fast.
  3. DynamicBatcher.run — the flush policy. Get the two invariants right: no batch exceeds max_batch_size; no request's batch-start exceeds its arrival + max_wait. Then latency accounting: completion = batch start + model time; request latency = completion − arrival.
  4. The frontier experiment (sweep) — run the provided workload across knob settings; the tests assert the qualitative shape (batching ≥5× throughput at bounded p99 cost vs batch-1).
  5. CanaryRouter — deterministic hash-based routing, per-arm metrics, the guardrail decision (promote / rollback / continue).

Success Criteria

You are ready for Phase 10 when you can, from memory:

  1. Match the four serving patterns to four product scenarios and name the precompute opportunity in each.
  2. Do the fan-out tail arithmetic and decompose an end-to-end latency budget.
  3. Explain both batching knobs, derive max_wait from a budget, and describe both traffic regimes of the frontier.
  4. Recite the caching tiers with each one's staleness contract.
  5. Distinguish shadow / canary / A/B by the question each answers.
  6. List autoscaling signals in usefulness order and the cold-start mitigations.

Interview Q&A

Q: Your p99 is 3× the SLO but p50 is fine. Where do you look? Tail-specific causes, in order: queueing under bursts (look at queue-depth percentiles — the mean hides the bursts), batching max_wait set too high for the low-traffic regime, GC/checkpoint pauses in the server process, cache-miss path (p50 = hits, p99 = misses — check hit rate), fan-out amplification (one slow dependency), and noisy neighbors on shared infra. The p50/p99 split itself is the diagnostic: the system is healthy on the common path and pathological on a minority path — find which minority.

Q: When would you NOT add dynamic batching? Low-traffic services (batches don't fill — you pay max_wait for nothing), models where per-item cost dominates (no fixed term to amortize — check the measured cost curve first), strict-latency paths whose budget can't fund any queueing, and sequence models with high padding waste under heterogeneous lengths (bucket or use continuous batching instead). The general answer: batching buys throughput with latency — if you don't need the throughput or can't spend the latency, don't.

Q: Design the rollout for a new fraud model where false negatives cost real money. Shadow first — weeks, not days: compare score distributions and disagreement cases against the incumbent on identical traffic; have analysts review the high-disagreement set (fraud labels lag, so offline outcome comparison needs patience). Then a small canary with conservative guardrails — block rate, decline rate, manual-review queue depth — and asymmetric rollback bias (rollback on any doubt; the cost asymmetry demands it). Promote on canary parity plus the lagged label readout. And the org answer: pre-agree the guardrail thresholds and decision owner before the rollout starts — mid-rollout threshold negotiation is how bad models reach 100%.

References