Track D — ML and Inference Infrastructure

The reported onsite design round was "design ChatGPT", with the interviewer caring about GPU allocation, autoscaling under non-stationary traffic, and distributed coordination (../../research/source-report.md rows 28–32). The reported advice was to abstract the model-serving layer unless told otherwise.

This is the round most senior generalists lose. It is also the one where your background transfers further than you would guess — see below.

→ Study guide: WARMUP.md — inference from zero: the KV cache, the roofline derivation, the memory budget, every batching technique, autoscaling, and a complete worked “design ChatGPT” answer at both altitudes.

Table of Contents

• Why This Is Closer to Your Background Than It Looks
• The Two Altitudes
• The One Fact Everything Follows From
• Concept Inventory
  • D1. Request Lifecycle
  • D2. GPU Memory and Economics
  • D3. Throughput Techniques
  • D4. Autoscaling Under Non-Stationary Traffic
  • D5. Distributed Coordination
  • D6. Surrounding Systems
• Numbers to Quote Cold
• The Eight Worked Designs
• The Calculator
• Drill Set
• Failure Modes
• Self-Assessment Rubric
• References

Why This Is Closer to Your Background Than It Looks

A decade on multilingual search and recommendation means you have shipped: a serving tier with a hard latency budget, an index that does not fit on one box, ranking under a compute constraint, cache hierarchies where hit rate is the whole economics, and traffic that is non-stationary by time zone.

Serving an LLM is the same problem class with three substitutions:

The gap is vocabulary and the memory-bandwidth constraint, not concepts. That is roughly six weeks of focused work, not six months — which is why this track gets 20% of hours rather than 40%, despite being the round you are weakest in today.

The Two Altitudes

The reported advice — abstract the model-serving layer unless told otherwise — is a scoping test, not a hint about depth. This is inference I2 in ../../research/findings.md.

The interviewer wants to see whether you can identify which component is load-bearing for this conversation and hold the rest at a stable interface. Candidates who dive straight into PagedAttention are demonstrating knowledge while failing the actual signal, which is judgement.

Altitude 1 — abstracted (your default). Name the abstraction explicitly and move on:

"I'll treat the inference engine as a service with three properties: it exposes tokens-per- second capacity rather than requests-per-second, it has an admission interface I can apply backpressure to, and it streams. I'll spend my time on traffic, coordination, and failure — tell me if you want me to open it up."

That sentence does three things at once: it proves you know the engine is special, it hands the interviewer the steering wheel, and it buys you the time for the parts they asked about.

Altitude 2 — opened. When they say "open it up," you have seconds, not minutes, to get into KV cache math, batching policy, and scheduling. Hesitating there undoes the credibility the abstraction bought.

Drill both. Default to altitude 1.

The One Fact Everything Follows From

If you internalize one thing in this track, this is it.

Autoregressive decode is memory-bandwidth-bound, not compute-bound.

The derivation, which you should be able to do on a whiteboard in ninety seconds:

To generate one token, the GPU must read every weight in the model. For a 70B model at FP16 that is 140 GB of reads per token, per sequence in the batch — except that a batch shares the weight read. So:

time per decode step  ≈  bytes_of_weights / memory_bandwidth
70B FP16 on an H100:  140 GB / 3.35 TB/s  ≈  42 ms   (weights alone, one step)

Meanwhile the FLOPs for one token are ~2 × 70e9 = 140 GFLOP. An H100 does ~989.5 TFLOP/s dense dense BF16, so the compute takes ~0.07 ms. Three orders of magnitude apart. The GPU is idle waiting on HBM.

Three consequences, and every technique in this track is one of them:

1. Batching is nearly free on the compute axis — you amortize the same weight read across more sequences. This is why continuous batching is the single biggest throughput lever.
2. Prefill is the opposite. Processing a 2,000-token prompt is a big matrix multiply: compute-bound, high arithmetic intensity. So prefill and decode want different scheduling, which is the entire reason chunked prefill exists.
3. The KV cache, not the weights, is what limits your batch size. It grows linearly with batch size and sequence length, and it is why memory management is the hard part.

The clean empirical proof: the H200 has identical compute to the H100 — same 989.5 TFLOP/s BF16 dense, same 1,979 TFLOP/s FP8 dense — and 43% more memory bandwidth (4.8 vs 3.35 TB/s). It is materially faster at decode. If decode were compute-bound, it would be exactly as fast.

Being able to state this, derive it, and cite the H100/H200 comparison as the evidence is worth more in this round than knowing the name of every serving framework.

Concept Inventory

D1. Request Lifecycle

D2. GPU Memory and Economics

D3. Throughput Techniques

Each with what it buys, what it costs, and when it loses.

The framing that makes this an answer rather than a list: these are not a stack of wins. They are points on a throughput-versus-tail-latency curve, and different traffic classes want different points. An interactive chat turn, a long agentic tool loop, and a batch API are three different curves. Saying that — and then asking whether they run separate pools or one priority-aware scheduler — is the staff-level move.

D4. Autoscaling Under Non-Stationary Traffic

Explicitly named by the interviewer in the source report. This is where your search background transfers and where it misleads you.

Why request-count autoscaling fails for LLMs. In a search tier, requests are roughly interchangeable, so QPS is a good proxy for load. In LLM serving, one request can be a 20-token prompt with a 5-token answer and another can be a 100k-token prompt with a 4,000-token answer. Their costs differ by four orders of magnitude. QPS-based HPA is measuring the wrong thing, and it will scale up on a burst of cheap requests and fail to scale on a handful of expensive ones.

Also required:

• Predictive vs reactive. GPU scale-up is minutes (allocation + weight load), so purely reactive scaling is always late. Forecast from historical diurnal patterns and pre-warm. The SageServe and ENOVA lines of work are exactly this.
• Warm pools. Pay for idle capacity to hide cold start. Size it from the forecast error, not from average load.
• Admission control and load shedding. When you cannot scale further, refuse work rather than accepting it and missing SLO for everyone. Little's law and the utilization knee are the argument.
• SLO classes and fairness. Interactive, batch, and free-tier want different queues. Per-tenant fairness so one heavy user cannot starve the rest. Weighted fair queueing on tokens, not on requests.

D5. Distributed Coordination

D6. Surrounding Systems

Numbers to Quote Cold

Every one verified and attributed. Prices are volatile — always attach a date.

Hardware

The H100→H200 comparison is your evidence sentence: same compute, 43% more bandwidth, materially faster decode. That is the proof decode is bandwidth-bound.

Cost anchors (2026-reported, order of magnitude only)

Derived rules

Run python3 gpu_math.py for all of these against a model you name.

The Eight Worked Designs

Track C has twelve worked designs; this track has eight of its own, in the same shape — nine sections, six hostile critiques, six revisions each. They are where the concept inventory above becomes a design round.

Attempt each before reading it. The index also carries what generalizes: which Track C primitives transfer, which distributed-systems instincts actively fail here, and the defect taxonomy across all 48 critiques.

The Calculator

cd tracks/ml-infra
python3 gpu_math.py --model llama-70b --gpu h100
python3 gpu_math.py --model llama-70b --gpu h100 --seq-len 8192 --batch 64
python3 gpu_math.py --list

It prints the memory budget, the maximum batch that fits, the decode-step floor, the roofline verdict, and the cost per million tokens. The output is the script for what you say out loud in the round — derive, do not recite.

Drill Set

Failure Modes

Self-Assessment Rubric

Hire-bar translation

References

Serving systems and papers

• Kwon et al. Efficient Memory Management for LLM Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
• Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022 — origin of continuous batching
• Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. https://arxiv.org/abs/2403.02310 — chunked prefill
• Leviathan et al. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
• Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models. https://arxiv.org/abs/2305.13245
• Pope et al. Efficiently Scaling Transformer Inference. MLSys 2023. https://arxiv.org/abs/2211.05102
• Zhong et al. DistServe: Disaggregating Prefill and Decoding. OSDI 2024. https://arxiv.org/abs/2401.09670
• SageServe: Forecast Aware Auto-Scaling for LLM Serving. https://arxiv.org/pdf/2502.14617
• ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving. https://arxiv.org/abs/2407.09486

Implementations to read

• vLLM — https://github.com/vllm-project/vllm · docs https://docs.vllm.ai/
• vLLM Blog. Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025-09-05). https://vllm.ai/blog/2025-09-05-anatomy-of-vllm
• NVIDIA TensorRT-LLM — https://github.com/NVIDIA/TensorRT-LLM
• NVIDIA Triton Inference Server — https://github.com/triton-inference-server/server
• Ray Serve — https://docs.ray.io/en/latest/serve/index.html
• SGLang — https://github.com/sgl-project/sglang (RadixAttention / prefix caching)

Operational grounding

• OpenAI. Scaling Kubernetes to 7,500 nodes. https://openai.com/index/scaling-kubernetes-to-7500-nodes/
• NVIDIA. H100 Tensor Core GPU. https://www.nvidia.com/en-us/data-center/h100/
• Hello Interview. Design ChatGPT. https://www.hellointerview.com/learn/system-design/problem-breakdowns/chatgpt
• Related tracks in this repo: llm-inference-engineer · Senior AI Engineer · pretraining-lead