« Phase 05 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a capacity request
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line here is slightly different from elsewhere in the track: buy the engine, build the model of it. Nobody should write a scheduler. Everybody should be able to predict, on paper, what their scheduler will do — because that prediction is what a capacity conversation, a procurement request and an incident post-mortem all need.

2. A decision framework for a capacity request

Someone asks for GPUs, or for PTUs, or for a new deployment. Seven questions:

1. What is the constraint? Sovereignty, reproducibility, latency, or cost. If it is sovereignty, most of the rest is moot — go straight to "does it fit?"
2. Does the model fit? Weights, then KV budget at the required context. A 70B fp16 model needs tensor parallelism before anything else is discussable.
3. What is the input:output mix? Every economic answer depends on it, and most people have not measured it. Ask for the number; if it does not exist, that is the first task.
4. What is the p50 and p99 volume? Size the floor to p50 and spill; sizing to peak buys idle capacity and sizing to p10 buys 429s.
5. What batch size does the latency target imply? And does that conflict with anyone already on the deployment? If yes, it is a new deployment, not a bigger one.
6. What is the lead time? Weeks for PTUs, longer for GPUs. The answer determines when this conversation should have started.
7. What happens when this model is deprecated? For a committed term, price the exit.

If the request is for self-hosting, add: who operates it, and is that person in this conversation? The engineering line is the one that decides the business case, and it is usually the one that is missing.

3. Review red flags

In a design document

• A concurrency number with no KV arithmetic behind it.
• A PTU sizing with no stated input:output mix.
• "Tokens per PTU" quoted from a datasheet rather than measured.
• Dedicated capacity sized to peak demand.
• A self-hosting business case with only GPU-hours.
• One deployment serving interactive and batch workloads.
• No model version pin.
• A quantization plan with no evaluation gate.
• Tensor parallelism at the node's maximum rather than the model's minimum.
• Autoscaling that assumes seconds of startup.
• No answer to "what happens when a self-hosted node fails?"
• A commitment term longer than the model generation, with no exit analysis.

In numbers

# Red flag: concurrency from FLOPs
"the GPU does 989 TFLOPs so it can serve hundreds of users"

# Red flag: attention heads, not KV heads
kv = 2 * layers * ATTENTION_HEADS * head_dim * 2       # 8x too big on GQA

# Red flag: no working-space reserve
kv_budget = total_memory - weights                     # you will OOM the batch

# Red flag: admission on the prompt
if kv_bytes(prompt) < free: admit()                    # it grows

# Red flag: a single break-even number
"PTUs pay off above 60% utilization"                   # at what output mix?

# Red flag: unit cost with no utilization
"self-hosting is $0.88 per million tokens"             # at 20B/month, which we don't do

In an incident review

• "We ran out of memory" → was admission budgeting the final length? Is there preemption?
• "Latency spiked for everyone" → was a long prompt stalling decode? Chunked prefill?
• "We hit the provider limit" → is headroom-against-quota on a dashboard, with a lead-time alert?

4. Production war stories

The GPUs that were "too slow." Batch-1 decode, arithmetic intensity ~1 against a ridge point near 300 — 0.3% of the machine. The proposal on the table was to double the fleet. The fix was a serving-stack configuration change, and the person who found it did so by computing bytes-moved per token on a whiteboard.

The 3 a.m. OOM. Admission budgeted prompt length. Ten sequences grew into a budget that fit their prompts and not their outputs. The OOM killed the entire in-flight batch, including requests that were 95% complete — and the retry storm that followed did it again.

The commitment sized to peak. A year of dedicated capacity running at 22% average utilization. The break-even was 57%. Nobody had computed it because nobody had the input:output mix, and nobody had the mix because nobody was measuring at the gateway.

The mix that moved. A team computed a 40% break-even at a 50% output fraction. Six months later their agents were stuffing retrieved documents into prompts, the mix had fallen to 10%, and the real break-even was 77%. The commitment was already signed. Measure the mix continuously; it is an input that drifts.

The self-hosting case that inverted. GPU hours only. Eighteen months later, two engineers were permanently on inference operations — drivers, model updates, evaluation, on-call — and the true unit cost was roughly triple the business case. The sovereignty argument for the deployment was still valid; the cost argument that had been used to sell it was not.

The quantization that quietly regressed. int4 weights, 2× concurrency, celebrated. Extraction accuracy on a specific document type fell noticeably and was found six weeks later by a business user, not by monitoring. There had been no eval gate because "it's the same model."

The 128k context that cost the batch. max_model_len set to the model's full context because "why not." Effective batch size collapsed. The p99 prompt was 4 000 tokens.

5. The interview signal

Signal 1 — you reach for the KV formula immediately. Asked "how many users can this serve", the strong answer starts with 2·L·H_kv·d·b and ends with a subtraction. The weak answer starts with TFLOPs.

Signal 2 — you know why decode is memory-bound, and can say it in one sentence: to produce one token the GPU reads every weight from memory. Then the batching argument follows without prompting.

Signal 3 — you refuse a single break-even number. "It depends on the output mix, and here is how much it moves." That is the sentence that shows you have actually done this rather than read about it.

Signal 4 — you separate constraint from preference. Sovereignty and reproducibility are constraints; unit cost is a preference. Candidates who self-host for price alone have not run one.

Signal 5 — you name the engineering line. Unprompted, in a self-hosting discussion. It is the term that decides the answer and the one most often omitted.

Signal 6 — you tier deployments rather than tune one. Interactive and batch want opposite batch sizes; that is a topology answer, not a configuration answer.

Anti-signals:

• Concurrency estimated from compute.
• Attention heads used where KV heads belong.
• "We'll just add GPUs" as a response to latency.
• A PTU recommendation with no mix and no p50/p99.
• Quantization proposed with no evaluation.
• Treating a commitment as risk-free.

The question to ask them: "What's your measured input-to-output token ratio, and how much capacity is sitting idle?" Both answers tell you whether capacity is managed or assumed, and the question itself signals that you know which two numbers matter.

6. Mentoring notes

Three exercises:

1. Compute the KV cache by hand for the model they serve. Then ask how many concurrent sequences fit. Most engineers have never done it, and the number is usually much smaller than they expect. It permanently reframes "capacity" from compute to memory.
2. Run the batch-1 versus batch-32 measurement on real hardware. Not the simulator — the real thing. Seeing per-token latency fall 25× while GPU utilization barely moves teaches the memory-bound lesson in a way no explanation does.
3. Have them build the break-even spreadsheet and then change the output mix. Watching the answer move from 40% to 77% is the moment they stop quoting single numbers.

And the framing for the platform team: capacity is a forecast, not a reaction. PTUs and GPUs have lead times measured in weeks; by the time utilization tells you that you need more, you are already late. The metric that matters is headroom against the provider limit, on a dashboard, with an alert that fires early enough to start a procurement conversation — and getting that on a dashboard is a much easier thing to ask for than an emergency capacity request in month nine.