« Phase 05 · Warmup · Track Overview

Principal Deep Dive — Architecture, Tradeoffs & Blast Radius

Table of Contents

• 1. The three tradeoffs
• 2. The capacity portfolio
• 3. Tiering: who shares a deployment
• 4. Scaling envelope
• 5. Failure modes and blast radius
• 6. GPU scheduling in a shared cluster
• 7. Decisions that look wrong but are intentional
• 8. What changes at 10×

1. The three tradeoffs

Tradeoff 1 — throughput vs latency. A bigger batch raises tokens/second per GPU and raises every individual request's inter-token latency. There is no configuration that optimizes both.

The resolution is tiering, not tuning: separate deployments for interactive and batch work, each with its own batch-size target, its own SLO, and its own capacity. A single deployment serving both is always misconfigured for one of them, and the one that complains is always the interactive tier. If they genuinely must share hardware, then at minimum separate the queues with priority and cap the batch the interactive tier can be pulled into — and put batch work first on the degradation ladder.

Tradeoff 2 — commitment vs elasticity. Dedicated capacity buys predictable latency and a lower unit price; it costs a commitment that is idle when you are not using it and a bet on the model still being the one you want.

The resolution is a portfolio (§2), not a choice — and the shape is floor-plus-spill, sized to p50 rather than peak.

Tradeoff 3 — control vs burden. Self-hosting gives sovereignty, reproducibility and the lowest unit cost at high utilization. It costs an inference service you now operate: node pools, drivers, model updates, evaluation, capacity planning, on-call.

The resolution: self-host where a constraint forces it, not where a spreadsheet suggests it. Sovereignty and reproducibility are constraints; unit cost is a preference that stops being true the moment utilization drops. A team that self-hosts for price and then runs at 30% utilization has bought the burden and lost the benefit.

2. The capacity portfolio

The mature shape has four layers, and each exists for a different reason:

The last row is the one that gets cut in a cost review and should not be. A failover path that has never carried live traffic is a hypothesis (Phase 04 §6). Route a small continuous percentage to it; the cost is a rounding error and it converts "we could switch" into a measurement.

How the layers are selected is the gateway's routing policy, not a per-team decision: classification and residency pick the admissible set, task class and cost pick within it, and the fallback chain crosses providers. That is why Phase 04 and Phase 05 are the same conversation from two directions — the gateway makes the choice, this phase supplies the numbers it chooses with.

3. Tiering: who shares a deployment

Four properties determine whether two workloads can share a deployment. If they differ on any of them, they should not:

A useful heuristic: the number of distinct deployments you need is the number of distinct (latency target × classification × model pin) combinations, and if that number is large, the right response is to reduce the combinations rather than to run thirty deployments. Most banks discover they have three latency tiers and two classifications, which is six — manageable — and then a long tail of one-off model pins that nobody has retired.

4. Scaling envelope

Two worth expanding.

TTFT degrades non-linearly under load in a way ITL does not. Prefill is a big, indivisible chunk of compute; a long prompt arriving mid-batch stalls decode for everyone unless the scheduler does chunked prefill. So a workload with a mix of short chats and 30 000-token document analyses produces a TTFT distribution with a very long tail — and the fix is a scheduler feature (chunked prefill) or a tiering decision (separate the document workload), not more GPUs.

Procurement lead time is a scaling dimension. GPU capacity and PTU commitments are measured in weeks, sometimes quarters. That makes capacity planning a forecasting activity with a real horizon, and it is the reason the gateway's headroom-against-provider-limit metric matters more than CPU utilization: it is the one that tells you to start a procurement conversation.

5. Failure modes and blast radius

The first row is the one to design against, because its blast radius is retroactive: an OOM destroys work already done. Every other failure loses the requests that arrive after it. That asymmetry is why the lab's admission control is conservative, and why over-commitment is only acceptable when paired with preemption.

The quantization row deserves a note: quantization is the most attractive lever in this phase (halve the weights, double the concurrency, roughly double the throughput) and the one with the most silent downside. Quality degradation from int8 or int4 is task-dependent and does not announce itself. Treat a precision change exactly like a model change: eval-gated, canaried, reversible.

6. GPU scheduling in a shared cluster

Self-hosting means Kubernetes now schedules GPUs, and GPUs do not behave like CPUs:

• They are not fractional by default. A pod gets whole GPUs unless you use MIG (hardware partitioning into fixed instances) or time-slicing (software sharing with no isolation). For inference, MIG is usually right for small models and whole-GPU is right for large ones.
• They are not fungible. A model pinned to a TP degree needs that many GPUs on one node with fast interconnect. That makes it a gang-scheduling problem, and a cluster that cannot schedule 8 co-located GPUs will leave a node's worth of capacity stranded.
• Startup is slow. Loading 130 GiB of weights is minutes, not seconds. Autoscaling an inference deployment is therefore a pre-warming problem, and reactive scaling on queue depth arrives too late. Scale on a leading indicator, and keep a warm pool.
• They are expensive enough that bin-packing matters. Taints, tolerations and node affinity keep non-GPU workloads off GPU nodes; without them, a stray daemonset can block a whole node.

None of this is exotic, and all of it is new to a platform team whose experience is stateless CPU services. Budget for it explicitly in the self-hosting business case — it is a large part of the engineering line in §6.5 of the WARMUP.

7. Decisions that look wrong but are intentional

Admission reserves the final length. Looks wasteful — real systems over-commit. It is deliberately conservative because the failure it prevents destroys completed work, and because over-commitment without preemption is not a strategy. State the simplification when you use the model.

A 10% flat overhead reserve. Looks arbitrary, and is. Real systems profile at startup. The constant is a placeholder for "measure this", and its presence is more important than its value — a model with no reserve at all is confidently wrong.

Tensor parallelism modelled as one big GPU. Looks like it ignores the hard part. It does, and the docstring says so. The simplification is right for memory (which is what the model is for) and optimistic for latency, which is the safe direction for a capacity question and the unsafe direction for a latency question. Use it for the first.

Static batching modelled analytically rather than simulated. Looks like it might be unfair to static batching. It is exactly fair: the two classes differ in one respect — slot reuse — and everything else is identical, so the measured difference is attributable.

Ties in cheapest_option go to PAYG. Looks like it undervalues commitment discounts. At equal cost, the option with no commitment and no operational burden is strictly better, and encoding that preference explicitly beats relying on enum ordering.

The engineering line defaults to zero. Looks like it invites the mistake it warns about. It forces the number to be supplied, which means someone has to think about it — a default guess would be quoted as if it were ours.

8. What changes at 10×

At one self-hosted deployment and one PTU commitment, the lab's model is enough. At a fleet:

• Capacity planning becomes a forecast with a horizon, driven by procurement lead time rather than by current utilization.
• Chunked prefill and preemption stop being extensions. With a mixed workload you need both, and choosing a serving stack becomes a decision about which of them it implements well.
• Quantization becomes a standing programme, with an eval gate per model per precision, because the concurrency win is too large to leave on the table and too risky to take casually.
• Deployment count needs governance. Every distinct (latency × classification × pin) combination is a deployment with an owner, an SLO and an on-call story. The right response to thirty of them is consolidation, not automation.
• GPU scheduling becomes a specialization: MIG profiles, gang scheduling, warm pools, staged driver upgrades. This is a person, not a ticket.
• Utilization becomes a reported metric with a target, because a self-hosted fleet at 30% utilization is a business case that has quietly inverted.
• Reserved-capacity renewal becomes a calendar event with a right-sizing analysis, or commitments outlive the workloads that justified them.

The seams to build now: measure your own tokens-per-unit rather than quoting a datasheet, record the input:output mix per tenant (it is the input to every capacity decision), pin model versions explicitly, and put utilization on the same dashboard as cost — because the two together are the only honest picture.