Knowledge 08 — Capacity Sizing & Solutions Architecture

The JD's other half: half of JD2 is engineering; the other half is being the "trusted technical advisor" who runs customer engagements — discovery, scoping, sizing, trade-off analysis, POCs, escalations, and documentation. This module turns the technical depth of Knowledge 00–07 into the business artifacts and conversations that actually win and keep customers. It's the layer that distinguishes a strong engineer from a principal who can found a company: the ability to connect silicon to a CFO's spreadsheet and a customer's success criteria.

Table of Contents

1. The bridge: model ↔ hardware ↔ customer

The JD's thesis sentence: "bridge AI model requirements ↔ hardware capabilities ↔ customer expectations." Every engagement is a triangulation between three things that are each underspecified when you arrive:

  • Model requirements: accuracy floor, latency targets, context length, modality, growth.
  • Hardware capabilities: the roofline, memory, interconnect, decode, power, and price of the accelerators on the table.
  • Customer expectations: budget, SLA, timeline, risk tolerance, operational maturity — usually expressed as a vibe ("we need it fast and cheap") that you must convert into numbers.

The principal's core move is turning vague expectations into a falsifiable specification, then finding the point where all three corners are simultaneously satisfiable — or honestly telling the customer they aren't and what must give. That conversion is the job. Everything below is technique for it.

The junior accepts "make it fast and cheap" and starts tuning. The principal replies: "Let's define fast (p95 TTFT and TPOT at what concurrency), cheap ($/1M tokens target or monthly budget), and good enough (accuracy floor on which eval). Then I'll show you the three configurations on that trade-off curve and what each costs." That reply is why they pay for you.

2. The sizing math: from SLO to silicon

The deterministic core of sizing. Given a workload and an SLO, compute the hardware. (This synthesizes Knowledge 00–05.)

Inputs you must extract from the customer:

  • Model (→ params P, architecture, GQA/MQA, config).
  • Input/output length distribution (→ prefill vs decode mix).
  • Target throughput (requests/sec or tokens/sec) and concurrency.
  • Latency SLO (p95 TTFT, p95 TPOT).
  • Accuracy floor (on a named eval).
  • Budget / power / footprint constraints.

The computation:

  1. Memory per instance = weights (P × bytes/param after quantization) + KV cache(batch, context) + activations + overhead. → determines parallelism (TP/PP) and #GPUs per instance.
  2. Per-instance throughput = from benchmarking the chosen config at the SLO — the goodput at the batch size where p95 latency just meets the target.
  3. #instances = ⌈ required throughput / per-instance goodput ⌉, then × a headroom factor (1.3–2×) for traffic peaks (§4).
  4. Total accelerators = #instances × GPUs-per-instance, rounded up to whole nodes.
  5. Sanity-check against the roofline: is the per-instance number physically plausible (near MBU/MFU limits), or did you assume a throughput the hardware can't reach?

The discipline: sizing is a calculation with stated assumptions, not a guess. Write the assumptions down (model, lengths, SLO, quantization, headroom) — because when the customer's real traffic differs, you can point to which input changed rather than looking wrong. A sizing memo is only as good as its explicit assumptions.

3. The $/token and TCO model

$/1M tokens (or $/inference, $/camera-hour) is the unit of competition and the number the CFO actually decides on.

$ per 1M tokens = (accelerator $/hour × #accelerators) / (tokens/hour produced) × 1e6
tokens/hour     = goodput_tokens_per_sec × 3600

A worked feel: if a node is $30/hr and produces 5M output tokens/hour at SLO, that's $30 / 5M × 1e6 = $6 per 1M tokens. Halving it (via quantization, better batching, spec decode, or a more efficient chip) is the entire commercial game.

TCO (Total Cost of Ownership) — the honest full picture, beyond the GPU sticker price:

  • Capex: accelerators, servers, networking (NVLink/IB switches), racks.
  • Opex: power (the big one at scale — perf/watt is a dollar metric, which is Qualcomm's whole pitch), cooling, data-center space, networking, software licenses.
  • People: ops, on-call, the cost of complexity (a fragile distributed setup has a human cost).
  • Utilization: an accelerator at 30% utilization costs the same as at 90% — effective $/token depends on keeping it busy. Idle hardware is the silent TCO killer.
  • Opportunity/lock-in: ecosystem switching cost (the NVIDIA-CUDA moat is real TCO even when the silicon is pricier).

Why perf/watt is a sales weapon (and the Qualcomm-archetype pitch): at data-center scale, power and cooling can rival hardware cost, and many sites are power-capped (can't add more draw regardless of budget). "Same throughput at half the watts" means more inference in the same power envelope — sometimes the only way a customer can grow. Framing efficiency as tokens-per-dollar and tokens-per-watt, not raw speed, is the principal's commercial fluency.

4. Capacity planning under real traffic

Sizing for average load is a rookie error — traffic is bursty and diurnal. Plan for the distribution:

  • Peak-to-average ratio: real traffic peaks 2–5× its average (lunchtime, business hours, launch spikes). Size goodput for the peak, or autoscale.
  • Percentile SLOs under burst: the SLO must hold at the peak, not the mean — that's where p99 violations happen (queueing explodes past the load knee).
  • Autoscaling: scale instances with load — but LLM instances are slow to spin up (load 100+ GB of weights, warm caches), so reactive autoscaling lags. Use predictive/scheduled scaling + a warm buffer. This is a real operational gotcha to raise with customers.
  • Headroom: run at ~60–70% of capacity steady-state so bursts have room. A system sized to 100% of average falls over on the first spike.
  • Graceful degradation: under overload, preempt/queue/shed rather than crash; consider a smaller fallback model for overflow.

The capacity conversation: "What's your peak-to-average, and is your SLO a peak-time promise? Because sizing to the average and hoping is how you get paged at noon." Surfacing the burst reality early is trusted-advisor behavior.

5. The trade-off frameworks

The JD repeatedly asks for "trade-off analysis." The principal's job is to present Pareto frontiers and let the customer choose with eyes open, not to declare one answer. The recurring axes:

Trade-offThe tensionThe tool
Throughput ↔ latencybigger batch = more throughput, worse per-request latencytune to the SLO; report the goodput curve
Accuracy ↔ cost/speedquantization/smaller model = cheaper, lower qualitysensitivity analysis; eval on the customer's hardest task
Latency ↔ costmore/faster chips = lower latency, more $$/token at each latency point
Capability ↔ feasibilitybigger model = more capable, harder to deploy"do you need 70B, or does a well-tuned 8B + RAG hit your accuracy floor at ¼ the cost?"
Flexibility ↔ performanceeager/general vs compiled/specializedtorch.compile vs TensorRT-LLM
Build ↔ buy ↔ APIself-host vs managed vs vendor APITCO at the customer's volume (self-host wins above a break-even QPS)

The model-rightsizing instinct (high-value advice): customers reflexively ask for the biggest model. The principal asks "what's the smallest model that clears your accuracy floor on your eval?" — because a tuned 8B at $X often beats a 70B at $8X for a task where both pass. Recommending down when justified builds enormous trust and is exactly the "recommend alternative model approaches" JD bullet.

Present trade-offs as "here are three points on the curve — fast/expensive, balanced, cheap/slower — with the numbers; which corner matches your priorities?" Customers respect being given the frontier and the agency to choose far more than being handed one number to take or leave.

6. Running a customer engagement (discovery → production)

The JD's lifecycle: "discovery, scoping, success criteria, and execution plans." The phases:

  1. Discovery / qualification: understand the actual problem, not the stated solution. What's the use case, the real workload, the constraints, the success metric, the decision-maker, the timeline? Distinguish "we want AI" from "we have a measurable problem worth solving." Ask for real traffic samples and the hardest task — they're technical gold (calibration, benchmarking).
  2. Scoping: convert the problem into a spec — model candidates, SLO numbers, accuracy floor, hardware options, budget. Define what's in and out of scope explicitly (scope creep kills POCs).
  3. Success criteria: agree up front on the measurable exit criteria — "POC succeeds if it serves X QPS at p95 TTFT < Y ms, accuracy ≥ Z on this eval, on this hardware." Written down. Signed off. This is the single most important risk-control move (§7).
  4. POC / demo: build the smallest thing that tests the riskiest assumption (§7).
  5. Implementation: production-grade deployment — the full Knowledge 00–07 stack.
  6. Post-implementation support + knowledge transfer: runbooks, training, handover (§9). The JD explicitly values this; it's where one-off projects become long-term relationships.

The discovery superpower: most customer "requirements" are solutions in disguise ("we need a 70B model on 8 H100s"). Dig to the underlying success metric ("we need to answer support tickets at 95% accuracy under 2s for $X/month") and you often find a cheaper, better architecture they hadn't considered. Reframing the problem is more valuable than answering the stated question.

7. The POC: scoping and exit criteria

The JD wants "technical demos, proofs-of-concept, and reference applications." A POC is a risk-reduction experiment, not a mini-product. Principles:

  • Test the riskiest assumption first. Usually: "does the quantized/ported model hold accuracy on their data at the target latency on this hardware?" Build the smallest thing that answers that yes/no.
  • Define exit criteria before building — the §6 measurable spec. A POC without pre-agreed success criteria becomes an infinite demo loop where the goalposts move; this is the #1 way engagements rot.
  • Use the customer's real data and real workload — a POC on synthetic data proves nothing about production.
  • Timebox it. Two weeks with a clear pass/fail beats two months of polishing.
  • Make the result honest and reproducible (benchmarking standards) — the customer's engineers will re-run it.

The exit-criteria sentence you write at the start: "This POC is a success if, on your 500 real tickets, the INT8-quantized model on a single Cloud AI 100 achieves ≥ the FP16 accuracy minus 1 point, at p95 latency < 800 ms and ≥ 50 req/s." Everyone signs. Now the POC can end, with a clear verdict — which is the only way it adds value.

8. Issue triage and technical escalation

The JD: "driving issue triage and technical escalations … coordinating across product, hardware, and software engineering teams." When a customer deployment is broken, you're the incident lead. The method:

  1. Triage / classify fast: is it model (accuracy), runtime (crash/perf), hardware (Xid/thermal/driver), data, or integration? The four bottleneck classes and the three port-failure modes are your taxonomy.
  2. Stabilize: mitigate first (rollback, scale out, reduce batch) to stop the bleeding, then root-cause. Don't debug a production outage live.
  3. Reproduce minimally (the debugging methodology).
  4. Escalate with evidence: a clean repro, versions, and a precise hypothesis — to the right team (vendor GPU driver team vs the model team vs the customer's integration). A good escalation includes the answer-shaped question, not "it's broken."
  5. Communicate status to the customer continuously — even "still investigating, here's what we've ruled out" maintains trust during an outage. Silence destroys it.
  6. Post-mortem: what broke, why, the fix, and the prevention — blameless, written, shared. Feeds the docs.

The calm-under-fire value: in a war room, the principal is the one who says "let's classify before we thrash — is the GPU idle or busy? Did this start after a deploy or a driver update? Let me get a minimal repro." Structured triage under pressure is the senior signal, and it's a learnable procedure, not a personality trait.

9. Customer-ready documentation

The JD: "getting started guides, deployment runbooks, troubleshooting guides … partner training." This is a deliverable, not an afterthought. The set:

  • Getting-started guide: how a customer engineer stands the system up from zero.
  • Deployment runbook: exact steps, configs, versions, hardware reqs, the bring-up checklist, rollback procedure.
  • Tuning guide: the knobs and the goodput curve for their workload, with the recommended operating point and why.
  • Troubleshooting guide: the common failures (driver mismatch, OOM, latency spikes, accuracy drift) and their fixes — your triage taxonomy, written down.
  • Benchmark/validation report: honest numbers with conditions (standards) — the artifact that justified every decision.
  • Architecture decision record: what you chose, the alternatives, and why — so the next engineer (and future you) understands the trade-offs baked in.

Documentation as trust infrastructure: in air-gapped/regulated/sovereign contexts (and the original folder's Digital Twin role), the customer operates the system after you leave — the runbook is the handover. Good docs turn a project into a renewable relationship and a reference you can sell from. Treat them as a first-class deliverable; the JD literally does.

10. Communicating to non-technical stakeholders

The JD: "present complex technical concepts clearly to both engineering and non-engineering audiences," "simplify complex ideas into useful advice." The skill:

  • Lead with the decision, not the mechanism. A CFO wants "this cuts your inference bill 40% with no quality loss," not a PagedAttention lecture. Give the so-what first; offer depth on request.
  • Translate metrics into business language: TPOT → "how fast it types for the user"; $/token → "monthly bill at your volume"; perf/watt → "fits in your existing power budget"; accuracy delta → "1 in 100 answers changes — here's whether that matters for your task."
  • Use the right analogy: KV cache = "the model's short-term memory of the conversation, which costs RAM"; quantization = "storing numbers in less detail to save space and time, like a smaller photo file"; batching = "serving many customers in one trip instead of one at a time."
  • Make trade-offs concrete and visual: a three-option table (fast/balanced/cheap with the numbers) beats a paragraph. Give the frontier and a recommendation.
  • Be honest about uncertainty and limitations — overpromising is how trusted advisors stop being trusted. "Here's what I'm confident in, here's what we need to test, here's the risk" ages far better than confident hand-waving.

The hardest and most valuable communication skill is knowing which layer of detail your audience needs and stopping there — full depth for the customer's engineers, the decision and its dollars for their executives, and the ability to move between the two in one meeting. That range is what "principal" means on the customer-facing axis.

11. Worked example: a full sizing memo

A compact version of the deliverable that integrates the whole track. (Numbers illustrative.)

Sizing Memo — Customer X, Internal Support Assistant

Requirement (from discovery): answer support questions, ~800-token prompts, ~300-token answers, 40 req/s sustained / 100 req/s peak, p95 TTFT < 1 s, p95 TPOT < 50 ms, answer quality ≥ baseline on Customer X's 500-question eval, budget pressure on monthly cost, existing power-capped racks.

Model choice: Customer asked for 70B. On their eval, a well-tuned Llama-3-8B + RAG (original folder Lab 2) reaches the accuracy floor at a fraction of the cost. Recommend 8B-class; keep 70B as a fallback for the ~5% hardest queries (router pattern). (Model-rightsizing, §5.)

Numerics: INT8 weight-only (AWQ) — measured −0.4 pts on their eval (within floor), ~2× decode throughput, half the memory. INT8 KV for concurrency.

Serving: vLLM, continuous batching + prefix caching (big shared system prompt → big TTFT win). Tuned to p95 TPOT < 50 ms ⇒ batch capped at the SLO knee from the goodput sweep.

Hardware/parallelism: 8B INT8 fits one accelerator → no model sharding; scale by replication (DP). Per-instance goodput ≈ G req/s at SLO ⇒ ⌈100 peak / G⌉ instances + 30% headroom.

Cost: $Y per 1M tokens; monthly ≈ tokens/month × $Y. Perf/watt fits the existing power cap because we chose 8B + INT8 — the 70B path would have exceeded the rack power budget, a hard blocker, not just a cost one. (TCO/power, §3.)

Risks & validation: accuracy holds on their hardest tickets (validated, §7); peak autoscaling lag mitigated with a warm buffer (§4); benchmark report attached with full conditions (§9).

The three-option frontier (§5): (a) 8B INT8, cheapest, meets SLO — recommended; (b) 8B FP16, +cost, marginal quality; (c) 70B, highest ceiling, breaks the power budget and triples cost — not recommended unless the accuracy floor rises.

This memo touches every module in the track and is the literal artifact JD2 produces. Being able to write and defend it is the role.

12. References

  • Pope et al., "Efficiently Scaling Transformer Inference" (2022) — the sizing math backbone.
  • MLPerf Inference results and rules — the industry's shared sizing/benchmark vocabulary.
  • NVIDIA, AWS, Google, Qualcomm inference sizing / deployment guides and TCO calculators (vendor, read critically).
  • Databricks/MosaicML and Anyscale LLM inference cost analyses ($/token, utilization).
  • The Trusted Advisor (Maister, Green, Galford) and pre-sales solutions-architecture practice — the customer-engagement craft (§6, §10).
  • The original folder's interview-prep and system-design — adjacent customer-facing material.

➡ Next: Knowledge 09 — Frontier Inference: MoE, Disaggregation, and Reasoning Models — how 2025–2026 broke this module's three quiet assumptions (dense model, colocated prefill/decode, chat-shaped outputs) and how to re-size for it. Then: interview-prep/ to rehearse defending all of this out loud.