Hardware Literacy for LLM Inference

Phase 6 · Document 01 · Local Inference Prev: 00 — Overview · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

You cannot reason about local inference without reasoning about hardware, because the two questions from 00 — does it fit? and is it fast enough? — are answered entirely by hardware properties: memory capacity (fit) and memory bandwidth (speed). An engineer who confuses "32 GB of RAM" with "fast 32 GB" will buy the wrong machine, mis-size a deployment, and misread every benchmark. This doc makes you fluent in the exact specs that matter — and, just as important, the ones that don't — so you can pick hardware, estimate throughput before you buy, and explain why the same model is 50× faster on a GPU than a CPU.

2. Core Concept

Plain-English primer: why two numbers decide everything

A model is a big pile of numbers (weights). Generating one token means: read (almost) all of those numbers out of memory, do a little math on them, produce one token, repeat (what-happens §1.C–1.D). So two hardware numbers dominate:

• Memory capacity (GB) — can the pile of numbers (+ KV cache + overhead) even fit? If not, nothing runs (or it spills to disk and crawls).
• Memory bandwidth (GB/s) — how fast can the chip read that pile? Because decode re-reads (almost) all weights per token, your ceiling is roughly:

max decode tokens/sec  ≈  memory_bandwidth (GB/s)  ÷  model_size_in_memory (GB)

Example: a 4-bit 8B model is ~5 GB. On a CPU at ~60 GB/s → ~12 tok/s ceiling. On an RTX 4090 at ~1,000 GB/s → ~200 tok/s ceiling. Same model, ~16× faster — purely because of bandwidth. That formula is the single most useful thing in this phase.

Compute (FLOPs) matters too — but mostly for prefill

GPUs also have enormous compute (tensor cores doing trillions of FLOPs/sec). Compute dominates prefill (reading your prompt — big parallel matmuls, Phase 2.07), which sets TTFT. Decode is bandwidth-bound and sets TPOT. So: big prompt, short answer → compute/prefill-bound; short prompt, long answer → bandwidth/decode-bound. A good chip needs both, but capacity+bandwidth is what makes or breaks local LLMs.

The four hardware families

CUDA (NVIDIA), ROCm (AMD), and Metal (Apple) are the GPU programming layers; engines compile kernels for whichever you have. CUDA has by far the most mature LLM ecosystem (vLLM, FlashAttention, TensorRT-LLM), which is why datacenters are NVIDIA-dominated.

3. Mental Model

   ONE TOKEN (decode) = read ~all weights from memory → tiny math → emit token → repeat
                              │                              │
                  MEMORY BANDWIDTH (GB/s)            COMPUTE (FLOPs/s)
                  sets TPOT / tokens-sec             sets TTFT / prefill
                              │
        tokens/sec  ≈  bandwidth ÷ model_GB        (the local-inference speed law)

   CAPACITY (GB)  → DOES IT FIT?      BANDWIDTH (GB/s) → IS IT FAST ENOUGH?
   CPU RAM: big & cheap & SLOW band   GPU VRAM: smaller & EXPENSIVE & FAST band
   Apple unified: big AND mid-fast (best of both for capacity)

Mnemonic: capacity = fit, bandwidth = speed. Buy/choose for both.

4. Hitchhiker's Guide

What to look for first: the chip's memory capacity and memory bandwidth. For a GPU, also VRAM size. These two predict fit and speed before you run anything.

What to ignore at first: core counts, clock speeds, marketing TFLOPS for fp32 (LLMs use fp16/bf16/int), and "AI TOPS" on NPUs that no LLM engine uses yet. They rarely change your decision.

What misleads beginners:

• Confusing capacity with bandwidth. A 64 GB CPU runs big models slowly; a 24 GB GPU runs medium models fast.
• Ignoring the offload cliff. If a model is 90% on GPU and 10% on CPU (layers offloaded to RAM), the slow 10% can halve throughput — partial offload is non-linear (08).
• Assuming PCIe speed matters for single-GPU inference. It mostly doesn't (weights live in VRAM); it matters for multi-GPU collective ops, where NVLink ≫ PCIe.

How experts reason: estimate tokens/sec ≈ bandwidth ÷ model_GB, check model_GB + KV + overhead ≤ VRAM × 0.8, then pick the cheapest chip that clears both with headroom. For multi-user, they switch to thinking in throughput (batched tokens/sec, Phase 7) rather than single-stream speed.

What matters in production: sustained (not burst) bandwidth, thermal throttling under long load, ECC/reliability for 24/7, and GPU utilization (an idle datacenter GPU is the most expensive thing you own).

How to debug/verify: nvidia-smi (VRAM, utilization, power, throttle), system_profiler/Activity Monitor (Mac), lscpu/free -h (CPU); compare achieved tok/s to the bandwidth ceiling — a big gap means you're prefill-bound, offloading, or throttling.

Questions to ask vendors: memory bandwidth (not just size), sustained vs peak, NVLink topology for multi-GPU, and whether the model fits at your context+concurrency.

What silently gets expensive: under-utilized GPUs, CPU offload, and buying capacity you can't feed with bandwidth.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Decode tokens/sec ≈ memory_bandwidth ÷ model_size_in_memory. Memorize this; it predicts speed across all hardware.
• Capacity decides fit; bandwidth decides speed. They are different numbers and often pull opposite ways (CPU = big+slow, GPU = small+fast).
• CPU RAM bandwidth (~50–100 GB/s) is ~10–30× lower than GPU VRAM (~0.4–3.35 TB/s) — the root cause of "GPU is faster."
• Apple unified memory removes the RAM/VRAM split: a 64 GB Mac can give the GPU ~48+ GB for weights — impossible on a same-priced discrete GPU.
• Prefill is compute-bound (TTFT); decode is bandwidth-bound (TPOT) (Phase 2.07).
• Partial CPU offload is non-linear — a small fraction on CPU can dominate latency (08).
• For multi-GPU, the interconnect (NVLink) matters more than PCIe; for single-GPU it's irrelevant.
• CUDA's ecosystem maturity (FlashAttention, vLLM, TensorRT-LLM) is a real, decision-level advantage over ROCm today.

9. Observations from Real Systems

• Datacenters are NVIDIA-dominated precisely because CUDA + HBM bandwidth + NVLink + a mature kernel ecosystem (FlashAttention, vLLM, TensorRT-LLM) compound.
• Apple Silicon punches above its price for local LLMs — a MacBook with 64–128 GB unified memory runs models that would need multiple discrete GPUs (05).
• Consumer RTX cards (24 GB) are the sweet spot for personal serving; datacenter A100/H100/H200 (80–141 GB) for production (00 tiers).
• llama.cpp benchmarks consistently track the bandwidth ceiling — community tok/s numbers line up with bandwidth ÷ model_GB, confirming the law.
• ROCm support in llama.cpp/vLLM has matured but still trails CUDA in coverage and stability — a vendor question to ask.

10. Common Misconceptions

11. Engineering Decision Framework

GOAL: pick/justify hardware for model M at quant Q, context C, concurrency N.
 1. model_GB = params × bytes_per_param(Q)                      [02,06]
 2. need_GB  = model_GB + KV(C,N) + overhead                    [02]
 3. CAPACITY: need_GB ≤ device_memory × 0.8 ?  (else: quantize/smaller/bigger device)
 4. SPEED:   est_tok/s ≈ device_bandwidth ÷ model_GB
             ≥ task requirement?  (else: faster device / spec-decode [07] / accept)
 5. SCALE:   N users → throughput thinking + batching → GPU + vLLM   [Phase 7]
 6. CHOOSE cheapest device clearing (3) and (4) with headroom.

12. Hands-On Lab

Goal

Profile your hardware, compute the bandwidth ceiling for a model, then measure actual tokens/sec and explain the gap.

Prerequisites

• A machine with a local engine installed (00 lab) and one small model.

Setup

# Inspect hardware
nvidia-smi                                  # GPU: VRAM, util, power (NVIDIA)
system_profiler SPHardwareDataType          # Mac: chip, unified memory
lscpu && free -h                            # CPU: cores, RAM (Linux)

Steps

1. Record specs: memory capacity (GB) and memory bandwidth (GB/s — look up your chip's datasheet; e.g. RTX 4090 ≈ 1,008 GB/s, M3 Max ≈ 400 GB/s, DDR5 desktop ≈ 60–90 GB/s).
2. Compute the ceiling: for your model's in-memory size model_GB, ceiling = bandwidth ÷ model_GB.
3. Measure actual tokens/sec on a short prompt, long answer (decode-bound) request (reuse the 00 lab script).
4. Compute efficiency: actual ÷ ceiling. Expect ~50–80% on GPU; much lower implies offload, throttling, or prefill-bound.
5. Probe prefill: rerun with a long prompt, short answer and watch TTFT rise — that's the compute/prefill axis.

Expected output

A table: device | capacity | bandwidth | model_GB | ceiling tok/s | actual tok/s | efficiency.

Debugging tips

• Efficiency ≪ 50% → check -ngl (GPU layers), offload, thermal throttle (nvidia-smi clocks), or that you measured a prefill-heavy request (08).

Extension task

Plot ceiling vs actual for two models of different size on the same device — the line should track bandwidth ÷ model_GB.

Production extension

On a GPU, run 1 vs 8 concurrent requests under vLLM and show throughput (total tok/s) rises even as per-request tok/s falls — the batching economics from what-happens §1.D.

What to measure

Capacity, bandwidth, ceiling vs actual tok/s, TTFT, efficiency, throughput under concurrency.

Deliverables

• A hardware profile with capacity + bandwidth.
• A ceiling-vs-actual table and a one-line explanation of the gap.

13. Verification Questions

Basic

1. Which hardware number sets fit, and which sets decode speed?
2. Write the tokens/sec ceiling formula.
3. What are CUDA, ROCm, and Metal?

Applied 4. A 4-bit 8B model (~5 GB) on a 60 GB/s CPU vs a 1,000 GB/s GPU — estimate tok/s for each. 5. Why can a 64 GB Mac run a model a 24 GB GPU can't, yet sometimes generate slower?

Debugging 6. Achieved tok/s is 20% of the bandwidth ceiling. List three causes. 7. TTFT is huge but tok/s is fine. What's the bottleneck and why?

System design 8. Spec a box for a 70B 4-bit internal assistant serving 50 concurrent users; justify capacity, bandwidth, and interconnect.

Startup / product 9. Your API bill is $X/mo at volume V. Which hardware specs determine whether owning GPUs is cheaper, and how?

14. Takeaways

1. Capacity = fit; bandwidth = speed. Two different numbers; size for both.
2. tokens/sec ≈ bandwidth ÷ model_GB — the local-inference speed law.
3. Prefill is compute-bound (TTFT); decode is bandwidth-bound (TPOT).
4. GPU ≫ CPU because of bandwidth, not capacity; Apple unified memory wins on capacity-per-dollar.
5. CUDA's ecosystem is a real advantage; NVLink matters only for multi-GPU.

15. Artifact Checklist

• Hardware profile: capacity + bandwidth for your device(s).
• Ceiling-vs-actual tokens/sec table with efficiency.
• TTFT measurement showing the prefill/compute axis.
• Hardware recommendation for one target model + workload.
• (Production) throughput-under-concurrency chart.

Up: Phase 6 Index · Next: 02 — RAM, VRAM, and Unified Memory