RAM, VRAM, Unified Memory — Sizing the Footprint

Phase 6 · Document 02 · Local Inference Prev: 01 — Hardware Literacy · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

"Does it fit?" is the gate that opens or closes every local deployment (00), and getting the answer wrong by 20% is the difference between a model that runs and one that OOMs (out-of-memory) — or worse, one that loads fine in your test and crashes in production when a long conversation or a few concurrent users blow up the KV cache. This doc turns memory sizing from a guess into arithmetic you can do on a napkin: weights, KV cache, and overhead, each with a formula, worked examples, and the headroom rule. It is the analytical core of Phase 6 and the input to every hardware (01) and quantization (06) decision.

2. Core Concept

Plain-English primer: the three buckets

When a model runs, memory is consumed by exactly three things. Get these and you can size anything:

Total memory ≈  WEIGHTS  +  KV CACHE  +  OVERHEAD
                (fixed)     (grows with     (engine, activations,
                            context×users)   projectors, draft model)

1) Weights — the big fixed cost. The model's parameters, each stored in some number of bytes per parameter set by precision/quantization:

weights_GB ≈ params × bytes_per_param / 1e9

bytes_per_param:  FP32=4 · FP16/BF16=2 · FP8/INT8=1 · INT4/4-bit≈0.5
                  (GGUF k-quants are ~0.5–1.1 depending on variant — see [06])

Worked: an 8B model → FP16 8e9×2 = 16 GB; 4-bit 8e9×0.5 = 4 GB. A 70B → FP16 140 GB; 4-bit ~35–40 GB.

2) KV cache — the cost everyone forgets. Every token you've processed stores a key and a value vector at every layer so attention doesn't recompute the past (Phase 2.06). It grows linearly with sequence length and concurrency:

kv_bytes ≈ 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes_per_elem × seq_len × batch

• n_kv_heads × head_dim is the per-token KV width. With Grouped-Query Attention (GQA), n_kv_heads ≪ n_heads, which shrinks the KV cache massively (a key reason modern models use it).
• bytes_per_elem is the KV precision (often FP16=2; many engines support KV cache quantization to 8-bit/4-bit to save memory).
• batch = concurrent sequences. 50 users at 8k context each can need more KV than the weights themselves.

Worked (Llama-3-8B-ish: 32 layers, 8 KV heads, head_dim 128, FP16 KV): per token ≈ 2×32×8×128×2 = 131,072 bytes ≈ 0.13 MB. At 8,192 tokens → ~1.0 GB for one sequence. At 32 concurrent 8k sequences → ~32 GB — more than the FP16 weights would be irrelevant; this alone exceeds a 24 GB GPU.

3) Overhead — the rest. Engine/runtime buffers (~0.5–2 GB), activations (scratch memory for the forward pass, scales with batch×context), the multimodal projector (vision/audio encoders, if any), and any draft/MTP model for speculative decoding (07). Budget a few GB and verify by measurement.

The full estimation formula (from the spec)

Total approximate memory =
    model weights
  + KV cache (× context × concurrency, ÷ GQA, × KV-precision)
  + runtime overhead
  + activation overhead (batch × context)
  + multimodal projector overhead
  + draft / MTP / speculation overhead
  + safety headroom (≥20%)

Headroom — never fill to 100%

KV grows during generation, the OS needs memory, and fragmentation wastes some. Leave ≥20% free (or 2–4 GB minimum):

available_for_model ≈ device_memory × 0.8 − fixed_overhead

If weights + max_KV doesn't clear this, you quantize harder (06), cap context/concurrency, quantize the KV cache, or pick a smaller model.

3. Mental Model

   DEVICE MEMORY (RAM / VRAM / UNIFIED)
   ┌───────────────────────────────────────────────────────────┐
   │ WEIGHTS  (fixed = params × bytes/param)                     │  ← shrink with quant [06]
   ├───────────────────────────────────────────────────────────┤
   │ KV CACHE (= 2·L·kv_heads·head_dim·bytes·SEQ·BATCH)          │  ← the sneaky one; grows!
   ├───────────────────────────────────────────────────────────┤
   │ OVERHEAD (engine + activations + projector + draft)         │
   ├───────────────────────────────────────────────────────────┤
   │ ░░ HEADROOM ≥20% ░░  (OS, KV growth, fragmentation)         │  ← never consume this
   └───────────────────────────────────────────────────────────┘
   FIT ⇔ weights + maxKV + overhead ≤ memory × 0.8
   "It loaded" only proves batch=1, short-context fit. Size for MAX seq × MAX users.

4. Hitchhiker's Guide

What to look for first: weights_GB (params × bytes/param) and max KV at your real context × concurrency. Those two dominate.

What to ignore at first: exact activation bytes and projector size — budget a couple GB and measure; don't over-model them before you've checked the big two.

What misleads beginners:

• Sizing only the weights. The classic OOM: weights fit, then a long chat or 10 users blow the KV cache.
• Forgetting GQA. Old MHA math overestimates KV by 4–8×; modern GQA models are far lighter.
• Ignoring KV precision. FP16 KV is the default, but quantizing KV to int8 roughly halves it.
• Assuming unified memory is "free" capacity. On a Mac, the OS + apps share that pool; you don't get all of it.

How experts reason: they compute weights + KV(max_ctx, max_batch) + overhead, demand ≥20% headroom, and treat context length and concurrency as first-class memory inputs — then choose quant level to make it fit.

What matters in production: the worst case, not the demo. Set a hard context cap and concurrency cap so KV can't exceed budget; consider KV quantization and paged KV (Phase 7.04) to pack more sequences.

How to debug/verify: watch memory live during a long generation and under concurrent load; compare to your estimate; if it OOMs late, it's KV (08).

Questions to ask: What's my max context and max concurrent requests? Does the model use GQA (how many KV heads)? Can the engine quantize the KV cache? What's the projector/draft overhead?

What silently gets expensive: long-context features and concurrency — both scale KV linearly and are the usual cause of "it worked yesterday."

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• weights_GB ≈ params × bytes_per_param / 1e9 — FP16=2, INT8=1, 4-bit≈0.5 bytes/param.
• kv_bytes ≈ 2 × layers × kv_heads × head_dim × bytes × seq_len × batch — linear in context and concurrency.
• GQA shrinks KV by n_heads / n_kv_heads (often 4–8×) — always use n_kv_heads, not n_heads.
• KV can exceed the weights at long context or high concurrency — size for the worst case.
• "It loaded" only proves batch-1, short-context fit — the OOM comes later from KV growth.
• MoE: all experts must be in memory even though only the active ones compute per token (Phase 2.08) — size by total, not active, params.
• Leave ≥20% headroom (or 2–4 GB) for OS, KV growth, fragmentation.
• KV-cache quantization (int8/int4) roughly halves/quarters KV at a small quality cost.

9. Observations from Real Systems

• vLLM's PagedAttention exists precisely because naïve contiguous KV allocation wastes memory and fragments under varied sequence lengths (Phase 7.04).
• Unsloth/HF GGUF pages publish per-quant memory requirements — the very params × bytes/param table you can now derive.
• llama.cpp exposes -c (context), --parallel (concurrency), and --cache-type-k/v (KV quant) — the three knobs that set KV memory (03).
• Long-context launches (100k–1M tokens) are gated by KV memory, not weights — providers cap served context for exactly this reason (Phase 5.10).
• Apple unified memory lets a 64 GB Mac hold a 4-bit 70B (~40 GB) plus KV — but you must leave the OS its share.

10. Common Misconceptions

11. Engineering Decision Framework

SIZE(model M, quant Q, max_context C, max_concurrency N):
 1. weights_GB = params × bytes(Q) / 1e9                       # MoE: use TOTAL params
 2. kv_GB      = 2 × layers × kv_heads × head_dim × kvbytes × C × N / 1e9
 3. overhead   = engine(~1–2GB) + activations + projector + draft   [07]
 4. need_GB    = weights_GB + kv_GB + overhead
 5. FIT?  need_GB ≤ device_memory × 0.8
       NO → in order: lower quant [06] → quantize KV → cap C or N → smaller model → bigger device
       YES → proceed; verify by measurement [12]

12. Hands-On Lab

Goal

Build a memory calculator, predict a model's footprint at several context/concurrency settings, then validate against measured memory.

Prerequisites

• A model's config.json (HF) for num_hidden_layers, num_key_value_heads, hidden_size/head_dim; a local engine to measure.

Setup

pip install huggingface_hub
# grab a config to read the architecture numbers
python -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('meta-llama/Llama-3.1-8B-Instruct','config.json'))"

Steps

1. Implement the formula:

def size_gb(params, bytes_per_param, layers, kv_heads, head_dim,
            seq_len, batch, kv_bytes=2, overhead_gb=1.5):
    weights = params * bytes_per_param / 1e9
    kv = (2 * layers * kv_heads * head_dim * kv_bytes * seq_len * batch) / 1e9
    return weights, kv, weights + kv + overhead_gb

# Llama-3.1-8B @ 4-bit, 8k ctx, 1 user
print(size_gb(8e9, 0.5, layers=32, kv_heads=8, head_dim=128, seq_len=8192, batch=1))

2. Sweep context (2k→32k) and concurrency (1→32); tabulate need_GB.
3. Find the fit frontier: for your device, mark where need_GB > memory × 0.8.
4. Validate: run the model at a chosen (ctx, batch) and watch peak memory (nvidia-smi/Activity Monitor). Compare to prediction (expect within ~10–20%).
5. Apply a lever: drop to a smaller quant or quantize KV; re-predict and re-measure.

Expected output

A need_GB table over (context × concurrency) with a clear fit/no-fit boundary that matches measurement.

Debugging tips

• Prediction too low → you used n_heads instead of n_kv_heads, or forgot activations under large batch.
• Measured ≫ predicted at high batch → activation memory; add a batch×context term.

Extension task

Add KV quantization (kv_bytes=1) and show how many more concurrent users fit.

Production extension

Encode the calculator as an admission check in a serving layer: reject/queue requests whose projected KV would exceed budget (Phase 7).

What to measure

Predicted vs actual peak memory across (ctx, batch); fit boundary; effect of each lever.

Deliverables

• A reusable memory calculator (code).
• A fit table/heatmap over context × concurrency for your device.
• A validation note: predicted vs measured, and which lever you'd pull.

13. Verification Questions

Basic

1. What three buckets sum to total inference memory?
2. Write the weights and KV-cache formulas.
3. Why does "it loaded" not prove it fits?

Applied 4. Llama-3-8B (32 layers, 8 KV heads, head_dim 128, FP16 KV) at 16k context, 4 users — estimate KV GB. 5. A 26B-A4B MoE: how much weight memory at 4-bit, and why isn't it "4B"?

Debugging 6. Model OOMs only after long conversations. Cause and two fixes. 7. Your KV estimate is 6× too high. What did you likely miss?

System design 8. Design memory admission control that caps KV so N users at C context never OOM a 24 GB GPU.

Startup / product 9. Your product promises 100k-token context. What memory does that imply per user, and how does it change your hardware/unit-economics?

14. Takeaways

1. Memory = weights + KV cache + overhead + headroom.
2. weights ≈ params × bytes/param; quantization moves bytes/param from 2 → ~0.5.
3. KV cache scales with context × concurrency and is the usual OOM — size for the worst case.
4. Use n_kv_heads (GQA) and consider KV quantization to fit more.
5. MoE loads all experts — size by total params.
6. Leave ≥20% headroom and validate predictions by measurement.

15. Artifact Checklist

• Memory calculator implementing the full formula.
• Fit table/heatmap over context × concurrency for your device.
• Predicted-vs-measured validation note.
• Chosen levers (quant level, KV quant, caps) to make a target model fit.
• Per-user KV cost for any long-context promise.

Up: Phase 6 Index · Next: 03 — GGUF and llama.cpp