Lab 06 — Tokenizer & KV-Cache Calculator

Runs fully offline (Python 3 only). Build the intuition + the math behind tokens, memory, and decode speed — the fundamentals you'll be grilled on. Concepts: Phase 2, Phase 6.02, cheatsheet 06.

Goal

Estimate, from first principles: model weight memory (by quantization), KV-cache memory (by context × batch), total VRAM, and decode tok/s via the bandwidth law.

Files

lab-06-tokenizer-and-kv-cache/
  README.md
  kv_cache_calc.py      ← runnable: VRAM + KV-cache + decode-speed estimator

Run

# 8B default
python3 kv_cache_calc.py
# 70B Q4, 32k context, batch 8, high-bandwidth GPU
python3 kv_cache_calc.py --params 70 --layers 80 --kv-heads 8 --head-dim 128 \
    --ctx 32768 --batch 8 --bits 4 --bandwidth 3350

Exercises

  1. Will it fit? Find the max context that fits a 13B model in 24 GB at FP16 vs Q4. (Vary --ctx until total < 24.)
  2. KV dominates: Show that at long context + batch, KV-cache exceeds weights. (The output's "KV vs context" table proves it.)
  3. Speed law: Estimate tok/s for a 7B Q4 model at 200 GB/s vs 3350 GB/s. Explain why decode is bandwidth-bound.
  4. GQA effect: Halve --kv-heads (grouped-query attention) and see KV-cache shrink — explain why.
  5. Tokenizer (stretch): add a function that estimates tokens from text (~chars/4) and compute input cost at a given $/Mtok.

Deliverables

  • A table of (model, quant, context, batch) → total VRAM for 3 configs.
  • A tok/s estimate for two bandwidths, with the bandwidth-law explanation.
  • A one-paragraph note: "why KV-cache, not weights, limits long context."

Why it matters (interview)

Being able to whiteboard "weights = params×bytes, KV-cache = 2·layers·kv_heads·head_dim·ctx·batch·bytes, decode tok/s ≈ bandwidth ÷ weight-bytes" is a top serving/infra signal (interview-prep 03–04).