Parameters, Weights, and Checkpoints

Phase 1 · Document 02 · LLM Vocabulary and Mental Models Prev: 01 — Tokenization and Context · Next: 03 — Inference Parameters

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The number you see on a model page — "8B", "70B", "671B", "26B-A4B" — is the single most misread figure in LLM engineering. It determines how much memory you need to run the model, roughly how fast it will be, and whether it fits on your hardware at all — but it does not directly determine quality. Confusing parameters with quality (or total with active parameters) leads to buying the wrong GPU, choosing an unrunnable model, or overpaying for capacity you don't use.

You will use this every time you read a Hugging Face page, plan local inference memory, compare a dense model to a Mixture-of-Experts model, or explain to a stakeholder why "bigger" isn't automatically the answer. It is the bridge between Phase 1 vocabulary and Phase 6 hardware literacy.

2. Core Concept

Plain English

Parameters (a.k.a. weights) are the learned numbers inside a model — the knobs that training tuned so the network predicts the next token well. "8 billion parameters" means 8 billion such numbers. A checkpoint is a saved snapshot of all those numbers at a point in training, stored as files you can download and load. The architecture is the wiring diagram (how layers connect); the parameters are the values that flow through that wiring.

Technical depth

• Parameters / weights: the floating-point values in every matrix of the network — attention projections (Q, K, V, O), feed-forward layers, embeddings, layer-norm scales. Together with the architecture and tokenizer, they fully determine the model's behavior. Two checkpoints with the same architecture but different weights behave completely differently.
• Checkpoint: the serialized weights, today almost always in safetensors format (safe, fast, memory-mapped), historically in PyTorch .bin/.pt (pickle-based, unsafe), or in GGUF for llama.cpp. A checkpoint may also bundle config (architecture hyperparameters) and the tokenizer.
• Architecture: the structural definition — number of layers, hidden size, number of attention heads, head dimension, FFN size, vocabulary size, position encoding. This is what config.json describes.
• Memory from parameters: the dominant rule of thumb —

  weight memory ≈ num_parameters × bytes_per_parameter
    FP32 = 4 bytes,  FP16/BF16 = 2 bytes,  FP8/INT8 = 1 byte,  INT4 ≈ 0.5 byte
  

  So a 7B model is ~14 GB in FP16, ~7 GB in INT8, ~3.5 GB in 4-bit. (This is weights only — KV cache and overhead are extra; see Phase 6.)

Dense vs Mixture-of-Experts (the active-vs-total distinction)

• Dense model: every parameter is used for every token. A 70B dense model does 70B-worth of compute per token. Total params = active params.
• Mixture-of-Experts (MoE): the FFN is split into many "experts"; a router activates only a few per token. A model labeled 26B-A4B has 26B total parameters but only ~4B active per token. You must hold all 26B in memory (so memory tracks total), but you pay compute roughly proportional to the 4B active (so speed tracks active).

This is why MoE models can feel "fast for their size" yet still demand large amounts of RAM/VRAM — a distinction that catches almost every beginner.

3. Mental Model

ARCHITECTURE = the empty building's blueprint (rooms, wiring)
PARAMETERS   = the furniture and contents that make it usable
CHECKPOINT   = a photograph of the fully-furnished building, saved to disk
TOKENIZER    = the language spoken inside

Behavior = Architecture + Parameters + Tokenizer   (all three required)

Memory  ∝ TOTAL parameters × bytes/param      ← what fits on your hardware
Speed   ∝ ACTIVE parameters per token         ← dense: all; MoE: a few experts
Quality ∝ (training data + method + scale)    ← NOT a direct function of param count

4. Hitchhiker's Guide

What to look for first: total parameters, and (for MoE) active parameters; the checkpoint format (safetensors / GGUF); the numeric precision (BF16 / FP8 / 4-bit).

What to ignore at first: exact layer/head counts in config.json. They matter for serving tuning later, not for first-pass model selection.

What misleads beginners:

• Treating parameter count as a quality score — a well-trained 8B can beat a poorly-trained 70B on your task.
• Reading an MoE "A4B" as "this is a 4B model" — you still need memory for all experts.
• Assuming "open weights" means "open source" — see the license; weights availability ≠ permissive license.

How experts reason: params → memory budget → does it fit my hardware at a precision I trust? Then, separately, evaluate quality. They never let the headline number stand in for measured quality.

What matters in production: memory footprint at your chosen precision, throughput implications (active params, MoE routing overhead), checkpoint format compatibility with your serving engine (vLLM wants safetensors; llama.cpp wants GGUF), and license terms for the weights.

How to verify: load the checkpoint and print the parameter count; multiply by bytes/param and compare to observed VRAM. For MoE, check config.json for num_experts and num_experts_per_tok.

Questions to ask: Is this dense or MoE? What are total vs active params? What precision are the published weights? What format (safetensors/GGUF)? What's the license for commercial use?

What silently gets expensive/unreliable: loading FP32 weights when BF16 suffices (2× memory); ignoring that MoE total memory blocks concurrency; pickle-format .bin checkpoints (security risk + slow load).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Weight memory ≈ params × bytes/param. FP16/BF16 = 2 bytes → a 7B model ≈ 14 GB of weights.
• 4-bit quantization ≈ 0.5 byte/param → ~4× smaller than FP16 (a 70B ≈ 35 GB).
• MoE: memory tracks total params, compute/speed tracks active params. 26B-A4B needs 26B in memory.
• "Open weights" ≠ "open source." You may be able to download weights under a restrictive (e.g. non-commercial or use-restricted) license.
• safetensors is the safe default; PyTorch .bin/.pt use pickle and can execute arbitrary code on load.
• Quality is not a function of parameter count alone — data quality, training method, and post-training dominate on most tasks.
• A checkpoint download size on Hugging Face roughly equals its weight memory at the published precision.
• The same model is often published in multiple formats (safetensors for GPUs, GGUF for llama.cpp) and precisions.

9. Observations from Real Systems

• Hugging Face model pages show parameter count, tensor type (e.g. BF16), file format, and "Files and versions" — your primary source for memory planning.
• models.dev lists weight availability and (for some) whether a model is open-weight, separating "the lab that made it" from "can I download it."
• vLLM loads safetensors and shards weights across GPUs via tensor parallelism; it cannot directly serve GGUF.
• llama.cpp / Ollama consume GGUF checkpoints (a quantized, single-file format) — a different artifact from the original safetensors release.
• Mixtral / Qwen MoE releases headline total size but document active params; serving them needs full-total memory even though throughput resembles a smaller dense model.

10. Common Misconceptions

11. Engineering Decision Framework

Can I run this model locally?
  weight_mem = total_params × bytes_per_param(precision)
  fits = weight_mem + kv_cache + overhead  <  available_RAM/VRAM   (see Phase 6)
  If not → quantize (lower bytes/param) OR pick a smaller model OR add hardware.

Dense vs MoE for my workload?
  Latency-sensitive, modest hardware → MoE can give big-model quality at small-model speed,
    IF you can afford total-param memory.
  Memory-constrained → a smaller DENSE model may be the only thing that fits.

Which checkpoint format?
  Serving on GPUs with vLLM/TGI/SGLang → safetensors.
  Local CPU/Apple Silicon / single-file → GGUF (llama.cpp/Ollama).

Quality decision:
  NEVER infer quality from param count — run the eval harness (Phase 12).

12. Hands-On Lab

Goal

Inspect a real checkpoint, count its parameters, predict its memory at several precisions, and verify against the download size — then compute the dense-vs-MoE memory/speed split.

Prerequisites

• Python 3.10+, ~5 GB disk free, Hugging Face account for gated models (or use an open one).

Setup

pip install transformers safetensors torch

Steps

from transformers import AutoConfig

# 1. Read architecture as data (no weights download needed for config)
cfg = AutoConfig.from_pretrained("Qwen/Qwen2.5-0.5B")
print(cfg)  # hidden_size, num_hidden_layers, vocab_size, etc.

# 2. Estimate parameter count from a loaded small model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
n = sum(p.numel() for p in model.parameters())
print(f"Parameters: {n/1e9:.3f} B")

# 3. Predict memory at precisions
for name, b in [("FP32",4),("FP16/BF16",2),("INT8",1),("INT4",0.5)]:
    print(f"{name:10} ≈ {n*b/1e9:.2f} GB of weights")

Expected output

• A parameter count close to the model's advertised size.
• Memory estimates that roughly match the Hugging Face download size at the published precision.

Debugging tips

• OOM on load? Load with torch_dtype="auto" or a smaller model; you only need to count params.
• Count off from the label? Some labels round; embeddings may be tied/untied — both fine for estimation.

Extension task

Find an MoE model card (e.g. a Qwen/Mixtral MoE). From config.json read num_experts and num_experts_per_tok; compute total vs active params and explain the memory-vs-speed implication in two sentences.

Production extension

Write can_i_run(total_params, precision_bytes, available_gb, overhead_gb=2) returning fit/no-fit and the headroom. This is the seed of the Phase 6 memory calculator.

What to measure

Counted params vs advertised; predicted vs actual download size; for MoE, total vs active param ratio.

Deliverables

• A table: model → params → memory at FP16/INT8/INT4.
• A short note: dense vs MoE memory/speed trade-off for one specific model.

13. Verification Questions

Basic

1. What is the difference between parameters, weights, and a checkpoint?
2. Why is safetensors preferred over a PyTorch .bin file?
3. What does 26B-A4B tell you about memory and speed?

Applied 4. How much weight memory does a 13B model need in FP16? In 4-bit? 5. You have 24 GB of VRAM. Which is more likely to fit and why: a 30B dense model in 4-bit, or a 14B dense model in FP16?

Debugging 6. A model "should fit" by weight math but OOMs at long context. What did you forget to account for? 7. vLLM refuses to load a model you downloaded as GGUF. Why?

System design 8. You're choosing between a 70B dense and a 100B-A12B MoE for a latency-sensitive API on fixed hardware. Walk through the memory and speed reasoning.

Startup / product 9. An advisor says "just use the biggest open model for best quality." Give the two-sentence rebuttal grounded in this document, and what you'd do instead.

14. Takeaways

1. Parameters/weights are the learned numbers; a checkpoint is their saved snapshot; architecture is the wiring.
2. Weight memory ≈ params × bytes/param — the core sizing formula.
3. MoE: memory tracks total, speed tracks active parameters.
4. Parameter count ≠ quality — always evaluate.
5. "Open weights" ≠ "open source" — read the license.
6. Match checkpoint format to your serving engine: safetensors for GPU serving, GGUF for local.

15. Artifact Checklist

• Code: parameter counter + memory predictor.
• Code: can_i_run(...) fit checker.
• Benchmark result: params/memory table for 3 models.
• Notes: dense vs MoE memory/speed explanation.
• Architecture decision record: which checkpoint format you'd choose for one serving stack and why.
• Cheat card: the bytes-per-parameter table.

Next: 03 — Inference Parameters