Warmup Guide — GenAI on Edge

Zero-to-expert primer for Phase 10. Builds edge LLM deployment from the memory budget up: AWQ quantization, QLoRA/NF4 adapters for accuracy recovery, and layer streaming for models that don't fit.

Table of Contents

• Chapter 1: The Edge Budget — Doing the Arithmetic First
• Chapter 2: AWQ — Protecting the Weights That Matter
• Chapter 3: LoRA — Low-Rank Adaptation from Zero
• Chapter 4: QLoRA — NF4, Double Quantization, and Recovery Training
• Chapter 5: Adapters as Accuracy-Recovery for Quantization
• Chapter 6: Layer Streaming — Running Models Bigger Than Memory
• Chapter 7: The Edge Serving Stack
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Edge Budget — Doing the Arithmetic First

Every edge deployment starts with three numbers. For a phone: ~6–8 GB total RAM (of which an app may use 2–4), ~50–100 GB/s memory bandwidth, ~3–5 W sustained thermal budget. Now budget a LLaMA-3.2-1B chat assistant:

And decode speed (Phase 07's bandwidth formula): reading 0.7 GB per token at an achieved 30 GB/s → ~40 tok/s ceiling; FP16 would cap at ~12. Quantization is the difference between shippable and not — and that's a 1B model. For 7B-class on device, INT4 is mandatory and Chapter 6's streaming may be too.

The phase's three labs are the three levers: make INT4 accurate (AWQ), recover what INT4 lost (QLoRA adapters), and escape the memory wall entirely (streaming).

Chapter 2: AWQ — Protecting the Weights That Matter

The observation (Lin et al.): weight importance is wildly non-uniform, and the important ~1% of weight channels are identifiable from activations — channels that multiply large activation magnitudes contribute most to outputs. Keeping just those channels in FP16 nearly recovers FP16 perplexity — but mixed-precision storage is hardware-hostile.

The trick — scale instead of split: for a per-(input-)channel scale $s > 1$,

$$y = \sum_j w_j x_j = \sum_j (w_j s_j) \cdot (x_j / s_j)$$

Scale the salient weight channels up by $s$ before quantization (their relative rounding error shrinks — the quantization grid covers them with more steps), and fold $1/s$ into the preceding op (the activation side), exactly like SmoothQuant in reverse direction. Choose $s$ per channel from activation magnitude statistics: $s_j = (\bar{|x_j|})^{\alpha}$, with $\alpha \in [0,1]$ grid-searched on a small calibration set against layer reconstruction error.

Why AWQ over GPTQ on edge: no Hessian, no Cholesky — calibration is a few hundred forward passes and a 1-D grid search per layer (minutes, not hours); it composes cleanly with per-group INT4 layouts that NPU kernels want; and it generalizes well from small calibration sets (less overfitting risk than GPTQ's error compensation). GPTQ still wins some perplexity benchmarks; AWQ wins deployment ergonomics — know both (Phase 03 Lab 03 built GPTQ; Lab 01 here builds AWQ).

Chapter 3: LoRA — Low-Rank Adaptation from Zero

The fine-tuning problem: full fine-tuning of even a 1B model needs gradients + optimizer states ≈ 4× weight memory in mixed precision — far beyond edge, and produces a full model copy per task.

The hypothesis: the change a fine-tune makes to a weight matrix has low intrinsic rank. So freeze $W \in \mathbb{R}^{d \times k}$ and learn only a low-rank correction:

$$W' = W + \Delta W = W + \frac{\alpha}{r} B A, \qquad B \in \mathbb{R}^{d \times r},; A \in \mathbb{R}^{r \times k},; r \ll \min(d,k)$$

• Parameters: $r(d + k)$ vs $dk$ — for $d{=}k{=}4096, r{=}16$: 0.8% of the matrix.
• $A$ initialized Gaussian, $B$ initialized zero → $\Delta W = 0$ at start: training begins exactly at the pretrained model (no cold-start damage).
• $\alpha/r$ decouples the learning-rate-like magnitude from the rank choice.
• At inference: either merge ($W' = W + \frac{\alpha}{r}BA$, zero overhead, loses swappability) or keep separate (tiny extra matmul, adapters hot-swappable per task — the edge-relevant mode: one base model, many ~10 MB task adapters).

Chapter 4: QLoRA — NF4, Double Quantization, and Recovery Training

QLoRA's contribution: train LoRA adapters on top of a quantized base model — backprop through frozen 4-bit weights into FP16 adapters. Three pieces:

• NF4 (NormalFloat-4): weights are approximately Gaussian; a uniform INT4 grid wastes levels on the tails. NF4's 16 levels are the quantiles of a standard normal — information-theoretically optimal for normal data, equal probability mass per bin rather than equal width. Implemented as a 16-entry lookup table per dequantize; absmax-scaled per block of 64.
• Double quantization: per-block FP32 absmax constants are themselves quantized (FP8 with a second-level constant) — saves ~0.4 bits/parameter, which at 7B scale is ~370 MB. Edge cares.
• The gradient path: forward dequantizes $W_{nf4}$ block-by-block to BF16 and adds $\frac{\alpha}{r}BA$; backward flows through the dequantize (it's just a scale — differentiable) into $A, B$ only. The base stays frozen and 4-bit; optimizer states exist only for the adapters. Memory: 7B fine-tune fits in a single 24 GB GPU — or a high-end devkit.

Chapter 5: Adapters as Accuracy-Recovery for Quantization

The Phase-10 synthesis (and Lab 02's actual shape): use the adapter to repair quantization damage, not to learn a new task.

1. Quantize the base model (AWQ INT4, Lab 01). Measure the accuracy gap vs FP16.
2. Add LoRA adapters; train them on general data (or distill: minimize KL between the quantized+adapter model's logits and the FP16 model's logits) — the adapter learns a low-rank approximation of the quantization error's functional effect.
3. Ship: INT4 weights + ~1% FP16 adapter ≈ FP16 accuracy at INT4 memory/bandwidth.

This works because quantization error is structured, not random — dominated by the salient channels AWQ couldn't fully protect — and structured error is exactly what a low-rank corrector can absorb. It is the practical answer to "we lost 2 points at INT4 and QAT is too expensive": adapter recovery costs hours, not days, and never touches the base weights (one artifact serves all tasks). Related production trick: QLoRA fine-tunes a task adapter directly on the quantized base, killing two birds — task adaptation and quantization recovery — in one training run.

Chapter 6: Layer Streaming — Running Models Bigger Than Memory

The idea: a transformer executes layer-by-layer; only the current layer's weights must be resident. Keep weights in fast storage (NVMe/UFS flash), stream layer $i{+}1$ while computing layer $i$, evict behind you — a sliding window of 2–3 layers in RAM.

The governing inequality: streaming is free only if

$$t_{load}(L_{i+1}) \le t_{compute}(L_i) \quad\Leftrightarrow\quad \frac{\text{bytes}{layer}}{\text{storage BW}} \le t{compute}$$

For batch-1 decode, $t_{compute}$ per layer is tiny (it's bandwidth-bound already!) — so streaming from flash (~1–4 GB/s UFS) is typically 10–30× slower than RAM-resident decode. Honest uses: (1) prefill-heavy or batched workloads where compute per layer is large; (2) MoE models — stream only the (few) experts the router selected (the Phase 02 synergy: sparsity makes streaming viable); (3) burst memory pressure (the OS reclaimed your pages) handled gracefully instead of crashing; (4) pipelines across NPU+CPU where overlap hides part of the cost.

Engineering pieces (Lab 03): double-buffered async prefetch, pinned staging buffers, eviction policy (sequential for dense; router-driven for MoE), and the measurement-honesty part — report tok/s vs resident-set-size curves, not one cherry point.

Chapter 7: The Edge Serving Stack

How the pieces assemble on a phone (the capstone's mental model):

• Format: GGUF (llama.cpp) or vendor DLC (Phase 06) — both pack per-group quantized weights + tokenizer + metadata in one mmap-able file. mmap means the OS pages weights on demand and shares them across processes — instant warm starts.
• Runtime split: prefill on NPU (compute-bound, int-friendly); decode often lands on CPU+NPU hybrid because per-token kernel launch overhead and dequant-fused GEMV quality decide it — measure, don't assume.
• KV cache: INT8-quantized KV is standard on edge (Phase 02's budget halves); sliding-window or cache eviction for long chats.
• Thermals: sustained tok/s ≠ burst tok/s (Phase 06 Ch. 7); a chat UX needs sustained. Energy per token (joules) is a first-class metric — battery is the real SLO.
• The accuracy loop: every artifact above changed the numbers — the Phase 09 harness runs against the device (or its bit-exact simulator), not the FP16 checkpoint.

Lab Walkthrough Guidance

Order: Lab 01 (AWQ) → Lab 02 (QLoRA adapter) → Lab 03 (layer streaming).

• Lab 01: reuse Phase 03's quantize/dequantize core. Implement activation-magnitude collection → salient channel identification → the scale fold → grid-search α per layer against reconstruction error. Verify the scaled-then-quantized layer beats plain RTN on both reconstruction error and PPL, and that the $s$/$1/s$ fold is numerically invisible pre-quantization (identity test).
• Lab 02: implement the NF4 codebook (compute the 16 normal quantiles yourself and compare to the published table); block-wise absmax + double quant; then the LoRA module (test: $\Delta W=0$ at init); then recovery training with the KL-to-FP16 objective; measure the recovered-accuracy table FP16 / INT4 / INT4+adapter.
• Lab 03: build the layer-window manager with async prefetch; verify outputs are bit-identical to resident execution (correctness first); then measure tok/s vs window size and storage bandwidth, and reproduce Chapter 6's inequality empirically — find the crossover where streaming stops hurting.

Success Criteria

You are ready for Phase 11 when you can, from memory:

1. Budget weights/KV/activations for any model on a given device and state the decode tok/s ceiling from bandwidth alone.
2. Write the AWQ scaling identity, say where $1/s$ folds, and why scaling beats mixed-precision storage on NPUs.
3. Write the LoRA update with init and the role of $\alpha/r$; state the merge-vs- separate tradeoff.
4. Explain NF4 in one sentence (normal quantiles → equal probability mass per level) and what double quantization saves.
5. Defend adapter-based quantization recovery vs QAT on cost and artifact-management grounds.
6. State the streaming inequality and the three honest use cases.

Interview Q&A

Q: 7B model, phone with 8 GB RAM (3 GB usable), 60 GB/s. Deployment plan? INT4 g=128 weights ≈ 3.9 GB — doesn't fit usable RAM. Options in order: 3-bit (≈3 GB, accuracy risk — adapter recovery mandatory), a smaller base (3B INT4 ≈ 1.7 GB — usually the right product answer), or MoE-style streaming if the model were sparse. If 3.5 GB usable: INT4 + INT8 KV + mmap, expect ~60 GB/s × utilization / 3.9 GB ≈ 10–12 tok/s sustained ceiling; validate accuracy with the Phase 09 harness against FP16. The point of the answer: arithmetic first, then options, then accuracy validation.

Q: Why NF4 over INT4 for QLoRA's base, but INT4 for AWQ deployment? NF4 is statistically optimal for storage of normal-ish weights when you'll dequantize to float anyway (QLoRA's compute is BF16 — the LUT dequant is fine). NPU deployment wants integer arithmetic: uniform INT4 grids map to MAC hardware; NF4's lookup table doesn't. Storage-optimal ≠ compute-friendly — choose per execution model.

Q: Your INT4+adapter model matches FP16 on perplexity but users report worse coding. Perplexity is a distribution-level canary, not a capability metric (Phase 09 Ch. 2); recovery training distribution probably lacked code. Evaluate task suites stratified by domain, retrain the adapter with code-heavy distillation data, and add a code-eval gate to the regression CI. The deeper lesson: recovery is only as broad as its training distribution.

Q: When does layer streaming actually pay on edge? When per-layer compute time ≥ per-layer load time: prefill/batch (compute-heavy), MoE expert streaming (load 2 of 8 experts), or fast storage + slow compute combos. For batch-1 dense decode it's a 10×+ regression — say so, and propose the smaller-model or sparsity alternative instead. Knowing when not to use the clever technique is the senior signal.

References

• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023) — arXiv:2306.00978
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021) — arXiv:2106.09685
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023) — arXiv:2305.14314
• Dettmers & Zettlemoyer, The case for 4-bit precision: k-bit Inference Scaling Laws (2022) — arXiv:2212.09720
• Alizadeh et al., LLM in a flash: Efficient LLM Inference with Limited Memory (2023) — arXiv:2312.11514 — the streaming reference
• llama.cpp — read the GGUF spec and the quantization kernels (ggml-quants.c)
• bitsandbytes NF4 implementation
• Frantar & Alistarh, Sparse-Quantized Representations (SpQR) (2023) — arXiv:2306.03078