The Hitchhiker's Guide — Phase 10: GenAI on Edge & Embedded Systems

Section 1: The Edge Memory Constraint — Why Everything Changes

A 7B LLM:

• FP32: 28 GB
• FP16/BF16: 14 GB
• INT8: 7 GB
• INT4: 3.5 GB
• INT2 (AQLM): ~1.75 GB

Snapdragon 8 Gen 3: 12 GB total DRAM shared with OS + all apps. Effective model budget: 2–4 GB.

This makes INT4 the minimum for 7B models on current flagship smartphones. For 13B+ models, you need layer streaming or INT2.

The Memory Bandwidth Bottleneck

At batch_size=1 (typical for mobile), LLM decode is memory-bandwidth-bound:

• Each decode step reads all model weights once: 3.5 GB for 7B INT4
• At 77 GB/s (Snapdragon DRAM): 3.5 GB / 77 GB/s = 45 ms/token → 22 tokens/sec
• HTP can't help much here — the bottleneck is reading weights from DRAM, not compute

This is why mobile LLM speed scales linearly with DRAM bandwidth — compute capacity doesn't matter.

Section 2: INT4 Quantization for Edge — Format Comparison

GPTQ (Generative Pre-Trained Quantization)

Based on second-order information (Hessian of the reconstruction error):

$$W_q = \arg\min_{W_q \in \mathcal{Q}} |W - W_q|{H}^2 = \arg\min{W_q} (W - W_q) H (W - W_q)^T$$

where $H$ is the input activation covariance matrix (Hessian proxy).

Algorithm (OBQ/GPTQ):

1. Compute $H = X^T X$ from calibration data
2. For each weight column $j$ in order:
   • Quantize $w_j$ to nearest INT4 value
   • Compute quantization error $\delta_j = w_j - \text{quantize}(w_j)$
   • Propagate error to remaining columns: $w_{j'} \mathrel{-}= \delta_j \cdot H_{jj'}^{-1} H_{jj}$ for $j' > j$

Properties:

• Group size $g=128$ per row means each group of 128 weights shares a scale
• Best accuracy among PTQ INT4 methods for language models
• Quantization time: ~1 hour for 7B model on A100

AWQ (Activation-Aware Weight Quantization)

Observation: not all weights are equally important. Weights corresponding to large-magnitude activations ("salient channels") cause more quantization error.

AWQ algorithm:

1. Find salient input channels: $\text{saliency}j = |X{:,j}|2 \cdot |W{j,:}|_2$ (activation × weight magnitude)
2. For top-1% salient channels: scale UP weights before quantization, scale DOWN activations
   • $W' = W \cdot \text{diag}(s)$, $X' = X \cdot \text{diag}(s)^{-1}$ where $s_j \propto \text{saliency}_j^{0.5}$
   • The product $WX = W'X'$ is identical, but $W'$ is "flatter" (easier to quantize)
3. Quantize $W'$ to INT4 with naive per-group scaling

Properties:

• No per-weight Hessian computation → fast (10 min for 7B vs 1 hour for GPTQ)
• Similar accuracy to GPTQ for most tasks
• Better for tasks sensitive to specific feature channels (e.g., math reasoning)

NF4 (Normal Float 4-bit)

Standard INT4 divides $[-\text{absmax}, \text{absmax}]$ into 16 equal intervals. But weight distributions are approximately Gaussian — the tails are rarely used.

NF4 places quantization levels at the quantiles of a standard normal distribution: $$q_i = \Phi^{-1}\left(\frac{i + 0.5}{2^b}\right), \quad i = 0, \ldots, 2^b - 1$$

This is information-theoretically optimal for normally distributed data — each quantization level covers the same probability mass.

# The 16 NF4 levels (precomputed from normal distribution quantiles):
NF4_LEVELS = [
    -1.0, -0.6961928009986877, -0.5250730514526367, -0.39491748809814453,
    -0.28444138169288635, -0.18477343022823334, -0.09105003625154495, 0.0,
    0.07958029955625534, 0.16093020141124725, 0.24611230194568634, 0.33791524171829224,
    0.44070982933044434, 0.5626170039176941, 0.7229568362236023, 1.0
]

NF4 is used by bitsandbytes in QLoRA — enables high-quality INT4 quantization of the base model while training LoRA adapters in BF16.

Section 3: LoRA Mathematics

The Low-Rank Hypothesis

Pre-trained model weight matrices $W_0 \in \mathbb{R}^{d \times k}$ have been shown to have low intrinsic dimensionality — the gradient updates $\Delta W$ during fine-tuning are approximately low-rank.

LoRA exploits this: instead of updating $W_0$ directly, we learn: $$W = W_0 + \Delta W = W_0 + BA$$

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d, k)$.

Trainable parameters: $r(d + k)$ instead of $dk$ — for $d=k=4096$, $r=64$: 524,288 instead of 16,777,216 — 32× reduction.

Initialization: $A \sim \mathcal{N}(0, \sigma^2)$, $B = 0$ — ensures $\Delta W = 0$ at start (full model capability preserved).

Scaling: $\Delta W = \frac{\alpha}{r} BA$ where $\alpha$ is a hyperparameter (typically $\alpha = 2r$).

QLoRA: Quantized Base + FP16 Adapters

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# Load model in NF4 (INT4 quantized)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # NF4 quantization
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in BF16 (not quant)
    bnb_4bit_use_double_quant=True,    # quantize the scale factors too
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare for training (needed for gradient checkpointing with quant models)
model = prepare_model_for_kbit_training(model)

# Add LoRA adapters (trained in BF16, base model stays in INT4)
lora_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 1,276,928,000 || trainable%: 3.28

Double quantization: bnb_4bit_use_double_quant=True quantizes the INT4 scale factors (which are FP32) with INT8 → additional 0.5 bits/weight savings.

LoRA Merging for Deployment

After training, merge LoRA weights back into the base model for efficient deployment (no adapter overhead at inference):

from peft import PeftModel

# Load base + adapters
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
model = PeftModel.from_pretrained(base_model, "./lora_checkpoint")

# Merge
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged_model")

Section 4: Memory-Constrained Inference Strategies

Layer Streaming with Prefetching

The key insight for layer streaming: GPU compute time for one layer > time to load next layer from NVMe.

Timeline (no prefetch):
[Load L1][Compute L1][Load L2][Compute L2][Load L3] ...

Timeline (with prefetch):
[Load L1][Compute L1 + Load L2][Compute L2 + Load L3] ...
                               ↑ overlap load and compute

If $T_{compute} > T_{load}$: computation is the bottleneck → prefetching hides all load latency → same speed as fully-loaded inference.

For a 7B LLM INT4 per layer: $3500 \text{ MB} / 32 \text{ layers} \approx 109 \text{ MB/layer}$. At eMMC bandwidth 1 GB/s: 109 ms to load. At Snapdragon HTP: ~50 ms to compute per layer for batch=1. So streaming adds ~50% overhead even with prefetch (load is slower than compute).

Sliding Window Attention (Infinite Context, Bounded Memory)

Standard KV cache grows unboundedly with sequence length. Sliding window attention limits the KV cache to a fixed window of $W$ tokens:

def sliding_window_attention(q, k, v, window_size=512):
    """Only attend to the last window_size tokens."""
    seq_len = q.shape[1]
    # Create causal mask that also masks tokens outside the window
    mask = torch.ones(seq_len, seq_len, dtype=torch.bool)
    mask = torch.tril(mask)  # causal
    # Block tokens older than window_size
    for i in range(seq_len):
        if i >= window_size:
            mask[i, :i - window_size] = False
    return scaled_dot_product_attention(q, k, v, attn_mask=~mask)

StreamingLLM extension: keep "attention sinks" (first 4 tokens, which accumulate high attention weights) + sliding window. Preserves positional encoding stability and prevents output degradation in very long sequences.

Section 5: Interview Pitfalls

Section 6: Resources

1. QLoRA paper — Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (NeurIPS 2023)
2. LoRA paper — Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (ICLR 2022)
3. GPTQ paper — Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (ICLR 2023)
4. AWQ paper — Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (MLSys 2024)
5. StreamingLLM — Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (ICLR 2024)
6. bitsandbytes library — Tim Dettmers — https://github.com/TimDettmers/bitsandbytes
7. llama.cpp — https://github.com/ggerganov/llama.cpp — reference implementation for edge LLM inference
8. MLC LLM — https://github.com/mlc-ai/mlc-llm — TVM-based mobile LLM deployment
9. Qualcomm AI Hub LLM models — https://aihub.qualcomm.com/mobile-accelerated/llm — official Snapdragon-optimized LLMs