Phase 10 — GenAI on Edge & Embedded Systems
Difficulty: ⭐⭐⭐⭐⭐
Estimated Time: 3 weeks (70–90 hours)
Roles Supported: Core for Qualcomm — all Snapdragon-based AI is on-device; this phase covers the unique constraints and solutions for edge deployment
Why This Phase Exists
The "edge" changes everything. A 7B LLM needs 14 GB in FP16 — more than the DRAM on any current smartphone. The Snapdragon 8 Gen 3 has 12 GB shared DRAM for the entire phone OS. This is not a software problem you optimize around with better code; it requires fundamentally different algorithms.
This phase teaches the engineering decisions that make LLMs possible on edge devices: aggressive quantization (INT4 → 3.5B bytes for 7B parameters), speculative decoding, memory splitting across chip and DRAM, on-chip VTCM tiling, and LoRA-based accuracy recovery when aggressive quantization degrades quality below acceptable thresholds.
The GenAI on edge space is Qualcomm's core business differentiation. Every major phone OEM (Samsung, Xiaomi, OnePlus, OPPO) uses Snapdragon for their on-device AI features. A Senior Staff engineer in this space must understand the full stack from silicon capabilities to end-user accuracy thresholds.
Concepts
- Memory budget for edge: smartphone DRAM = 8–16 GB shared with OS + all apps; effective model DRAM budget = 2–4 GB; requires INT4 or INT2 quantization for 7B+ models
- VTCM (Vector Tightly Coupled Memory): 8 MB on-chip SRAM on HTP; much faster than DRAM (>4× bandwidth); weights that fit in VTCM during a layer's computation avoid DRAM; tiling strategy critical
- Prefill chunking: process long prompts in chunks to limit peak memory; each chunk = one forward pass; KV cache fills incrementally
- Layer-by-layer offload: keep only the current layer's weights in DRAM; offload unused layers to flash/eMMC storage; latency ~10× worse but enables 70B models on 4 GB DRAM
- Quantization formats for edge: GGUF (llama.cpp), GPTQ, AWQ, QNN DLC — each has different trade-offs for edge hardware; GGUF is most portable, DLC is optimal for Snapdragon
- LoRA (Low-Rank Adaptation): train low-rank adapter matrices $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$ for each weight; $W' = W + BA$; only $r \ll \min(d,k)$ new parameters; enables domain-specific fine-tuning with <1% of full parameters
- QLoRA: quantize the base model to INT4 (NF4 format) and fine-tune LoRA adapters in BF16; significantly reduces GPU memory for fine-tuning; PEFT library
- NF4 (Normal Float 4-bit): information-theoretically optimal quantization for normally distributed weights; ~0.5 PPL better than INT4 for same bit-budget
- Accuracy recovery strategies: LoRA fine-tuning on domain data, knowledge distillation from FP16 teacher, QAT with domain data, GPTQ per-layer accuracy recovery
- On-device learning: fine-tune LoRA on the device using only local data; privacy-preserving; federated learning; requires INT8 gradient computation
- Memory-constrained attention: ring attention (distribute sequence across devices), sliding window (limit effective context), streaming LLM (fixed-size attention window with sink tokens)
- Prefill/decode asymmetry on NPU: prefill is compute-bound (many tokens at once → large matmuls); decode is memory-bound (one token at a time → small matmuls); requires different optimization strategies
Labs
Lab 01 — On-Device LLM: INT4 Quantize, Benchmark, Accuracy Delta
| Field | Value |
|---|---|
| Goal | Quantize LLaMA-3.2-1B to multiple INT4 formats (GPTQ, AWQ, NF4/QLoRA), benchmark inference on CPU and AI Hub NPU simulator, and comprehensively measure accuracy degradation vs FP16 baseline |
| Concepts | GPTQ vs AWQ vs NF4 at INT4, memory footprint calculation, tokens/sec on NPU, accuracy delta (PPL, MMLU, WinoGrande), size/accuracy/latency trade-off table |
| Steps | 1. Compute FP16 baseline metrics (PPL, MMLU 5-shot, WinoGrande); 2. Apply GPTQ INT4 (g=128) using auto-gptq; 3. Apply AWQ INT4 (g=128) using autoawq; 4. Apply NF4 quantization via bitsandbytes; 5. For each format: (a) measure model file size, (b) measure DRAM usage during inference, (c) measure tokens/sec on CPU, (d) submit to AI Hub for HTP timing, (e) measure accuracy metrics; 6. Produce comprehensive comparison table |
| Stack | auto-gptq, autoawq, bitsandbytes, qai_hub, transformers |
| Datasets | WikiText-2, MMLU (200 questions for speed), WinoGrande val |
| Output | Comparison table: format × (size, DRAM, latency_cpu, latency_npu, PPL, MMLU, WinoGrande); recommended configuration for 3 deployment scenarios (battery-constrained, quality-constrained, latency-constrained) |
| How to Test | pytest test_lab.py — checks: quantized models produce valid text, accuracy metrics are within expected ranges (PPL < 8 for INT4), DRAM usage is less than FP16, AI Hub job completes |
| Talking Points | "Which INT4 format would you recommend for Snapdragon deployment?" — explain: NF4 for accuracy, AWQ for speed, GPTQ for best accuracy if you can spend the quantization time; DLC format for actual deployment |
| Resume Bullet | Benchmarked 3 INT4 quantization formats for LLaMA-3.2-1B on Snapdragon 8 Gen 3; GPTQ achieved best quality (PPL 6.4 vs 5.9 FP16 baseline) at 1.8 GB DRAM and 14 tok/s on HTP |
| Extensions | Add INT2 (AQLM) for extreme compression; test LLaMA-3.2-3B with layer-by-layer offload; measure per-query energy consumption |
Lab 02 — LoRA Fine-Tuning for Accuracy Recovery
| Field | Value |
|---|---|
| Goal | After aggressive INT4 quantization (which degraded domain accuracy), use QLoRA to fine-tune the quantized model on domain data, recovering accuracy while maintaining the INT4 memory footprint |
| Concepts | LoRA rank and alpha hyperparameters, PEFT library, QLoRA (fine-tuning INT4 base + BF16 adapters), gradient checkpointing, domain dataset preparation, merge vs separate LoRA weights for deployment |
| Steps | 1. Take the INT4 GPTQ model from Lab 01 that showed the most accuracy degradation; 2. Prepare domain fine-tuning dataset (use medical QA or legal text from HuggingFace, ~10K examples); 3. Configure QLoRA: rank=64, alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"]; 4. Fine-tune for 2 epochs with AdamW + cosine LR; 5. Evaluate domain accuracy before/after LoRA; 6. Try ranks r=[8, 16, 32, 64, 128] and plot accuracy vs trainable params; 7. Merge LoRA weights into base and re-quantize for deployment |
| Stack | peft, bitsandbytes, transformers, trl |
| Datasets | MedQA (medical questions) or CourtListener (legal), WikiText-2 |
| Output | LoRA-recovered model; accuracy vs rank plot; domain accuracy before/after recovery; training curves; merged deployable model |
| How to Test | pytest test_lab.py — checks: LoRA adapter shapes are correct, merged model weights differ from base, domain accuracy improves by >1% vs unrecovered model, trainable parameter count matches expected for given rank |
| Talking Points | "When would you use LoRA rank=64 vs rank=8?" — larger rank = more capacity but more memory and slower inference; "How do you choose target modules for LoRA?" — typically attention projections; "Can LoRA recovery work at INT2?" — usually not — minimum INT4 for gradient flow |
| Resume Bullet | Applied QLoRA to recover domain accuracy in INT4-quantized LLaMA model; achieved 89% of FP16 accuracy on medical QA using only 1.2% additional trainable parameters (rank=64 LoRA) |
| Extensions | Add LoRA rank adaptation during training; implement DoRA (Weight-Decomposed Low-Rank Adaptation); merge and re-quantize with quantization-aware LoRA merge |
Lab 03 — Memory-Constrained Serving: Layer-by-Layer Offload
| Field | Value |
|---|---|
| Goal | Implement a memory-constrained inference engine for a model that doesn't fit in DRAM: stream layers from disk (simulated NVMe/eMMC), run inference one layer at a time, and measure the latency overhead vs fully-loaded inference |
| Concepts | Layer streaming, prefetching (overlap compute and load), CPU offload, mmap for zero-copy loading, sliding-window context (for infinite context with bounded memory), activation checkpointing at inference |
| Steps | 1. Save each transformer layer's weights as a separate .pt file (simulates NVMe storage); 2. Implement StreamingTransformer that loads layer $i$ → runs it → deletes activations → loads layer $i+1$; 3. Add async prefetching: start loading layer $i+1$ while layer $i$ is computing; 4. Implement sliding window for long contexts (keep only last W tokens in KV cache); 5. Benchmark: fully-loaded vs layer-streaming (no prefetch) vs layer-streaming (with prefetch); 6. Find minimum DRAM budget that still achieves acceptable latency |
| Stack | PyTorch 2.3+, asyncio, mmap, transformers |
| Datasets | WikiText-2 long documents (1000+ tokens) |
| Output | StreamingTransformer class; latency vs DRAM budget trade-off curve; prefetch effectiveness analysis; minimum viable DRAM configuration |
| How to Test | pytest test_lab.py — checks: streaming output matches full-load output, memory peak is bounded by streaming config, prefetch reduces latency vs sequential load, sliding window handles sequences longer than cache |
| Talking Points | "How would you deploy a 70B LLM on a 4 GB DRAM device?" — layer streaming + INT4 + sliding window + prefetch; "What determines the optimal prefetch depth?" — ratio of compute time to memory transfer time |
| Resume Bullet | Implemented layer-streaming inference engine enabling 7B LLM inference at 2 GB DRAM; async prefetching recovered 71% of the streaming latency overhead vs fully-loaded inference |
| Extensions | Add CPU↔GPU offload (for GPU with limited VRAM); implement KV cache compression (group quantize to INT4); add speculative prefetching based on predicted layer access pattern |
Deliverables Checklist
- INT4 quantization comparison across 3 formats with AI Hub benchmarks
- QLoRA recovery workflow with rank sensitivity analysis
- Layer-streaming inference engine with prefetch optimization
-
All
test_lab.pysuites pass
Interview Relevance
- "How would you run LLaMA-7B on a device with 4 GB DRAM?" — you describe: INT4 quantization (1.75 GB) + layer streaming + prefetch + sliding window context
- "After INT4 quantization, our chatbot accuracy on medical queries dropped 8%. What do you do?" — QLoRA fine-tuning on medical domain data, targeting attention projections, rank=64
- "What is the difference between LoRA rank and the number of fine-tuned parameters?" — trainable params = 2 × r × (sum of target module dimensions); explain the rank-capacity trade-off