Phase 06 — Qualcomm NPU & Hardware Accelerator Architecture
Difficulty: ⭐⭐⭐⭐⭐
Estimated Time: 3 weeks (70–90 hours)
Roles Supported: Directly differentiating for Qualcomm roles — this is the company-specific stack
Why This Phase Exists
Every other phase in this curriculum applies broadly to AI engineering. This phase is specifically about Qualcomm's AI stack and the knowledge that makes you indispensable for this exact role.
Qualcomm's Snapdragon silicon contains three AI accelerators: the Adreno GPU, the Qualcomm Kryo CPU, and the Hexagon DSP with the HTP (Hexagon Tensor Processor) — an NPU. The HTP achieves >100 TOPS for INT8 operations and is the target for every deployed AI model on a Snapdragon device (900 million+ devices in 2024).
The deployment pipeline is:
- PyTorch/TF model → ONNX or TFLite
- ONNX → SNPE / QNN (Qualcomm Neural Processing SDK / Qualcomm AI Engine Direct)
- QNN SDK performs quantization + graph partitioning → DLC (Deep Learning Container)
- At runtime: Hexagon Runtime executes on HTP
Understanding where models fail in this pipeline — why 3% of ops fall back to CPU, why accuracy drops 2% after HTP quantization, why throughput is 40% of theoretical — is the entire job. This phase teaches you to reason about all of these.
Concepts
- Qualcomm AI Engine stack:
SNPE(older, simpler) →QNN(newer, more flexible) →QAI AppBuilder(high-level Python API); all three are in production - HTP (Hexagon Tensor Processor): matrix accelerator inside the Hexagon DSP; operates on INT8/INT16/FP16; has VTCM (Vector Tightly Coupled Memory) = fast on-chip SRAM analogous to GPU shared memory
- VTCM: ~8MB on Snapdragon 8 Gen 3; weights that fit in VTCM avoid DRAM bandwidth bottleneck; tiling strategy must account for VTCM capacity
- QNN Execution Providers: CPU, GPU, HTP (NPU), DSP; operations automatically or manually partitioned across backends
- DLC format: Qualcomm's compiled model format — weights + compute graph + quantization params; analogous to TensorRT
.engine snpe-onnx-to-dlc: command-line tool that converts ONNX to DLC; accepts quantization encoding files- QNN SDK: lower-level than SNPE; more control over graph partitioning, custom ops, and execution context; requires C++ SDK for custom operators
- Hardware-aware graph partitioning: assign each op to the device (CPU/GPU/HTP) that executes it most efficiently; account for: (1) HTP-supported ops, (2) data transfer cost between devices, (3) memory constraints
- NPU operator constraints: HTP has strict op constraints: weight shape must be aligned (e.g., multiples of 32), certain combination of dtypes not supported, some ops (e.g., scatter, dynamic shapes) always fall back to CPU
- HTP quantization encoding: QNN uses an
.encodingsJSON file specifying per-layer scale/offset for weights and activations; generated by thesnpe-dlc-quantizeorqnn-onnx-convertertools - Hexagon SDK: C/C++ SDK for writing custom HTP ops as "HVX (Hexagon Vector eXtensions)" kernels; SIMD intrinsics for 128-byte vectors
- Qualcomm AI Hub: cloud service for testing models on real Snapdragon devices without owning hardware; free tier available; used in Labs when physical hardware is unavailable
Labs
Lab 01 — SNPE/QNN Deployment Pipeline
| Field | Value |
|---|---|
| Goal | Build a complete SNPE/QNN deployment pipeline: PyTorch → ONNX → DLC → quantized DLC → benchmark accuracy and latency on Qualcomm AI Hub |
| Concepts | snpe-onnx-to-dlc, snpe-dlc-quantize, encodings JSON, Qualcomm AI Hub Python API, accuracy delta FP32 vs INT8 on HTP |
| Steps | 1. Export MobileNetV3-Small to ONNX (opset 13); 2. Generate quantization encodings using qai_hub.quantize_model() with 512 calibration images; 3. Convert ONNX to DLC using snpe-onnx-to-dlc; 4. Apply INT8 quantization to DLC; 5. Submit profiling job to Qualcomm AI Hub targeting Snapdragon 8 Gen 3 simulator; 6. Compare accuracy: PyTorch FP32 vs DLC FP32 vs DLC INT8 on ImageNet val-100 |
| Stack | qai_hub (Qualcomm AI Hub SDK), snpe (or QAI AppBuilder), PyTorch 2.3+, ONNX |
| Datasets | ImageNet val-100 (first 100 images per class) |
| Output | .dlc artifacts; AI Hub profiling report; accuracy/latency comparison table; ops fallback analysis (which ops run on HTP vs CPU) |
| How to Test | pytest test_lab.py — checks: ONNX passes checker, DLC file size < 2x ONNX (quantization should reduce size), accuracy delta < 1%, AI Hub job completes successfully |
| Talking Points | Why does .dlc require an encodings file? What causes ops to fall back to CPU in the QNN pipeline? How do you debug a DLC that produces different results than PyTorch? |
| Resume Bullet | Built end-to-end SNPE/QNN deployment pipeline for MobileNetV3; validated <0.8% accuracy delta under INT8 quantization on Snapdragon 8 Gen 3; achieved 4.2ms inference latency |
| Extensions | Deploy LLaMA-3.2-1B to AI Hub; add custom op using QNN SDK; implement pipeline for Qualcomm Cloud AI 100 (data center NPU) |
Lab 02 — Hardware-Aware Graph Partitioning
| Field | Value |
|---|---|
| Goal | Write a graph partitioner that analyzes an ONNX graph and assigns each op to CPU, GPU, or HTP; minimize estimated total latency accounting for device execution time and data transfer cost between devices |
| Concepts | Op capability tables, data transfer latency model, networkx for graph analysis, integer linear programming (ILP) for optimal partitioning, heuristic greedy partitioner |
| Steps | 1. Build HTPCapabilityTable — dict mapping ONNX op_type to supported/unsupported + precision support; 2. Build DeviceLatencyModel — parameterized cost function for each device per op type and shape; 3. Build TransferCostModel — latency to transfer tensor between CPU↔HTP given size; 4. Implement GreedyPartitioner — assign ops to HTP if supported and no costly data transfer boundary; 5. Implement ILPPartitioner using scipy.optimize.linprog for optimal solution; 6. Compare greedy vs ILP on 5 model architectures; 7. Visualize partitioning as colored graph |
| Stack | PyTorch 2.3+, onnx, networkx, scipy, matplotlib |
| Datasets | ONNX graphs of: MobileNetV3, ResNet-50, BERT-base, LLaMA-tiny, ViT-Small |
| Output | Partition assignments for 5 models; greedy vs ILP latency comparison; colored graph visualization |
| How to Test | pytest test_lab.py — checks: partition is valid (no cycles across device boundaries), all ops assigned, ILP solution is optimal (better or equal to greedy), transfer boundaries counted correctly |
| Talking Points | Why is the partitioning problem NP-hard? When does greedy outperform ILP (hint: overhead)? How does the actual QNN SDK handle partitioning? |
| Resume Bullet | Implemented hardware-aware graph partitioner for QNN deployment; demonstrated ILP-optimal partitioning reduces estimated cross-device transfer cost by 31% vs greedy baseline on transformer models |
| Extensions | Add VTCM budget constraint (weights that fit must stay on HTP); add pipeline parallelism (CPU and HTP running simultaneously); implement dynamic profiling using AI Hub timing data |
Lab 03 — NPU Operator Mapping & Fallback Strategy
| Field | Value |
|---|---|
| Goal | For a given model (Whisper-small for speech, LLaMA-3.2-1B for text), systematically identify unsupported NPU ops, develop workaround strategies, and measure accuracy/latency impact of each workaround |
| Concepts | Op decomposition (replace unsupported op with supported primitives), op fusion (combine two ops into one supported op), fallback policies (per-layer, per-batch, always-CPU), custom HTP op stubs |
| Steps | 1. Run model through QAI Hub to get op support report; 2. Classify unsupported ops by category: (a) decomposable, (b) fusible with adjacent op, (c) must fall back to CPU; 3. Implement decomposition rewrites for top-3 unsupported ops (e.g., GeLU → polynomial approx, LayerNorm → simpler form); 4. Measure accuracy delta of each decomposition on task metric; 5. Implement fallback cost estimator; 6. Produce final recommendation report: which ops to decompose, which to fall back, expected end-to-end latency |
| Stack | qai_hub, onnx, PyTorch 2.3+, onnxoptimizer |
| Datasets | LibriSpeech test-clean (for Whisper); MMLU (for LLaMA) |
| Output | Op support report for 2 models; decomposition implementations; accuracy table before/after decomposition; final deployment recommendation |
| How to Test | pytest test_lab.py — checks: decomposed models are ONNX valid, GeLU approximation error < 1e-3, fallback cost correctly accounts for CPU↔HTP data transfer |
| Talking Points | How does GeLU polynomial approximation compare to erf-based implementation on an NPU? What are the risks of operator decomposition for accuracy? When would you reject a decomposition? |
| Resume Bullet | Systematically resolved NPU op compatibility for Whisper-small on Snapdragon 8 Gen 3; 3 operator decompositions reduced CPU fallback rate from 22% to 7% with <0.4% WER accuracy impact |
| Extensions | Write a custom HVX kernel for an unsupported op; submit custom op to QNN SDK via NHWC layout; implement a regression test comparing HTP vs CPU output for all fallback ops |
Deliverables Checklist
- End-to-end SNPE/QNN pipeline with AI Hub profiling report
- Graph partitioner (greedy + ILP) with visualization
- Op mapping analysis with decomposition strategies for 2 models
-
All
test_lab.pysuites pass
Interview Relevance
- "How would you deploy a new model to our Snapdragon NPU?" — you have the exact pipeline: PyTorch → ONNX → QNN conversion → encodings → DLC → AI Hub validation
- "Our model loses 3% accuracy after HTP quantization. What's your debugging process?" — you check: op fallbacks, per-layer quantization error, activation outliers, calibration dataset quality
- "What is VTCM and why does it matter for model performance?" — you can explain tiling strategy, DRAM bandwidth bottleneck, and the latency cliff when weights don't fit