Warmup Guide — Qualcomm NPU Hardware & Deployment

Zero-to-expert primer for Phase 06. Builds the NPU mental model from "why accelerators exist" through Hexagon/HTP architecture, the SNPE/QNN toolchains, AI Hub, hardware-aware quantization, and on-device profiling.

Table of Contents

• Chapter 1: Why NPUs Exist — The Energy Argument
• Chapter 2: Anatomy of an NPU
• Chapter 3: The Snapdragon Compute Complex
• Chapter 4: The Toolchain — SNPE, QNN, and AI Hub
• Chapter 5: Graph Compilation for NPUs
• Chapter 6: Hardware-Aware Quantization
• Chapter 7: On-Device Profiling — What the Numbers Mean
• Chapter 8: The Fallback Problem
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why NPUs Exist — The Energy Argument

Zero background: a phone has a thermal budget of ~3–5 W sustained. Energy per operation, roughly (45nm-era figures from Horowitz, still directionally right):

Two conclusions define the entire field:

1. DRAM access costs ~100× the arithmetic. Accelerator design is therefore memory choreography — keep data on-chip, reuse it maximally. Compute is nearly free.
2. INT8 is ~10–30× cheaper than FP32 in energy and silicon area. That is why Phase 03 (quantization) is the entry fee for NPU deployment, not an optimization.

A CPU spends most of its silicon on flexibility (caches, branch prediction, out-of-order machinery). An NPU spends it on exactly one thing: dense low-precision multiply- accumulate with software-managed data movement. Same transistor budget, ~10–100× the inferences per joule — for the workloads it fits.

Chapter 2: Anatomy of an NPU

The recurring blueprint (Qualcomm HTP, Apple ANE, Google Edge TPU all rhyme):

• MAC array: a grid of multiply-accumulate units (systolic array or vector-tensor hybrid). Peak TOPS = MACs × 2 × clock. Peak is achieved only when the array is fed — utilization is the real metric.
• Tightly-coupled SRAM (TCM), single-digit MB: software-managed, not a cache. The compiler must explicitly stage tiles in and out (Phase 05's tiling chapter is literal here). If a layer's working set exceeds TCM, it gets split — or spills to DRAM and performance falls off the energy cliff above.
• Vector unit (HVX on Hexagon): wide SIMD for the ops that aren't matmuls — elementwise, normalization reductions, requantization scaling.
• Scalar core + DMA engines: orchestration and asynchronous DRAM↔TCM transfers, double-buffered so compute overlaps movement.

Dataflow taxonomy (worth knowing the words): weight-stationary (weights pinned in the array, activations stream — good for conv), output-stationary (accumulators pinned), row-stationary (Eyeriss's hybrid). The choice determines which reuse pattern is free and which costs bandwidth — and explains per-op efficiency differences you'll see in profiling.

Chapter 3: The Snapdragon Compute Complex

A Snapdragon SoC offers four compute targets, and placement is a first-class decision:

The Hexagon HTP is the "fused AI accelerator": scalar threads + HVX vector + tensor accelerator sharing TCM. Generations matter operationally — INT4 weight support, FP16 paths, and per-op coverage vary by SoC generation, which is why runtime capability query + per-device validation is part of any serious deployment pipeline (and why AI Hub's device farm exists).

Chapter 4: The Toolchain — SNPE, QNN, and AI Hub

Three layers, frequently confused:

• QNN (Qualcomm AI Engine Direct): the low-level per-backend API — a graph API with op packages per target (HTP, GPU, CPU). Vendor-IR altitude: you hand it a graph, it compiles a context binary for a specific SoC.
• SNPE: the older/higher-level SDK — converts ONNX/TFLite into DLC (Deep Learning Container) files, with runtime selection (CPU/GPU/DSP-HTP) at load. Still everywhere in production; conceptually a vendor compiler whose input is ONNX (Phase 05's ladder, made concrete).
• Qualcomm AI Hub (what Lab 01 uses): cloud service — submit a PyTorch/ONNX model, it compiles for a chosen device (via QNN/TFLite/ONNX-RT paths), runs it on real hardware in a device farm, and returns artifacts + per-layer profiles. The fastest legitimate way to get real-device numbers without a bench full of phones.

The pipeline shape (memorize; it generalizes to every NPU vendor): train (PyTorch) → export (ONNX) → quantize (calibration data!) → compile (per-SoC binary) → validate accuracy on-device → profile → iterate. Every arrow is a place accuracy or performance silently changes; the capstone builds gates at each arrow.

Chapter 5: Graph Compilation for NPUs

What the NPU compiler does with your graph (all Phase 04/05 concepts, specialized):

1. Partitioning: which ops run on HTP vs fall back (Chapter 8).
2. Layout assignment: HTP consumes blocked/channel-aligned layouts; transposes appear at partition boundaries. Channel counts that aren't multiples of the vector width get padded — a 3-channel input conv wastes lanes; a 36-channel depthwise layer may run at half the efficiency of a 32-channel one. Architecture choices echo in silicon.
3. Fusion: conv+requant+activation chains fused to avoid TCM round-trips; the requantization step (INT32 accumulator → INT8 output via scale multiply) is fused into the MAC epilogue.
4. Tiling & memory planning: split every op so working sets fit TCM; plan double-buffered DMA. Failures here appear as "this one layer is 10× slower" — usually a spill.
5. Scheduling: order islands to overlap DMA with compute.

You don't write these passes — but you read their output (profiles, op coverage reports) and you change the model to make them succeed. That model-side empathy is the differentiating skill this phase trains.

Chapter 6: Hardware-Aware Quantization

Phase 03 taught quantization math; the hardware adds constraints (Lab 02's content):

• Symmetric weights, asymmetric activations (typical HTP-friendly config) — zero-point multiplies in the inner loop are silicon cost; weight zero-points are usually forced to 0.
• Power-of-two vs arbitrary scales: some pipelines prefer shift-friendly scales for the requantization step; arbitrary scales need a fixed-point multiply.
• Per-channel weights are supported; per-channel activations are not — activation outliers must be solved at the model level (SmoothQuant/AWQ, Phase 03 Ch. 9), not by finer activation granularity.
• INT16 activations (where supported) as the accuracy escape hatch for sensitive layers (softmax inputs, attention scores, SSM states) at ~2× cost — mixed-precision within the graph is a per-op placement problem.
• BN folding and cross-layer equalization (CLE) before calibration — AIMET-style pre-processing that equalizes per-channel ranges across consecutive layers, often worth several accuracy points on mobile CNNs before any fancier method.
• The golden rule: quantize with the same convention the hardware executes (rounding mode, accumulator width, requant order). A simulator that rounds differently than silicon produces accuracy numbers that lie — this is why on-device validation (Lab 01/03) is non-negotiable.

Chapter 7: On-Device Profiling — What the Numbers Mean

The profile (AI Hub returns these per layer) and how to read it (Lab 03):

• End-to-end latency ≠ Σ layer times: includes runtime dispatch, DMA, and inter-partition transfers. A big gap = orchestration overhead, usually fallbacks.
• First-inference vs steady-state: graph init, weight loading, and warmup (clock ramp) dominate call #1; report steady-state median + p99, never single runs.
• Per-layer time: rank, then compare against a roofline estimate (Phase 07): a layer at 5% of peak is mis-tiled, spilling, padded, or on the wrong unit. The top-5 layers usually hold 80% of the opportunity.
• Sustained vs burst: phones throttle. A 5 ms inference at burst clock is a 9 ms inference after two minutes of thermal soak — measure both regimes; products live in sustained.
• Memory traffic counters (where exposed): the DRAM-bytes-per-inference number converts directly to energy (Chapter 1) — often a better optimization target than latency itself.

Chapter 8: The Fallback Problem

When the NPU can't run an op (unsupported type, dynamic shape, exotic op), the runtime falls back to GPU/CPU. The cost is not the op itself — it's the boundary: flush NPU pipeline, transfer activations (possibly with layout conversion + dequantize), run elsewhere, transfer back, requantize. One fallback in the middle of a graph can cut partition sizes such that end-to-end latency doubles, while every individual layer "looks fine".

Fallback triage (the single most common real-world NPU performance bug):

1. Get the partition report (which ops, which runtime, how many islands).
2. If islands > ~2–3, find the breakers: usually shapes (dynamic dims, rank >4), exotic ops (erf-GELU, complex slicing), or dtype transitions.
3. Fix at the model level: replace the op (tanh-GELU), make shapes static, move the offending op to the graph edge (pre/post-processing), or decompose it into supported primitives (Phase 04 Lab 02's judgment call).

Lab Walkthrough Guidance

Order: Lab 01 (AI Hub deployment) → Lab 02 (hardware-aware quantization) → Lab 03 (on-device profiling) — get a model on real silicon first, then make it accurate, then make it fast.

• Lab 01: take the smallest model that's interesting (MobileNet-class or a tiny transformer); run the full pipeline; keep every artifact (ONNX, compiled binary, accuracy numbers at each stage) — the pipeline discipline is the deliverable. No AI Hub account? The lab's mock path exercises identical structure.
• Lab 02: start from your Phase 03 PTQ pipeline; add the hardware constraints (symmetric weights, per-tensor activations); measure the accuracy delta each constraint costs; then add CLE/bias-correction and measure the recovery.
• Lab 03: profile before reading the code, rank layers, predict the cause for the top 3 (roofline + Chapter 5 reasoning), then check. Train the prediction muscle, not the lookup muscle.

Success Criteria

You are ready for Phase 07 when you can, from memory:

1. Reproduce the energy table's two conclusions (DRAM ~100× arithmetic; INT8 ~10–30× FP32) and derive accelerator design from them.
2. Sketch the NPU anatomy (MAC array, TCM, vector unit, DMA) and explain why TCM is software-managed.
3. Name the four Snapdragon compute targets and a placement rule for each.
4. Walk the deployment pipeline naming what can silently break at each arrow.
5. List four hardware quantization constraints that don't exist in Phase 03's pure math.
6. Run the fallback triage procedure on a partition report from memory.

Interview Q&A

Q: Model hits 30% of advertised TOPS. Diagnose. TOPS assumes the MAC array is saturated with INT8 at burst clock. Check, in order: (1) fallbacks fragmenting the graph (partition report); (2) bandwidth-bound layers — roofline says they can't reach compute peak (decode-phase LLMs live here); (3) tiling/spills — per-layer profile shows the outlier; (4) layout/padding waste from awkward channel counts; (5) thermal state — sustained vs burst. Most real cases are (1) or (2); say that.

Q: INT8 model is accurate in simulation, wrong on device. Why? Convention mismatch between simulator and silicon: rounding mode (round-half-even vs half-up), accumulator saturation behavior, requantization order (scale-then-add vs add-then-scale), or a fused activation clamping at different points. Validate layer-by- layer on device against the simulator to find the first diverging layer, then read that op's hardware spec. This is why the pipeline keeps per-layer dump hooks.

Q: Which layers would you keep at higher precision in a transformer on an NPU, and why? Softmax inputs/outputs and attention score paths (small dynamic range errors change rankings), LayerNorm/RMSNorm statistics, the router in MoE, and any recurrent state (Mamba). These are cheap (small tensors) but sensitivity-critical — classic mixed- precision placement: spend bits where error propagates, not where tensors are big.

Q: Why does Qualcomm care that you understand model architecture (Phase 02) for an NPU role? Because the cheapest fixes are model-side: replacing erf-GELU with tanh-GELU removes a fallback; choosing GQA shrinks the KV cache that DMA must stream; padding-friendly channel counts recover MAC utilization; static shapes unlock compilation. The NPU engineer who can negotiate with the model ships; the one who only files compiler bugs waits.

References

• Horowitz, Computing's Energy Problem (and what we can do about it) — ISSCC 2014 keynote (the energy table)
• Sze et al., Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017) — arXiv:1703.09039
• Chen et al., Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow (ISCA 2016) — dataflow taxonomy
• Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit (ISCA 2017) — the systolic-array reference
• Qualcomm AI Hub documentation
• Qualcomm AI Engine Direct (QNN) docs
• AIMET — AI Model Efficiency Toolkit — CLE, bias correction, AdaRound
• Nagel et al., Data-Free Quantization Through Weight Equalization and Bias Correction (2019) — arXiv:1906.04721