The Hitchhiker's Guide — Phase 06: Qualcomm NPU Hardware & Deployment

"Understanding Snapdragon is not optional for this role. It is the product." — Qualcomm Engineering

Section 1: Snapdragon SoC Architecture

The Snapdragon 8 Gen 3 (SM8650) is a heterogeneous SoC with multiple compute units:

┌─────────────────────────────────────────────────────────────┐
│                    Snapdragon 8 Gen 3                       │
│                                                             │
│  ┌──────────────────┐    ┌──────────────────────────────┐  │
│  │   CPU (Kryo)     │    │   GPU (Adreno 750)           │  │
│  │  1×Cortex-X4     │    │   Vulkan/OpenCL compute      │  │
│  │  5×Cortex-A720   │    │   ~3.8 TFLOPS FP16           │  │
│  │  2×Cortex-A520   │    └──────────────────────────────┘  │
│  └──────────────────┘                                       │
│                          ┌──────────────────────────────┐  │
│  ┌──────────────────┐    │   HTP (Hexagon Tensor Proc.) │  │
│  │   DSP (Hexagon)  │    │   ~73 TOPS INT8              │  │
│  │   HVX (512-bit   │    │   VTCM: 8 MB on-chip SRAM    │  │
│  │   vector unit)   │    │   HMX (Hexagon Matrix Ext.)  │  │
│  └──────────────────┘    └──────────────────────────────┘  │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │   DRAM (LPDDR5X @ 77 GB/s, 12 GB shared)           │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

HTP (Hexagon Tensor Processor) Deep Dive

The HTP is the AI accelerator on Snapdragon. Its key components:

HMX (Hexagon Matrix Extension): matrix multiply unit; processes INT8 8×8 matrix blocks in a single cycle; achieves ~73 TOPS INT8 throughput.

HVX (Hexagon Vector Extension): 512-bit SIMD vector unit; 64× INT8 multiplies per cycle; used for elementwise ops, activations, normalization.

VTCM (Vector Tightly Coupled Memory): 8 MB on-chip SRAM connected directly to HMX/HVX datapaths; bandwidth: ~2 TB/s (vs DRAM's 77 GB/s); critical: weights that fit in VTCM during a layer's computation avoid the 26× DRAM bandwidth penalty.

Tiling for VTCM: the QNN compiler tiles large weight matrices to fit in VTCM. For example, a 4096×4096 INT8 matrix = 16 MB — doesn't fit in 8 MB VTCM. The compiler tiles into 4 strips of 4096×1024 = 4 MB each, processes sequentially, accumulates output.

Power states: HTP has multiple power levels (turbo, nominal, SVS, SVS2). At SVS, throughput drops to ~20 TOPS but power consumption drops by 5×. The QNN runtime can switch states dynamically.

Section 2: SNPE vs QNN — When to Use Which

SNPE Conversion Pipeline

# Convert ONNX → SNPE DLC
snpe-onnx-to-dlc \
    --input_network model.onnx \
    --output_path model.dlc \
    --input_dim input "1,3,224,224"

# Quantize with calibration data
snpe-dlc-quantize \
    --input_dlc model.dlc \
    --input_list calibration_list.txt \
    --output_dlc model_quantized.dlc

# Profile on device (connected Android device)
snpe-net-run \
    --container model_quantized.dlc \
    --input_list input_list.txt \
    --perf_profile high_performance \
    --backend HTP \
    --profiling_level detailed

QNN SDK Pipeline

# Convert ONNX → QNN
# From command line:
# qnn-onnx-converter --input_network model.onnx --output_path model_qnn

# Compile for device
# qnn-model-lib-generator --model model_qnn.cpp --lib_name model_qnn

# Run with QNN runtime
# qnn-net-run --model libmodel_qnn.so --backend libQnnHtp.so --input_list inputs.txt

Section 3: Qualcomm AI Hub — Cloud-Based Testing

Qualcomm AI Hub allows submitting models for profiling on real Snapdragon devices without owning one. It is the primary tool for the labs in this course.

import qai_hub

# Upload model
model = qai_hub.upload_model("model.onnx")  # or torch model, tflite, etc.

# Compile for device
compile_job = qai_hub.submit_compile_job(
    model=model,
    device=qai_hub.Device("Snapdragon 8 Gen 3"),
    input_specs={"input": ((1, 3, 224, 224), "float32")},
)
compiled_model = compile_job.get_target_model()

# Profile
profile_job = qai_hub.submit_profile_job(
    model=compiled_model,
    device=qai_hub.Device("Snapdragon 8 Gen 3"),
)
profile_result = profile_job.download_profile()

# Inspect per-layer timing
for layer in profile_result["layers"]:
    print(f"{layer['name']:40s}  {layer['execution_time_us']:8.1f} µs  {layer['compute_unit']}")

# Inference
inference_job = qai_hub.submit_inference_job(
    model=compiled_model,
    device=qai_hub.Device("Snapdragon 8 Gen 3"),
    inputs={"input": [np.random.randn(1, 3, 224, 224).astype(np.float32)]},
)
outputs = inference_job.download_output_data()

Reading the Profile Output

The profile output from AI Hub contains:

• layer name: maps back to ONNX node name
• execution_time_us: time in microseconds on the target device
• compute_unit: "HTP" (good), "CPU" (bad — means HTP fallback), "GPU"
• data_type: INT8 (fast), FP16 (slower on HTP)

Optimization workflow:

1. Run initial profile → identify top-5 slowest layers
2. Check compute_unit for each slow layer — if "CPU", that op is falling back
3. For HTP fallbacks: check QNN op support matrix, then either (a) decompose the unsupported op, (b) replace with a supported alternative, or (c) quantize the op to INT8 (many FP32 ops unsupported on HTP but INT8 variants are)
4. For slow HTP layers: check data type — FP16 is 4× slower than INT8 on HMX

Section 4: Op Mapping — PyTorch to HTP

Key insight: dynamic-shape ops (NonZero, Unique, scatter with dynamic indices) are almost always CPU-only. Design models to avoid them in the forward pass.

Section 5: Hardware-Aware Quantization for HTP

HTP achieves peak throughput with INT8 weights AND INT8 activations. Mixed-precision strategies:

Per-channel quantization (preferred for weights):

• Each output channel has its own scale/zero_point
• Better accuracy than per-tensor for weights
• Supported: INT8 per-channel for all linear/conv ops

Per-tensor quantization (for activations):

• Single scale/zero_point for the entire tensor
• Required for activations (per-channel activation quant not standard)

INT16 activations (Qualcomm extension):

• Some ops support INT16 activations with INT8 weights
• Better accuracy for sensitive layers (attention logits)
• 2× memory cost for activations vs INT8

Quantization-aware calibration on Qualcomm:

import qai_hub
import numpy as np

# Use 128–512 calibration samples (representative of deployment distribution)
calibration_data = {
    "input": [sample.numpy() for sample in calibration_dataset[:128]]
}

compile_job = qai_hub.submit_compile_job(
    model=model_onnx,
    device=qai_hub.Device("Snapdragon 8 Gen 3"),
    input_specs={"input": ((1, 3, 224, 224), "float32")},
    options="--quantize_full_type int8 --quantize_io",  # INT8 weights and activations
    calibration_data=calibration_data,
)

Section 6: Power and Thermal Constraints

On mobile, thermal and power matter as much as performance:

• TDP: Snapdragon 8 Gen 3 sustained power ≈ 5W; burst ≈ 15W
• Thermal throttling: if SoC temperature > 85°C, clock speeds drop automatically
• Performance hints: qai_hub.Device options include performance_profile (burst, sustained, low_power, power_saver)
• Battery vs latency: at low_power mode, throughput ≈ 25% of burst, but battery drain ≈ 10% of burst

Engineering trade-off: at Qualcomm, the optimization target is often ops/joule (efficiency), not ops/second (throughput). A model that runs at 80% of peak TOPS but uses 40% of peak power may be preferred.

Section 7: Interview Pitfalls

Section 8: Resources

1. Qualcomm AI Hub documentation — https://aihub.qualcomm.com/docs
2. QNN SDK — https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/
3. Qualcomm Innovation Center — AI Models — https://github.com/quic/ai-hub-models (100+ models with AI Hub integration)
4. "Dissecting the Qualcomm Snapdragon NPU" — AnandTech deep-dive
5. SNPE documentation — https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/
6. Hexagon DSP SDK — https://developer.qualcomm.com/software/hexagon-dsp-sdk
7. qai-hub-models GitHub — https://github.com/quic/ai-hub-models — production examples of PyTorch → HTP deployments