Phase 11 — Capstone: End-to-End AI Performance Engineering
Hitchhiker's Guide
"The answer to the ultimate question of life, the universe, and everything is 42 — but in ML, the answer to latency is 'it depends on your quantization scheme.'"
What This Phase Is
You have spent ten phases absorbing the complete toolchain of a Qualcomm Senior Staff ML Engineer — from transformer architecture internals to NPU graph optimization, from McNemar's regression tests to speculative decoding. Phase 11 is where you build production-grade systems that combine everything.
Each capstone lab simulates a real deliverable that a Senior Staff Engineer at Qualcomm might own end-to-end.
The Five Capstones
Lab 01 — Accuracy Regression CI Pipeline
You own: The automated accuracy regression gate that runs on every model commit.
Real-world context: At Qualcomm, every quantized model must pass accuracy gates before shipping to device OEMs. A regression in WER (speech), BLEU (translation), or accuracy (vision/NLP) can trigger customer escalations. You build the entire pipeline: baseline management, McNemar's test, Pareto analysis, GitHub Actions integration.
Skills combined: Phase 09 (eval at scale), Phase 07 (profiling), Phase 04 (quantization accuracy).
Lab 02 — NPU Graph Optimization Pipeline
You own: The torch.compile → ONNX → QNN compilation pipeline with accuracy verification.
Real-world context: A model trained in PyTorch must be compiled for the Hexagon NPU via ONNX → SNPE/QNN path. You own the graph analysis (unsupported ops detection), optimization passes (operator fusion, constant folding), and the accuracy gate that ensures <1% degradation after compilation.
Skills combined: Phase 05 (ML compilers), Phase 06 (Qualcomm NPU), Phase 03 (quantization).
Lab 03 — Custom torch.compile Backend
You own: A minimal torch.compile backend that inserts custom NPU operator calls.
Real-world context: At Qualcomm AI Research, ML engineers write custom compiler backends to target the Hexagon DSP. You implement the backend callable interface, register custom operators, and verify correctness with numerical tolerance tests.
Skills combined: Phase 05 (ML compilers / FX graphs), Phase 02 (model architecture), Phase 08 (inference).
Lab 04 — Production Quantization Suite
You own: A unified quantization pipeline: PTQ → sensitivity analysis → mixed-precision selection → QAT → deployment.
Real-world context: A model arrives from research team. You must deliver an INT4/INT8 mixed-precision version with <2% accuracy drop. You run ADAROUND, measure per-layer sensitivity, select a mixed-precision assignment, run QAT fine-tuning, and export.
Skills combined: Phase 03 (quantization), Phase 04 (post-training quant depth), Phase 06 (hardware-aware quant), Phase 10 (AWQ).
Lab 05 — Multi-Model Eval Dashboard
You own: A benchmarking harness that evaluates multiple model variants across tasks and generates a Pareto analysis report.
Real-world context: You receive 8 model variants (different quantization levels, architectures). You run them through 5 benchmark tasks, collect accuracy + latency, compute the Pareto front, detect regressions vs baseline, and generate a JSON report that feeds the CI system.
Skills combined: Phase 09 (eval at scale), Phase 07 (profiling), Phase 08 (inference), Phase 04 (benchmarking infra).
Interview Scenarios Each Lab Prepares You For
| Lab | Likely Interview Question |
|---|---|
| 01 | "How do you ensure a quantized model hasn't regressed before shipping to OEM?" |
| 02 | "Walk me through your ONNX export debugging process for a model with unsupported ops." |
| 03 | "How would you implement a custom operator that runs on Hexagon HTP in a torch.compile pipeline?" |
| 04 | "A model has 3% accuracy drop after INT8 quantization. What's your recovery strategy?" |
| 05 | "You have 8 model variants and 3 days. How do you decide which to ship?" |
Suggested Completion Order
- Lab 01 (CI Pipeline) — builds on Phase 09, standalone
- Lab 05 (Multi-Model Dashboard) — builds on Lab 01
- Lab 02 (NPU Graph Optimization) — builds on Phase 05/06
- Lab 04 (Production Quant Suite) — builds on Phase 03/04
- Lab 03 (Custom Backend) — most advanced, builds on all phases
Time Estimate
Each lab: 3–5 hours of focused work if you've done all prior phases. Total: 15–25 hours for the complete capstone.