Phase 11 — Capstone Projects

Difficulty: ⭐⭐⭐⭐⭐
Estimated Time: 4 weeks (80–120 hours)
Roles Supported: Every capstone is a portfolio-ready demonstration of Senior Staff competency; select 2–3 based on target role

Why This Phase Exists

Interviews for Senior Staff engineers have a recurring pattern: "Tell me about the most technically complex system you've built." The capstone projects exist to give you a truthful, impressive, technically deep answer. Each capstone integrates skills from multiple phases into a coherent system.

More importantly: the capstone projects are designed to be continued. After your first job, these are systems you can extend, benchmark against new hardware, and publish. They become long-term portfolio projects, not throw-away exercises.

Capstone Selection Guide

Capstone 01 — Production Accuracy Regression CI System

Overview

A production-ready GitHub Actions workflow that:

1. Runs eval harness (MMLU, HellaSwag, PPL) on a PR's model checkpoint
2. Compares to the main-branch baseline using McNemar's test
3. Fails the PR if any task drops >0.5% (customizable threshold) with p<0.05
4. Posts a detailed report as a PR comment with per-task accuracy, confidence intervals, and Pareto position

Architecture

PR triggered →
  GitHub Actions: checkout model checkpoint
  → run eval harness (200 examples per task, ~5 min)
  → load baseline from artifact store
  → apply McNemar's test per task
  → post Markdown report as PR comment
  → PASS/FAIL based on policy

What to Build

• eval/eval_harness.py — from Phase 09 Lab 01
• eval/regression_check.py — McNemar's test + policy engine
• eval/report_generator.py — Markdown report with tables and badges
• .github/workflows/accuracy-ci.yml — full CI pipeline
• eval/baselines/ — artifact store for baseline results
• tests/test_regression_check.py — property-based tests for regression logic

Deliverables

• Functional GitHub Actions CI that blocks PRs with accuracy regressions
• Demo: commit a model with injected 2% MMLU regression → CI blocks it; report shows which examples changed
• README with architecture diagram, configuration guide, and threshold selection rationale

Resume Bullet

Built production accuracy regression CI system used to guard model quality across 50+ optimization PRs; system blocked 7 regressions in 3 months, preventing customer-facing quality degradation

Capstone 02 — Full NPU Inference Pipeline

Overview

End-to-end pipeline from PyTorch training checkpoint to profiled, optimized inference on Qualcomm HTP:

1. Export to ONNX with full opset validation
2. Optimize ONNX graph (fold constants, eliminate redundancies)
3. Convert to Qualcomm AI Hub DLC format
4. Profile HTP execution, identify bottleneck layers
5. Quantize to INT8/INT4, measure accuracy delta, iterate

Architecture

PyTorch FP16 checkpoint
→ torch.export (FX-based)
→ ONNX opset 18
→ onnxoptimizer + onnxsim
→ Qualcomm AI Hub: compile for Snapdragon 8 Gen 3
→ HTP execution + layer-level profiling
→ Identify top-3 bottleneck ops
→ INT8 quantization → accuracy delta
→ INT4 quantization → accuracy delta
→ Pareto: accuracy vs latency per config

What to Build

• pipeline/export.py — PyTorch → ONNX with shape validation
• pipeline/optimize.py — ONNX graph optimization passes
• pipeline/qai_submit.py — AI Hub submission + result polling
• pipeline/profile_analyzer.py — parse HTP profile, identify bottlenecks
• pipeline/quantize.py — INT8/INT4 pipeline with calibration
• pipeline/accuracy_delta.py — accuracy delta measurement at each stage
• pipeline/run_pipeline.sh — single-command E2E run

Models to Run

• MobileNetV3 (classification baseline)
• Whisper-tiny (audio, tests decoder with cross-attention)
• LLaMA-3.2-1B (LLM, tests token generation pipeline)

Deliverables

• Single-command pipeline that produces accuracy + latency + memory report for any HuggingFace model
• HTP profiling report showing per-layer timing for LLaMA-3.2-1B decode step
• Pareto frontier plot: accuracy vs latency for 5 quantization configs per model
• README with deployment instructions for each model

Resume Bullet

Built end-to-end NPU inference pipeline (PyTorch → ONNX → QNN → HTP profiling) for 3 model families; identified and resolved 2 HTP-incompatible ops; LLaMA-3.2-1B achieved 14.2 tok/s at INT4 with 97.3% of FP16 MMLU accuracy

Capstone 03 — Custom torch.compile Backend for Mock NPU ISA

Overview

Implement a custom torch.compile backend that lowers FX IR to a fictional but realistic NPU instruction set (inspired by Qualcomm's QNN op set). This demonstrates deep compiler knowledge — the most differentiating skill for this role.

Mock NPU ISA

Define 15 instructions:

• MATMUL, CONV2D, DEPTHWISE_CONV2D — compute ops
• LOAD_TILE, STORE_TILE — memory ops (DRAM ↔ VTCM)
• QUANTIZE, DEQUANTIZE — precision ops
• RELU, GELU, SOFTMAX, LAYERNORM — activation/norm ops
• ELEMENTWISE_ADD, ELEMENTWISE_MUL, BROADCAST — elementwise
• CONCAT, RESHAPE — shape ops

What to Build

• npu_backend/isa.py — instruction set dataclasses + validator
• npu_backend/lowering.py — FX node → ISA instruction mapper
• npu_backend/fusion.py — graph optimization: fuse MATMUL + RELU, LAYERNORM + DROPOUT, etc.
• npu_backend/codegen.py — ISA instruction list → JSON "NPU program" + pretty-printer
• npu_backend/simulator.py — simulate execution: count cycles, memory traffic, VTCM pressure
• npu_backend/register_backend.py — register as torch.compile(backend="mock_npu")
• tests/ — test every lowering rule + fusion pass

What it Produces

model = nn.Sequential(nn.Linear(512, 512), nn.ReLU())
compiled = torch.compile(model, backend="mock_npu")
output = compiled(x)  # runs PyTorch for correctness

# Export NPU program:
program = get_npu_program(model, x)
print(program)
# NPU_PROGRAM:
#   LOAD_TILE weight[512,512] → VTCM
#   MATMUL in[B,512] × VTCM[512,512] → out[B,512]
#   RELU out[B,512] → out[B,512]  (fused with MATMUL)
#   STORE_TILE out[B,512] → DRAM
# Estimated cycles: 2,359,296
# VTCM peak: 1.0 MB / 8 MB capacity

Deliverables

• Fully functional torch.compile backend that handles MLP and Conv models
• Fusion pass that correctly fuses MATMUL+RELU, MATMUL+GELU, LAYERNORM+ADD
• Cycle estimator with validation against a hand-calculated reference
• Blog-post-quality README explaining the architecture

Resume Bullet

Implemented custom torch.compile backend targeting mock NPU ISA; registered 15 lowering rules, built 6 fusion passes; backend correctly compiles and simulates MLP/Conv/Transformer ops with cycle estimation within 5% of hand-calculated reference

Capstone 04 — Production Quantization Comparison Suite

Overview

Comprehensive comparison of all major post-training quantization methods on 3 model families, with automated accuracy recovery, and a final recommendation report — the kind of analysis a Senior Staff engineer would deliver to an architecture review board.

Methods Compared

Models

• ResNet-50 (ImageNet top-1 accuracy)
• BERT-base (GLUE benchmark)
• LLaMA-3.2-1B (MMLU, PPL)

Deliverables

• Automated pipeline that produces full comparison table for any model
• Accuracy vs latency Pareto plot for all methods × models
• Recommendation engine: input (model_type, accuracy_floor, latency_target) → output (best method + config)
• Technical report (10-page PDF generated from Markdown): methodology, results, recommendations

Resume Bullet

Conducted systematic evaluation of 6 quantization methods across 3 model families; published comparison showing AWQ outperforms GPTQ by 0.9% MMLU at identical bit-width for LLM workloads; results used to standardize quantization methodology for production deployment

Capstone 05 — Multi-Model Eval + Pareto Dashboard

Overview

A web-based dashboard for tracking model accuracy and performance across multiple checkpoints over time: evaluates models, stores results in SQLite, displays interactive Pareto frontiers, and sends alerts when accuracy regresses.

What to Build

• server/eval_worker.py — async eval worker (SQLite-backed job queue)
• server/api.py — FastAPI REST API: submit eval job, query results, get Pareto
• frontend/dashboard.html — interactive Plotly.js dashboard (no build system, single file)
• server/alert_engine.py — detect regressions, send GitHub notifications
• server/db.py — SQLite schema: models, evals, tasks, metrics

Demo Scenario

Submit 10 model checkpoints (from different quantization steps of LLaMA-3.2-1B) → dashboard shows accuracy degrading as quantization gets more aggressive → Pareto frontier identifies the INT8 SmoothQuant configuration as the optimal trade-off → CI webhook fires alert when INT4 variant drops below accuracy floor.

Resume Bullet

Built multi-model eval tracking dashboard (FastAPI + SQLite + Plotly.js); tracks 10+ metrics across model checkpoints; powers quantization decisions for 3 product teams; identified optimal deployment config saving 40% DRAM while staying within accuracy requirements

Recommended Completion Order

1. Capstone 01 first — gets CI/CD infrastructure in place for all other projects
2. Capstone 04 — builds on phases 02-03, most directly reuses lab code
3. Capstone 02 — requires QAI Hub access, do after account setup
4. Capstone 03 — hardest; do this when you're comfortable with FX and torch.compile
5. Capstone 05 — integrates everything; do last

Portfolio Presentation

For each completed capstone:

1. GitHub repo with clean README, architecture diagram, and demo GIF
2. One-paragraph LinkedIn post linking to the repo
3. One 5-minute demo video (Loom) showing the tool in action
4. One conference submission (NeurIPS Demo, MLSys, OSDI) if the work is strong

The capstones are designed to be real, useful tools — not exercises. After you've built them, keep using them. The best way to deepen your understanding is to encounter real problems with the tools you built.