« All Roles

Model Accuracy & AI Performance — Senior Staff Engineer Curriculum

Target Roles:

• Qualcomm — Senior Staff Machine Learning Engineer, AI Frameworks & Performance
• Apple — Senior Staff Engineer, Core ML Accuracy & Inference
• NVIDIA — Principal Engineer, Model Optimization & Accuracy
• Google DeepMind — Senior Staff Engineer, ML Systems Performance
• Meta — Staff Engineer, AI Infrastructure & Model Quality
• Microsoft — Principal Engineer, AI Model Optimization
• Arm — Senior Staff ML Compiler & Accuracy Engineer
• AMD — Principal Engineer, ROCm Model Optimization
• Cerebras / Groq / Tenstorrent — Staff ML Systems & Accuracy Engineer

Duration: 24 weeks core (6 months) — extendable to 12 months with advanced research projects
Seniority Target: Senior Staff / Principal IC (L6–L7 equivalent)
Goal: Equip you to lead end-to-end model accuracy and AI performance work on NPU/GPU/CPU hardware — from ML framework internals through hardware-aware graph optimization, quantization with accuracy recovery, profiling, eval systems, and production deployment on edge silicon.

Why This Curriculum Exists

The "Model Accuracy & AI Performance" discipline sits at the hardest intersection in AI engineering: you must simultaneously understand model internals deeply enough to reason about why accuracy changes after optimization, and hardware/compiler internals deeply enough to know where performance is left on the table. Most engineers are strong in one dimension; Senior Staff engineers are expected to own both.

The Qualcomm JD (and every equivalent role at other silicon companies) makes this explicit: they want someone who can take a research-grade PyTorch model, understand its architecture from literature, apply quantization and graph transformations, run it through a compiler toolchain targeting an NPU, profile the result, recover lost accuracy, and repeat — all while influencing the ML framework design itself. This requires 12+ years of compounding depth.

This curriculum builds that depth from first principles. Every phase ends with quantified artifacts you can demo in interviews and publish in your GitHub portfolio. The final capstone projects are designed to be genuine open-source contributions, not toy exercises.

Reference Job Targets

What You Will Build

By the end of this curriculum you will have shipped and can demo:

1. Custom autograd engine — implement reverse-mode autodiff from scratch in Python+C++, matching PyTorch semantics on standard test cases within 1e-8 numerical tolerance
2. Custom C++ PyTorch operator — registered via TORCH_LIBRARY, with CPU+CUDA kernels, torch.compile-compatible, passing all OpCheck tests
3. Transformer with full architectural variants — RoPE, ALiBi, GQA, MoE routing, all swappable — benchmarked on Wikitext-103
4. Full PTQ → QAT → GPTQ → AWQ pipeline — automated, with accuracy-latency Pareto curves and regression gating
5. torch.compile custom backend — FX graph lowering to a mock NPU ISA with >95% op coverage on LLaMA-style models
6. TVM compilation pipeline — Relay → TIR → ARM/x86 compiled module with AutoTVM tuning, 2x+ speedup over baseline
7. SNPE/QNN model deployment pipeline — ONNX → DLC → quantized NPU inference with full accuracy benchmarking vs FP32 baseline
8. Roofline analysis toolkit — measure model arithmetic intensity, plot compute/memory ceilings, automatically identify bottleneck ops
9. FlashAttention v2 from scratch in Triton — tiled SRAM-efficient implementation matching official FA2 output within 1e-5
10. Speculative decoding engine — draft + verifier with acceptance rate profiling, 2–4x throughput improvement demonstrated
11. Eval harness from scratch — MMLU, HellaSwag, GSM8K, perplexity; pluggable task registry, 3 sampling strategies, statistical significance reporting
12. Accuracy regression CI — PR-gate pipeline that fails builds if accuracy drops >0.5% on key tasks after any optimization change
13. On-device LLM end-to-end — LLaMA-3.2-1B → INT4 quantized → edge-benchmarked with accuracy delta vs FP16 documented
14. Multi-model evaluation dashboard — Pareto frontier visualization across quantization levels, latency budgets, and accuracy metrics

Folder Structure

model-accuracy-ai-performance/
├── README.md                                          ← this file
├── phase-01-pytorch-tf-framework-internals/
│   ├── README.md
│   ├── WARMUP.md                                      ← zero-to-expert primer
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-custom-autograd-engine/
│   ├── lab-02-custom-cpp-operator/
│   └── lab-03-tf-function-tracing/
├── phase-02-model-architecture-mastery/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── DEEP-DIVE-MOE-SSM.md                           ← MoE + Mamba from first principles
│   ├── lab-01-transformer-architectural-variants/
│   ├── lab-02-mixture-of-experts/
│   └── lab-03-state-space-models/
├── phase-03-quantization-model-compression/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── DEEP-DIVE-QAT-GPTQ.md                          ← PTQ/QAT/GPTQ from first principles
│   ├── lab-01-ptq-pipeline/
│   ├── lab-02-qat/
│   └── lab-03-gptq/
├── phase-04-torch-compile-fx-graph/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-fx-graph-passes/
│   ├── lab-02-custom-compile-backend/
│   └── lab-03-triton-fused-kernel/
├── phase-05-ml-compilers-ir/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-onnx-export-optimization/
│   ├── lab-02-tvm-compilation/
│   └── lab-03-mlir-python-bindings/
├── phase-06-qualcomm-npu-hardware/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-ai-hub-deployment/
│   ├── lab-02-hardware-aware-quantization/
│   └── lab-03-on-device-profiling/
├── phase-07-profiling-performance-engineering/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-roofline-analyzer/
│   ├── lab-02-profiling-context/
│   └── lab-03-transformer-perf-bugs/
├── phase-08-inference-optimization-depth/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-flash-attention/
│   ├── lab-02-speculative-decoding/
│   └── lab-03-continuous-batching/
├── phase-09-model-accuracy-evaluation/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── DEEP-DIVE-EVALUATION-AT-SCALE.md               ← Pareto, McNemar, CI gates
│   ├── lab-01-eval-harness/
│   ├── lab-02-pareto-analyzer/
│   └── lab-03-regression-ci/
├── phase-10-genai-on-edge/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-awq-quantization/
│   ├── lab-02-qlora-adapter/
│   └── lab-03-layer-streaming/
├── phase-11-capstone/
│   ├── README.md
│   ├── HITCHHIKERS-GUIDE.md
│   ├── lab-01-regression-ci-pipeline/
│   ├── lab-02-npu-graph-optimization/
│   ├── lab-03-custom-compile-backend/
│   ├── lab-04-production-quant-suite/
│   └── lab-05-multi-model-dashboard/
├── interview-prep/
│   ├── README.md
│   ├── 01-concepts-cheatsheet.md
│   ├── 02-framework-internals-coding.py
│   ├── 03-optimization-coding.py
│   ├── 04-systems-design-questions.md
│   ├── 05-research-engineering-questions.md
│   └── 06-behavioral-star-stories.md
└── system-design/
    ├── README.md
    ├── 01-model-accuracy-regression-platform.md
    ├── 02-npu-graph-optimization-pipeline.md
    ├── 03-quantization-accuracy-recovery-system.md
    ├── 04-distributed-eval-platform.md
    └── 05-model-perf-benchmarking-infra.md

24-Week Schedule

Each Lab Structure

Prerequisites

• Python 3.10+ with pip
• PyTorch 2.3+ (pip install torch torchvision)
• CUDA 12.x (or MPS on Apple Silicon; labs work on CPU with reduced scale)
• CMake 3.20+ and g++ / clang for C++ labs
• Triton 2.3+ (comes with PyTorch nightly for some labs)
• Basic familiarity with: calculus (chain rule), linear algebra, Python, C++ basics
• Git — all labs tracked in version control
• Optional: Qualcomm AI Hub account (free) for Phase 06 simulator runs

Hardware Recommendations