Phase 02 — Model Architecture Mastery Across Modalities

Difficulty: ⭐⭐⭐⭐☆
Estimated Time: 3 weeks (60–80 hours)
Roles Supported: All Senior Staff ML roles — architecture expertise underpins every optimization decision

Why This Phase Exists

You cannot optimize what you don't understand. Before a Senior Staff engineer at Qualcomm can ask "why does this model's accuracy degrade more under INT4 quantization than that one?", they need to know: GQA uses fewer KV heads → fewer unique weight rows → friendlier to per-group quantization. MoE models have sparse activation → different quantization strategies per expert. SSMs (Mamba) have recurrent structure → completely different memory access patterns than attention.

The JD requires "expertise in model architecture, backed by literature rationale and intuition" and "understanding of broad-spectrum model architectures across modalities." This phase builds that expertise through implementation — you will implement every major architectural variant that appears in production models (LLaMA, Mistral, Mixtral, ViT, DINO, LLaVA, Mamba) from scratch or near-scratch.

After this phase, when asked in an interview "how would you adapt your NPU tiling strategy for a Mixture-of-Experts model?", you can answer specifically because you understand the routing mechanism and its sparsity implications.

Concepts

• Scaled Dot-Product Attention (SDPA): $\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$ — computational complexity $O(n^2 d)$, memory $O(n^2)$
• Multi-Head Attention (MHA): split $d_{model}$ into $h$ heads, concatenate outputs — fully parallelizable across heads
• Grouped Query Attention (GQA): $g$ query groups share one KV head pair — reduces KV cache size by factor $h/g$; used in LLaMA-3, Mistral
• Multi-Query Attention (MQA): extreme GQA with $g=1$ — single KV head, all query heads; MQA = GQA with groups=1
• Rotary Position Embeddings (RoPE): encode position via rotation of Q,K vectors in complex plane: $f_q(x_m) = R^m_d x_m$ — length extrapolation via RoPE scaling (LLaMA)
• ALiBi (Attention with Linear Biases): position bias added to attention logits as $-m \cdot |i-j|$ — train short, generalize long
• SwiGLU / GeGLU: gated feedforward: $\text{FFN}(x) = (\sigma(W_1 x) \odot W_2 x) W_3$ — used in PaLM, LLaMA; ~25% more params but better performance per FLOP
• Pre-norm vs Post-norm: pre-norm (RMSNorm before attention) = more stable training; used in all modern LLMs
• RMSNorm: $\hat{x} = \frac{x}{\text{RMS}(x)} \cdot \gamma$ where $\text{RMS}(x) = \sqrt{\frac{1}{n}\sum x_i^2}$ — no mean subtraction, faster than LayerNorm
• Mixture of Experts (MoE): sparse activation — each token routes to top-K of E experts; active params per token = dense_params + (K/E × expert_params)
• Vision Transformer (ViT): divide image into non-overlapping patches → linear projection → standard transformer; patch size controls trade-off between sequence length and feature resolution
• DINO / DINOv2: self-supervised ViT — student-teacher distillation with no labels; produces strong dense features for segmentation/detection
• CLIP: contrastive vision-language pretraining — align image and text embeddings via InfoNCE loss; zero-shot classification and image-text retrieval
• LLaVA-style multimodal: visual tokens from CLIP vision encoder → linear projection → prepend to LLM token sequence → standard LLM forward
• Mamba / SSMs (Structured State Space Models): recurrent models with selective state transitions; linear time in sequence length; competitive with transformers on long sequences

Labs

Lab 01 — Transformer with Full Architectural Variants

Lab 02 — Mixture of Experts (MoE)

→ Lab folder: lab-02-mixture-of-experts/

Lab 03 — State Space Models (S4 / Mamba)

→ Lab folder: lab-03-state-space-models/

Lab 04 — ViT + DINO Self-Distillation

Implemented at testable scale in lab-04-vit-dino/ — ViT from scratch, DINO loss with centering+sharpening, momentum teacher, multi-crop step, 13-test suite that runs on synthetic data in seconds. The full CIFAR-10 training run described below is the lab's extension target.

Lab 05 — Multimodal Bridge (CLIP → LLM Projection)

Implemented at testable scale in lab-05-multimodal-bridge/ — trainable MLP projector between a frozen vision tower and a frozen causal LM, with the loss masking and causal-shift logic identical to LLaVA's, 12-test suite, no downloads. The full CLIP + TinyLlama + LLaVA-Instruct version below is the extension target.

Deliverables Checklist

• Configurable transformer with 5 tested architecture configs and perplexity table
• MoE block with balanced expert utilization under aux loss
• Mamba block with ZOH discretization and selective scan, memory comparison vs attention
• ViT + DINO lab passing (13 tests); (stretch) CIFAR-10 run with attention map visualizations
• Multimodal bridge lab passing (12 tests); (stretch) CLIP+TinyLlama with VQAv2 score
• All test_lab.py suites pass

Guides in This Phase

• HITCHHIKERS-GUIDE.md — attention math, ViT, MoE, Mamba overview, interview numbers
• DEEP-DIVE-MOE-SSM.md — Mixture of Experts and State Space Models from first principles (supports Labs 02–03)

Interview Relevance

• "What is the memory advantage of GQA over MHA at 32K context?" — you can compute: (h×2×n×d) vs (h/g × 2×n×d), show 8x reduction for g=8
• "How does DINO work without labels?" — explain centering + sharpening, why teacher EMA is key, why augmentation multi-crop works
• "What is the bottleneck in deploying LLaVA on an NPU?" — you understand the KV cache growth with visual tokens, the projection cost, and the LLM decode bottleneck
• "Why does RoPE generalize to longer sequences than learned position embeddings?" — explain the frequency decomposition and length-independent rotation property