Interview Prep — File 04: Systems Design Questions

=====================================================

10 realistic system design walkthroughs for senior staff ML engineer interviews.

Each question includes: problem statement, clarifying questions, architecture,

key decisions, back-of-envelope calculations, failure modes, and talking points.

Design Prompts Overview

Design 1: Accuracy Regression CI Pipeline

Prompt: Design a CI system that detects accuracy regressions when engineers submit PRs with model changes.

Clarifying Questions

• What benchmarks are required? (MMLU, HellaSwag, task-specific)
• What is the acceptable CI time? (<10 min vs <2 hr)
• Is 0.1% accuracy drop OK? Or 0% regression tolerance?
• How many model checkpoints per PR? (1 vs N experiments)
• Infrastructure: on-prem GPU cluster or cloud?

Architecture

PR Webhook
    │
    ▼
[Trigger Service] → picks eval tier based on PR diff size
    │
    ├── Tier 1 (fast, <5 min): N=100 MMLU subset, perplexity probe on 1k tokens
    │
    └── Tier 2 (full, <2 hr):  full MMLU, HellaSwag, MT-Bench, task-specific evals
         │
         ▼
[Eval Workers] (GPU fleet, Ray cluster)
    │   - Download model checkpoint from artifact store
    │   - Run harness in parallel across benchmarks
    │   - Write results → central DB
    │
    ▼
[Stats Engine]
    │   - Fetch baseline results (main branch) from DB
    │   - Compute Δacc, McNemar's test p-value for each benchmark
    │   - Apply regression policy: flag if any Δacc < -0.5% with p < 0.05
    │
    ▼
[PR Comment Bot] → posts table of before/after accuracy per benchmark
    │
    └── Pass/Fail gate → blocks merge if regression detected

Key Design Decisions

1. Baseline pinning: Store baseline results per model variant in a versioned database (PostgreSQL + TimescaleDB). Never recompute baselines at PR time — they drift.

2. Statistically valid comparisons: Use McNemar's test (paired accuracy), not just point estimate difference. Threshold: p < 0.05 AND |Δacc| > 0.5%.

3. Fast path vs full path: Tier 1 runs on every PR (5 min). Tier 2 runs only when model weights change (detected via md5 of checkpoint), not just code changes.

4. Parallelism: Each benchmark runs on a separate GPU. For 10 benchmarks at 2 min each = 2 min wall time with 10 GPUs.

5. Caching: Cache forward pass outputs (logits) for unchanged model files. Eval reuses cached logits to recompute metrics without re-running the model.

Back-of-Envelope

• MMLU (57 subjects × 100 examples × 4 choices) = 22,800 forward passes
• GPT-2 @ 10ms/pass on A10G = 228 seconds
• LLaMA-7B @ 100ms/pass = 2,280 seconds → need batching or sampling
• With batch_size=32: 22,800/32 = 712 batches × 30ms = ~22 seconds

Failure Modes & Mitigations

• GPU OOM on eval worker: Cap batch size; use gradient checkpointing flag; shard model
• Flaky results (stochastic decoding): Use greedy decoding for classification; set seeds
• Baseline drift: Recompute baselines on a weekly cadence; alert when baseline changes
• False positives: Two-sided test with Bonferroni correction for multiple benchmarks

Design 2: NPU Deployment Pipeline for Snapdragon

Prompt: Build an automated pipeline that takes a PyTorch model and produces a validated .dlc (SNPE) or .bin (QNN) artifact for Snapdragon deployment.

Pipeline Stages

PyTorch Model (FP32)
      │
      ▼ [Stage 1: Export]
  torch.export() / torch.onnx.export()
  → model.onnx
      │
      ▼ [Stage 2: Optimization]
  onnxoptimizer.optimize() + onnxsim
  Fusion: Conv+BN+ReLU, GELU approximation
  → model_optimized.onnx
      │
      ▼ [Stage 3: Quantization Calibration]
  SNPE quantization tool or QNN quantize_model
  Calibration dataset: 500 representative samples
  → model_quantized.onnx (INT8 weights + INT8 activations)
      │
      ▼ [Stage 4: Compilation]
  snpe-onnx-to-dlc --input_dim "input_1:1,3,224,224"
  OR: qnn-onnx-converter + qnn-model-lib-generator
  → model.dlc / model.bin
      │
      ▼ [Stage 5: Validation on-device / AI Hub]
  snpe-net-run --container model.dlc --input_list inputs.txt
  Compare: FP32 baseline vs INT8 output (CosSim > 0.99 threshold)
  Latency: measured on HTP (target < 10ms for 224×224 images)
      │
      ▼ [Stage 6: Artifact Registry]
  Store: model.dlc + metadata.json + benchmark_results.json
  Trigger: mobile CI test suite

Key Decisions

1. Calibration data: Must be representative of production distribution. Use 500 images from each class for classification; 100 conversations for LLMs. Never use test set.

2. Accuracy validation metric: Cosine similarity between FP32 and INT8 outputs (layer-by-layer). Any layer with CosSim < 0.99 triggers a precision upgrade to FP16 for that layer.

3. Op coverage: SNPE/QNN don't support all PyTorch ops. Pre-check with snpe-onnx-to-dlc --dry_run. Common unsupported ops: custom activations, unfoldable reshapes, complex attention patterns.

4. Fallback strategy: CPU/GPU fallback for unsupported HTP ops. Profile: target >80% of ops on HTP.

5. AI Hub integration: Use qai_hub.submit_compile_job() + qai_hub.submit_profile_job() for on-device measurement without physical hardware.

Accuracy / Latency Trade-off Table

Design 3: Quantization Error Recovery System

Prompt: Build a system that automatically applies quantization-aware training (QAT) to recover accuracy lost during PTQ.

Decision Framework

PTQ Results
    │
    ▼
Is accuracy drop < threshold?  (e.g., <1% for INT8, <3% for INT4)
  YES → ship PTQ model
  NO  → run sensitivity analysis
           │
           ▼
       Per-layer sensitivity:
         Quantize one layer at a time, measure accuracy drop
         Rank layers by sensitivity
           │
           ▼
       Mixed-precision:
         Sensitive top-20% layers → FP16 or INT8
         Rest → INT4
           │
           ▼
       If still insufficient → QAT
         Initialize from pre-trained weights
         Insert fake-quant nodes (torch.quantization.prepare_qat)
         Fine-tune for 1-5% of original training steps (LR = 1e-4 × base_LR)
         Use straight-through estimator (STE) for quantization gradient
           │
           ▼
       Final accuracy check → pass/fail

Key Algorithms

ADAROUND (Nagel et al. 2020): Instead of round-to-nearest, optimize rounding decisions per weight:

min_v ||W - Q(W, v)||_F^2 + λ * f_reg(v)
where Q(W, v) = floor(W/scale) + v  (v ∈ {0,1})
f_reg(v) = regularization to push v toward 0 or 1

Recovers ~0.5-1% vs standard rounding.

SmoothQuant (Xiao et al. 2022): Transfer quantization difficulty from activations to weights:

Y = (X * diag(s)^{-1}) @ (diag(s) * W)
Choose s_j = max(|X_j|)^α / max(|W_j|)^{1-α}

Makes activations easier to quantize (less outliers).

Failure Modes

• QAT divergence: LR too high; use warm-up + cosine decay
• Calibration set overfitting: Use held-out validation set for sensitivity ranking
• INT4 fails for attention: Key/Query projections need INT8 or FP16 (high sensitivity)

Design 4: Real-time Model Serving at 100k QPS

Prompt: Design serving infrastructure for an LLM-based feature at 100k requests per second with p99 latency < 100ms.

Back-of-Envelope

• 100k QPS, avg input 50 tokens, avg output 20 tokens
• LLaMA-7B on A100: 100 tokens/s decode → 20 tokens × 10ms = 200ms → TOO SLOW for single request
• Need: batching, speculative decoding, or smaller model

Revised approach:

• 50ms budget → max 50 tokens/s per request → max 50 tokens output
• With batch_size=64 on A100: 64 × 50 tokens/s = 3200 tokens/s throughput
• A100 fleet needed: 100k req/s × (70 tokens/req) / 3200 tokens/s/GPU = 2,188 GPUs
• Alternative: distilled 1B model → 10× faster → 219 GPUs

Architecture

Load Balancer (L7, consistent hash on user_id)
    │
    ├── Frontend pods (stateless, auth, rate limit)
    │
    ▼
Request Router (picks inference cluster based on model version)
    │
    ▼
Inference Cluster (per model version, continuous batching)
    │
    ├── vLLM / TGI serving with:
    │   - PagedAttention (KV cache pooling)
    │   - Continuous batching (no padding waste)
    │   - Speculative decoding (small draft model)
    │
    ▼
Response Streaming (SSE / WebSocket for token-by-token output)

Latency Budget

Key Optimizations

1. Continuous batching (vs static batching): 30-50% throughput improvement
2. KV cache paging (PagedAttention): eliminates memory fragmentation, 2× capacity
3. Speculative decoding: 3-5× speedup for repetitive text at cost of draft model
4. INT8 serving: 2× throughput, marginal accuracy loss with calibrated models

Design 5: Distributed Offline Evaluation Platform

Prompt: Build a platform that runs weekly accuracy evaluations of 100+ model checkpoints across 50 benchmarks.

Scale

• 100 models × 50 benchmarks × 1000 examples = 5M evaluations
• Each eval: ~100ms → 140 GPU-hours → need parallelism

Architecture

Job Scheduler (Airflow DAG, weekly trigger)
    │
    ▼
Task Graph:
  For each (model, benchmark) pair:
    1. Fetch model checkpoint (S3/GCS) — dedup by sha256
    2. Run eval worker (Ray remote task)
    3. Write results to Results DB (PostgreSQL + partitioned by date)
    4. Aggregate into dashboard

Ray Cluster:
  Head node: DAG orchestration
  Worker nodes: 50× A10G GPUs
  GPU memory: load model once, eval all benchmarks, release

Deduplication:
  - Cache eval results by (model_sha, benchmark_version)
  - Skip re-evaluation if hash unchanged → typical: only 30% of checkpoints changed

Results Database Schema

CREATE TABLE eval_results (
    id           BIGSERIAL PRIMARY KEY,
    model_id     TEXT NOT NULL,
    model_sha    CHAR(64) NOT NULL,
    benchmark    TEXT NOT NULL,
    n_correct    INT,
    n_total      INT,
    accuracy     FLOAT,
    eval_time_s  FLOAT,
    evaluated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX ON eval_results (model_id, benchmark, evaluated_at DESC);
CREATE UNIQUE INDEX ON eval_results (model_sha, benchmark);  -- dedup

Monitoring

• Alert: job fails to complete in 24h
• Alert: accuracy drops > 2% vs previous week for any (model, benchmark)
• Dashboard: accuracy over time per model family

Design 6: Model Performance Benchmarking Infrastructure

(See system-design/05-model-perf-benchmarking-infra.md)

Design 7: Online Accuracy Monitoring for Deployed Models

Prompt: Monitor accuracy of a production model serving 10M requests/day without ground truth labels.

Approach: Shadow Evaluation + Proxy Metrics

Production Traffic (10M/day)
    │
    ├─── Production Model → response to user
    │
    └─── Shadow Model (previous version)
              │
              ▼
         Compare outputs:
           - Token-level agreement rate
           - Semantic similarity (embedding cosine)
           - Length distribution (statistical test)
              │
              ▼
         Sample 0.1% for human annotation (10k/day)
              │
              ▼
         Accuracy estimate with confidence interval

Drift Detection

• Concept drift: Track input distribution (embedding PCA, feature statistics)
• Data drift: Track input length, vocabulary distribution (KL divergence alert)
• Output drift: Track token entropy, refusal rate, sentiment

Alert Thresholds

• Agreement rate drops by > 5% vs 7-day rolling average → page on-call
• Human annotation accuracy drops by > 1.5% → rollback candidate
• KL divergence of input distribution > 0.5 → model retraining trigger

Design 8: GenAI On-Device Inference Pipeline

Prompt: Deploy a 7B parameter LLM on Snapdragon 8 Gen 3 (16 GB LPDDR5, 45 TOPS NPU).

Memory Budget

Layer Streaming Strategy

For models >4 GB (e.g., 13B INT4 = 6.5 GB):

Flash storage (60 GB/s NVMe) → prefetch next 4 layers while computing current 4
Working set: 4 layers in HBM × 2 (double-buffer) = ~1 GB active
Effective: overlap compute and I/O

Compilation Pipeline

1. GGUF format (llama.cpp) → direct quantization, no conversion needed
2. QNN compilation: qnn-context-binary-generator for HTP caching
3. First-run: compile + cache .qnncache binary (30s)
4. Subsequent runs: load from cache (200ms)

Optimization Checklist

• INT4 weights (AWQ or GPTQ)
• FP16 KV cache (INT8 possible with <0.2% accuracy loss)
• GQA with n_kv_heads=4 (8× KV cache reduction vs MHA)
• Sliding window attention (max context 2048, window 512)
• Speculative decoding with 68M draft model

Design 9: A/B Testing Framework for Model Variants

Prompt: Design an A/B testing system for safely rolling out a new model version.

Requirements

• Traffic split: 10%/90% initially → ramp to 50%/50%
• Primary metric: task-specific accuracy (requires labels or human eval)
• Guardrail metrics: latency p99, error rate, user engagement
• Statistical significance: 95% confidence, minimum 10k samples per variant

Architecture

User Request
    │
    ▼
[Experiment Assignment] (consistent hash by user_id for sticky sessions)
    ├── Group A (control, current model)
    └── Group B (treatment, new model)
    │
    ▼
[Logging] → event stream (Kafka)
    │
    ▼
[Stats Engine] (hourly batch)
    │   - Compute accuracy / latency per group
    │   - Run two-proportion z-test for accuracy
    │   - Run Mann-Whitney U for latency
    │   - Check guardrail violations
    │
    ▼
[Dashboard + Auto-ramp]
    - If p < 0.05 AND accuracy_B > accuracy_A AND no guardrail violations:
        ramp traffic to 20% → 50% → 100% over 24h
    - If guardrail violated: auto-rollback, page on-call

Sample Size Calculation

For accuracy test:

• Baseline accuracy: 75%
• MDE: 1% (detect 74% vs 75%)
• α = 0.05, β = 0.20 (80% power)
• n ≈ 2 × (z_α/2 + z_β)² × p(1-p) / (MDE)²
• n ≈ 2 × (1.96 + 0.84)² × 0.75 × 0.25 / 0.01² ≈ 29,400 samples per arm

Design 10: Continual Training + Accuracy Preservation

Prompt: Update a deployed model on new data without forgetting previously learned tasks (catastrophic forgetting).

Strategies

1. Elastic Weight Consolidation (EWC)

L_total = L_new_task + λ * Σ_i F_i (θ_i - θ*_i)²
F_i = Fisher information = E[∂²log P/∂θ_i²]  (importance of param i)

• Cost: 2× memory (store F and θ*)
• Effect: weights important for old tasks resist change

2. Replay Buffer

• Store 1-5% of old task examples
• Mix with new task data during training
• Effective but expensive for large datasets

3. LoRA + Task-specific Adapters

• Freeze base model, train lightweight adapters per task
• No forgetting by construction
• Inference: load base + current task adapter

Accuracy Tracking During Continual Training

for each training step t:
    train on new_task_batch
    if t % eval_every == 0:
        for task in old_tasks:
            acc[task][t] = evaluate(model, task.validation_set)
        backward_transfer = mean(acc[task][end] - acc[task][start] for task in old_tasks)
        if backward_transfer < -0.02:  # >2% forgetting
            log_alert(f"Catastrophic forgetting detected: {backward_transfer:.1%}")
            apply_ewc_regularization()

Metrics

• Backward Transfer (BT): Accuracy on old tasks after training on new task. Target: BT > -2%.
• Forward Transfer (FT): Accuracy on new task vs training from scratch. Target: FT > 0%.
• Plasticity-Stability Score: BT × FT normalized → composite metric.