System Design 01 — Model Accuracy Regression Platform

Scenario: "Design a platform that automatically detects when a model's accuracy regresses after a code or optimization change. The platform must support 50+ models, run in CI, and notify engineers within 10 minutes of a change landing."

1. Requirements Clarification

Functional:

• Trigger accuracy evaluation on every PR merge to main
• Compare against stored baseline with statistical significance testing
• Support pluggable benchmark suites (MMLU, custom domain tasks)
• Send alerts (Slack, GitHub PR comment) when regression detected
• Store historical accuracy trends (for dashboards and root-cause analysis)
• Support multiple model families: LLMs, vision, audio

Non-functional:

• P95 eval completion < 10 minutes (requires fast eval mode)
• 99.9% availability for CI integration
• Scale to 50 model families × 10 bit-width configs = 500 eval jobs/day
• False positive rate < 5% (avoid blocking valid PRs)

Out of scope: Full model training, production serving metrics, human evaluation.

2. Capacity Estimates

• 500 eval jobs/day, each ~5 min (fast mode) = 2,500 GPU-minutes/day
• 3 × A100 GPU nodes = 4,320 GPU-minutes/day → ~1.7× headroom
• Storage: 500 jobs/day × 200 examples/task × 5 tasks × 4 bytes = 2 MB/job × 500 = 1 GB/day
• Historical data: 1 GB/day × 365 days = ~365 GB/year → standard object storage

3. High-Level Architecture

┌──────────────────────────────────────────────────────────┐
│                    CI/CD Integration                      │
│  GitHub/GitLab webhooks → Job Queue → Eval Workers       │
└──────────────────────────────────────────────────────────┘
           ↓ submit job           ↑ results
┌──────────────────────────────────────────────────────────┐
│                    Eval Service                           │
│  ┌─────────────┐  ┌────────────┐  ┌──────────────────┐  │
│  │  Task       │  │  Model     │  │  Comparison      │  │
│  │  Registry   │  │  Loader    │  │  Engine          │  │
│  └─────────────┘  └────────────┘  │  (McNemar's)     │  │
│                                   └──────────────────┘  │
└──────────────────────────────────────────────────────────┘
           ↓ store                 ↑ baseline
┌──────────────────────────────────────────────────────────┐
│                    Storage Layer                          │
│  ┌─────────────────┐    ┌───────────────────────────┐   │
│  │  Results DB      │    │  Baseline Store           │   │
│  │  (PostgreSQL)    │    │  (S3/GCS per model)       │   │
│  └─────────────────┘    └───────────────────────────┘   │
└──────────────────────────────────────────────────────────┘
           ↓ alert
┌──────────────────────────────────────────────────────────┐
│                    Alert Service                          │
│  Slack notifications, GitHub PR comments, PagerDuty      │
└──────────────────────────────────────────────────────────┘

4. Core Components

4.1 Task Registry

class TaskRegistry:
    """Pluggable task registry — teams register their own tasks."""
    
    def register(self, task_name: str, task_cls: Type[Task], 
                 model_families: list[str]) -> None:
        """Register a task for a set of model families."""
        
    def get_tasks_for_model(self, model_id: str, mode: str) -> list[Task]:
        """
        mode = 'fast': ~5 min (200 examples per task)
        mode = 'full': ~60 min (full test sets)
        """

4.2 Comparison Engine (Statistical Testing)

class ComparisonEngine:
    def compare(
        self,
        baseline: PerExampleResults,
        candidate: PerExampleResults,
        policy: RegressionPolicy,
    ) -> ComparisonResult:
        """
        For each task:
          1. Run McNemar's test
          2. Compute absolute accuracy delta
          3. Apply policy: fail if p < policy.p_threshold AND |delta| > policy.delta_threshold
        """

@dataclass
class RegressionPolicy:
    p_threshold: float = 0.05         # statistical significance
    delta_threshold: float = 0.005    # 0.5% absolute minimum
    tasks_requiring_all_pass: list[str] = field(default_factory=list)
    tasks_requiring_any_pass: list[str] = field(default_factory=list)

4.3 Baseline Management

The baseline for each model is the latest main branch result. Strategy:

• On merge to main: run full eval (60 min), store as new baseline
• On PR: run fast eval (5 min), compare to baseline
• Baseline stored per (model_id, bit_width, quantization_method, task) tuple
• Immutable — once stored, a baseline is never overwritten (only new baselines added)

5. Fast Eval vs Full Eval

Fast eval must complete in < 10 minutes:

• Run 200 examples per task (calibrated to detect 2% absolute regression with p<0.05)
• Use stratified sampling (proportional to task subject distribution for MMLU)
• Run 3 tasks instead of 5 (drop the two least sensitive tasks)
• Use batch_size=32 (vs 64 for full) to reduce memory pressure and start faster

Full eval (nightly):

• All examples, all tasks
• Store results as new baseline if score is higher than current baseline
• Required before any production model release

6. Calibration of Fast Eval Thresholds

The fast eval uses only 200 examples per task. With 200 examples: $$\text{SE} = \sqrt{\frac{p(1-p)}{200}} \approx 0.035 \text{ (3.5%)}$$

To detect a 2% regression with 80% power at p<0.05:

• Need approximately 120 examples where models disagree
• With 200 examples and 50% overlap: expect 100 disagreements → marginal but workable for 2% regression

Recommendation: increase to 500 examples per task for reliable 1% regression detection.

7. Failure Modes and Mitigations

8. Monitoring and Observability

Metrics to track:

• Eval job P95 latency (alert if > 10 min)
• False positive rate per week (alert if > 5%)
• Baseline staleness (alert if > 7 days without new baseline)
• GPU utilization per node (alert if < 50% — indicates queue backup)

Dashboard:

• Accuracy trend per model over last 90 days
• Pareto frontier (accuracy vs latency) updated daily
• Model health matrix: green/yellow/red per (model, task, bit-width)

9. Trade-offs

10. Scale Considerations

At 10× scale (500+ models):

• Introduce priority queues: production models get fast-lane eval; experimental models batch
• Shard the eval cluster by model family (LLM workers vs vision workers have different GPU requirements)
• Cache model weights across eval runs (large models take minutes to load; LRU cache on workers)
• Use gradient checkpointing or batch-size-1 inference for memory-constrained evals