System Design 05 — Model Performance Benchmarking Infrastructure

================================================================

Design a reproducible, standardized infrastructure for measuring

ML model performance: latency, throughput, memory, and efficiency.

Problem Statement

Build a benchmarking infrastructure that:

• Provides reproducible latency/throughput measurements across hardware targets
• Supports GPU (A100, H100, 4090), edge (Snapdragon 8 Gen 3), and CPU
• Enables apples-to-apples comparison across models, frameworks, and backends
• Integrates into CI/CD to catch performance regressions
• Publishes reproducible benchmark artifacts

Scale: 50+ model variants, 10+ hardware targets, run weekly and on every PR.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                  Model Performance Benchmarking Infrastructure              │
└─────────────────────────────────────────────────────────────────────────────┘

[Benchmark Spec]
  - Model: model_id, checkpoint_sha
  - Input shapes: batch sizes, sequence lengths
  - Backend: torch, torch.compile, onnxruntime, TensorRT, SNPE, QNN
  - Hardware: GPU (A100), Edge (Snap 8 Gen 3), CPU
  - Metrics: latency (p50/p95/p99), throughput (req/s), memory (peak HBM), power (mW)

[Benchmark Runner]
  1. Provision hardware via hardware pool
  2. Load model in specified backend
  3. Warm up (N=20 iterations)
  4. Measure (N=1000 iterations, record all)
  5. Compute statistics: median, p95, p99, stddev, CV
  6. Collect system metrics: GPU utilization, memory, power
  7. Write to Results DB + publish artifact

[Hardware Pool]
  GPU cluster (A100, H100): SLURM or Kubernetes managed
  Edge farm: physical Snapdragon devices + AI Hub cloud
  CPU farm: ARM64 + x86 cloud VMs

[CI Integration]
  PR → fast benchmark (batch=1, seqlen=512, 50 iterations)
  Weekly → full benchmark matrix (all batch sizes, all seq lengths)

[Regression Detector]
  Compare new vs. baseline: alert if p50 latency increases > 5%

[Dashboard]
  Grafana: latency over time, efficiency radar, hardware comparison

Measurement Protocol

Latency Measurement (GPU)

def benchmark_latency_gpu(model, inputs, n_warmup=20, n_measure=1000):
    """
    Accurate GPU latency measurement using CUDA events.
    Returns: dict with latency statistics in milliseconds.
    """
    model.eval()
    
    # Warmup (JIT compilation, caching)
    with torch.no_grad():
        for _ in range(n_warmup):
            _ = model(*inputs)
    torch.cuda.synchronize()
    
    # Measurement
    times_ms = []
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    
    with torch.no_grad():
        for _ in range(n_measure):
            start.record()
            _ = model(*inputs)
            end.record()
            torch.cuda.synchronize()
            times_ms.append(start.elapsed_time(end))
    
    times_ms.sort()
    n = len(times_ms)
    
    return {
        "p50_ms": times_ms[n // 2],
        "p95_ms": times_ms[int(n * 0.95)],
        "p99_ms": times_ms[int(n * 0.99)],
        "mean_ms": sum(times_ms) / n,
        "stddev_ms": (sum((t - sum(times_ms)/n)**2 for t in times_ms) / n) ** 0.5,
        "cv": (sum((t - sum(times_ms)/n)**2 for t in times_ms) / n) ** 0.5 / (sum(times_ms) / n),
    }

Throughput Measurement

def benchmark_throughput(model, inputs, target_duration_s=30):
    """
    Measure sustained throughput by running for a fixed duration.
    More accurate than latency × batch_size for batched inference.
    """
    model.eval()
    batch_size = inputs[0].shape[0] if hasattr(inputs[0], 'shape') else 1
    
    # Warmup
    with torch.no_grad():
        for _ in range(10):
            model(*inputs)
    torch.cuda.synchronize()
    
    n_iters = 0
    t_start = time.perf_counter()
    
    with torch.no_grad():
        while time.perf_counter() - t_start < target_duration_s:
            model(*inputs)
            n_iters += 1
    
    torch.cuda.synchronize()
    total_time = time.perf_counter() - t_start
    
    return {
        "throughput_req_per_s": n_iters * batch_size / total_time,
        "throughput_tokens_per_s": None,  # set if applicable
        "n_iters": n_iters,
        "duration_s": total_time,
    }

Memory Measurement

def measure_peak_memory(model, inputs):
    """Measure peak GPU memory during inference."""
    if not torch.cuda.is_available():
        return {"peak_mb": 0, "allocated_mb": 0}
    
    torch.cuda.reset_peak_memory_stats()
    torch.cuda.synchronize()
    
    with torch.no_grad():
        _ = model(*inputs)
    torch.cuda.synchronize()
    
    return {
        "peak_mb": torch.cuda.max_memory_allocated() / (1024 ** 2),
        "allocated_mb": torch.cuda.memory_allocated() / (1024 ** 2),
        "reserved_mb": torch.cuda.memory_reserved() / (1024 ** 2),
    }

Benchmark Matrix

Dimension 1: Model sizes

• 1B, 3B, 7B, 13B, 70B parameter models

Dimension 2: Batch sizes

• 1 (online serving), 8, 32, 128 (batch serving)

Dimension 3: Sequence lengths

• 128, 512, 2048, 8192 tokens

Dimension 4: Backends

Dimension 5: Precision

• FP32, FP16, BF16, INT8, INT4 (W4A8, W4A16)

Results Database Schema

CREATE TABLE benchmark_runs (
    id              BIGSERIAL PRIMARY KEY,
    run_id          UUID NOT NULL DEFAULT gen_random_uuid(),
    model_sha       CHAR(64) NOT NULL,
    model_id        TEXT NOT NULL,
    backend         TEXT NOT NULL,
    hardware        TEXT NOT NULL,
    batch_size      INT NOT NULL,
    seq_len         INT NOT NULL,
    precision       TEXT NOT NULL,
    
    -- Latency (ms)
    latency_p50_ms  FLOAT,
    latency_p95_ms  FLOAT,
    latency_p99_ms  FLOAT,
    latency_cv      FLOAT,       -- coefficient of variation
    
    -- Throughput
    throughput_req_s  FLOAT,
    throughput_tok_s  FLOAT,
    
    -- Memory
    peak_memory_mb  FLOAT,
    
    -- Efficiency
    flop_per_s      FLOAT,
    roofline_efficiency FLOAT,  -- measured / attainable
    
    -- Metadata
    gpu_model       TEXT,
    cuda_version    TEXT,
    torch_version   TEXT,
    benchmark_hash  CHAR(16),   -- hash of benchmark spec (for exact repro)
    measured_at     TIMESTAMPTZ DEFAULT NOW()
);

-- View: latest results per (model, hardware, batch, precision)
CREATE VIEW latest_benchmarks AS
SELECT DISTINCT ON (model_id, hardware, batch_size, precision)
    *
FROM benchmark_runs
ORDER BY model_id, hardware, batch_size, precision, measured_at DESC;

Reproducibility

What makes benchmarks reproducible:

1. Pinned environment: Docker image with pinned torch==2.3.0+cu121, triton==2.3.0, driver version
2. Fixed hardware state: GPU boost clocks locked (nvidia-smi -lgc 1410,1410 for A100), no concurrent workloads
3. Fixed input: deterministic input generation (torch.manual_seed(42))
4. Warmup protocol: standardized warmup (20 iterations before measurement)
5. Artifact storage: save requirements.txt, Dockerfile, benchmark config, input tensors, and all raw timing measurements

Reproducibility score metric:

def reproducibility_score(run1_latencies, run2_latencies):
    """
    CV = std / mean of latencies across two independent runs.
    Score: 1 - max(CV_run1, CV_run2) — higher is more reproducible.
    """
    cv1 = torch.tensor(run1_latencies).std() / torch.tensor(run1_latencies).mean()
    cv2 = torch.tensor(run2_latencies).std() / torch.tensor(run2_latencies).mean()
    return 1.0 - max(cv1.item(), cv2.item())

Target: reproducibility score > 0.98 (< 2% CV).

Regression Detection

def detect_performance_regression(new_result, baseline_result, threshold=0.05):
    """
    Detect if new benchmark result is worse than baseline.
    Uses p95 latency as primary metric (not p50 — tail matters for SLO).
    
    threshold: relative degradation to flag (0.05 = 5%)
    """
    p95_delta = (new_result["latency_p95_ms"] - baseline_result["latency_p95_ms"]) \
                / baseline_result["latency_p95_ms"]
    
    throughput_delta = (new_result["throughput_req_s"] - baseline_result["throughput_req_s"]) \
                       / baseline_result["throughput_req_s"]
    
    regressions = []
    if p95_delta > threshold:
        regressions.append({
            "metric": "p95_latency",
            "delta": p95_delta,
            "baseline": baseline_result["latency_p95_ms"],
            "new": new_result["latency_p95_ms"],
        })
    if throughput_delta < -threshold:
        regressions.append({
            "metric": "throughput",
            "delta": throughput_delta,
            "baseline": baseline_result["throughput_req_s"],
            "new": new_result["throughput_req_s"],
        })
    
    return regressions

CI/CD Integration

Fast benchmark (every PR, < 5 min)

# .github/workflows/benchmark_ci.yml
name: Performance Benchmark CI
on: [pull_request]
jobs:
  benchmark:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - name: Run fast benchmark
        run: |
          python benchmark_ci.py \
            --model gpt2 \
            --batch_sizes 1,8 \
            --seq_lens 128,512 \
            --n_iters 50 \
            --compare_baseline \
            --fail_threshold 0.1

Full benchmark (weekly)

name: Weekly Full Benchmark
on:
  schedule:
    - cron: "0 2 * * 1"  # Monday 2 AM
jobs:
  full_benchmark:
    strategy:
      matrix:
        model: [gpt2, llama-7b, mistral-7b]
        batch_size: [1, 8, 32]
        seq_len: [512, 2048]
        backend: [eager, inductor, tensorrt]
    runs-on: [self-hosted, a100]
    steps:
      - name: Run benchmark
        run: python run_benchmark.py --model ${{ matrix.model }} ...
      - name: Upload results
        run: python upload_results.py --db ${{ secrets.DB_URL }}

Dashboard Metrics

Key views:

1. Latency over time: p50/p95/p99 per (model, hardware, backend) — detect regressions
2. Throughput vs. batch size: shows batching efficiency curve
3. Efficiency radar chart: 5 dimensions — throughput, memory, latency, accuracy, power
4. Pareto frontier: accuracy vs. latency/throughput for all model variants
5. Hardware comparison: same model across A100, H100, Snap 8 Gen 3

SLA definitions: