System Design 01 — Multi-Tenant GPU Cloud / Compute Platform

"Design a multi-tenant GPU cloud." Phases: 04 (drivers/containers), 06 (scheduling/sharing), 10 (observability/release), 11 (security/tenancy).

1. Drive the requirements

Ask before designing:

  • Scale: how many GPUs/nodes? thousands → cluster scheduling + fragmentation matter (P06).
  • Tenancy: multi-tenant (isolation is contractual) vs single-org? Sets the sharing mechanism (P06).
  • Workloads: training (gang, long, checkpointable — P08) vs serving (admission, SLO — P07)? Different schedulers.
  • Hardware: single-vendor or heterogeneous (→ HAL, P09)?
  • Sovereignty: on-prem/air-gapped tier (→ P11)?

Assume: thousands of GPUs, mixed training+serving, multi-tenant with paid/free tiers, NVIDIA + one other vendor, an on-prem tier.

2. Architecture

            +------------------ Control Plane -------------------+
   users -> | API / quotas / billing / licensing (P11)          |
            | Scheduler: bin-pack + gang + topology (P06/P08)    |
            | Tenancy & isolation policy (P06/P04/P11)           |
            +----------------------------------------------------+
                    |                         |
            +--------------+          +----------------+
            | Node agent   |  ...     | Node agent     |   (per node)
            | device plugin (P06)     | container runtime + GPU injection (P04)
            | DCGM exporter (P10)     | health/Xid monitor (P04/P10)
            +--------------+          +----------------+

3. The hard decisions (and the tradeoffs)

  • Sharing mechanism per tier (P06): dedicated/MIG for paid+isolated, time-slicing for free/dev, MPS only within a trusted pool. Tradeoff: isolation vs utilization; never co-place free+paid on one physical GPU.
  • Scheduler (P06/P08): best-fit to limit stranding; gang scheduling for multi-GPU jobs; topology-aware placement so TP stays on NVLink islands. Tradeoff: utilization vs fragmentation vs fairness (DRF).
  • Isolation (P04/P11): namespaces+cgroups+device injection per tenant; MIG for hardware isolation; remember allocation ≠ isolation (P04/P06).
  • Heterogeneous hardware (P09): the HAL lets workloads target either vendor; capability routing.

4. Failure modes & operations (P10)

  • Xid / GPU off the bus (P04): health monitor → cordon → drain → RMA.
  • Fragmentation/stranding (P06): alert on stranded-GPU count; best-fit + reserved whole-node pool for big jobs.
  • Node/collective failure for training (P08): checkpointing, elastic, fast detection (timeouts).
  • Observability: DCGM metrics, SLOs, the release pipeline (P10) for the control plane and node agents.
  • Driver fleet upgrades (P04 Q6): inventory → canary → drain → upgrade → progressive, image-pinned for rollback.

5. Tenant security (P11)

Per-tenant isolation (MIG/dedicated), licensing/quota at the control plane, audit logs; for the on-prem tier, the sovereign architecture (offline licensing, attestation — see system-design 04).

6. Commercial framing (P12)

Pricing tiers map to sharing mechanisms (isolation guarantee = price point, P06); heterogeneous support (the HAL) is lock-in freedom + cost; the on-prem/sovereign tier opens the regulated market. Utilization (best-fit, sharing) is gross margin.

The senior signal

You drove requirements, picked sharing mechanisms per tier with the isolation-vs-utilization tradeoff named, made the scheduler topology-aware (connecting P06 to P08), and closed with operations (P10) and the pricing model (P12) — breadth across the stack at the right altitude.