System Design 03 — Hardware-Agnostic Runtime (the HAL)

"Decouple our software from hardware." The JD's core mandate. Phases: 09 (HAL/ABI — the keystone), 03 (compilers/backends), 05 (runtime).

1. Drive the requirements

  • Which hardware, by when? NVIDIA today, AMD in 6 months, a startup in 12 — sets the backend roadmap.
  • Performance bar: near-native on the hardware that matters, or "runs everywhere, fast enough"? Sets the abstraction altitude (P09 Ch. 5).
  • Who writes backends? You, or the vendors (→ conformance + ABI versioning, P09).
  • Workloads: serving (P07), training (P08), or both — sets which ops the HAL must expose.

2. Architecture (the HAL pattern, P09)

   application -> STABLE API (hal_alloc, hal_op, hal_select_backend)
                     |  vtable dispatch
                +-----------+
                | HAL core  |  capability routing, ABI version guard, conformance
                +-----------+
                  |     |        |
        nvidia_backend  amd_backend  startup_backend   (dlopen'd, versioned ABI)
        (cuBLAS/cuDNN)  (rocBLAS)    (vendor lib / Triton)

The three pieces (P09 Ch. 3): stable API, versioned backend ABI (vtable + capabilities), dynamic loading. The core links no backend — a new chip is a new file.

3. The hard decisions (and tradeoffs)

  • Abstraction altitude (P09 Ch. 5): operation-level (attention, gemm) with capability-gated fast paths and an escape hatch — not instruction-level (lowest-common-denominator, slow) nor vendor-exposed (re-coupled). Tradeoff: portability vs peak.
  • Backend strategy (P09 Ch. 8 / P03 Ch. 9): vendor-library backends for hardware you must support today (near-peak, time to market) + a Triton/MLIR backend for kernel velocity and future chips. Build vs buy: build the HAL (moat), adopt the compiler.
  • ABI versioning (P09 Ch. 6): version-first, append-only, reserved fields, flags-over-fields, semver range — so a year-old vendor backend still loads. This is what sustains the vendor ecosystem.
  • Capability negotiation (P09 Ch. 4): backends declare; platform routes; beats LCD.
  • The AMD asymmetry (P03 Ch. 9): no stable PTX-equivalent → per-gfx rebuilds; budget the backend-engineering cost.

4. Conformance — what makes it trustworthy (P09 Ch. 7)

One test matrix, every backend passes: correctness (cross-vendor rounding tolerance — P03), declared-capability verification, and a performance tier ("supported" vs "recommended"). "Certified backend" status gates the ecosystem and is an OEM lever (P12 Ch. 5).

5. Runtime concerns (P05)

The HAL sits on the runtime contract: stream-ordered lifetime, pool segregation, no driver alloc/free in hot paths — these guarantees must hold across backends, so they're part of the ABI's behavioral contract, not just the syntactic one.

6. Failure modes & operations

  • A backend fails conformance → not certified; capability routing won't select it.
  • A backend dlsym fails / wrong ABI version → cleanly rejected (P09 Ch. 2).
  • Performance regression in a backend → the perf-conformance tier catches it (P10 release gate analog).

7. Commercial framing (P12 Ch. 6, 9)

The HAL is the moat and the OEM on-ramp: every certified backend = market access for a chip + lock-in freedom for customers = revenue/adoption. It inverts NVIDIA's software moat. Conformance certification is a quality promise and a negotiation chip. This is why the Head of Engineering owns it — it's where engineering and commercialization fuse.

The senior signal

You chose the right abstraction altitude (operation-level + capability-gated + escape hatch, not LCD), committed to a build-the-HAL/adopt-the-compiler strategy with a sequenced backend roadmap, made ABI versioning the ecosystem-trust mechanism, gated everything on conformance, and framed it as the moat — the JD's central architectural thesis, owned end to end.