Warmup Guide — Hardware Abstraction & Portability (The Keystone)

Zero-to-expert primer for Phase 09 — the JD's central mandate. Builds on every prior phase (you abstract hardware you understand). By the end you can design, build, and version a hardware-abstraction layer and argue the portability strategy to a board.

Table of Contents

Chapter 1: The Mandate — "Independent of the Underlying Hardware"

The JD's first responsibility: "Define and execute the strategy to decouple software from hardware environments" and build "a platform that operates independently of the underlying hardware." This phase is the technical content of that sentence.

Why it's valuable: NVIDIA's dominance is a software moat (Phase 03 Ch. 10) — CUDA + 18 years of kernels. Customers and OEMs want out from under the lock-in: AMD MI300X has more HBM, Intel Gaudi is cheaper, startups promise better tokens/joule — but switching means rewriting the stack. A platform that lets software run unchanged across vendors captures exactly that pain. That's the commercial thesis, and a hardware-abstraction layer (HAL) is its core artifact.

The hard truth this phase teaches: a HAL is easy to build badly (a thin wrapper that runs everywhere at 20% of peak) and very hard to build well (one API, multiple backends, near-native performance, evolvable without breaking shipped vendor plugins). The difference is Chapters 2–7.

Chapter 2: API vs ABI — the Crux of Every Plugin System

The single most important distinction in this phase, and one most engineers blur.

  • API (Application Programming Interface): the source-level contract — function names, signatures, types you compile against. "Call hal_matmul(a, b, c)." Breaking the API breaks compilation.
  • ABI (Application Binary Interface): the binary contract — how those calls are actually made at the machine level: struct memory layout, field offsets, calling convention, symbol names, enum values, alignment. Breaking the ABI breaks already-compiled binaries — silently, often as corruption rather than a clean error.

Why this is the crux of a HAL: backends are separately compiled binaries (vendor .so files you dlopen at runtime — possibly built by a different team, a different compiler, a year apart). They share no source with your core; they share only the ABI. So the contract between your platform and a vendor backend is binary, and getting it stable is the whole game.

The ABI rules you must internalize (Lab 01 enforces them):

  • Struct layout is the contract. If your core thinks Capabilities is {int version; int flags; int max_dtype;} and a backend was compiled when it was {int version; int flags;}, every field after flags is read at the wrong offset → garbage. Never reorder or insert fields in the middle; only append, and only with a version guard (Ch. 6).
  • Pass a version field first, so the core can refuse incompatible backends before reading misaligned fields.
  • Symbol visibility: a backend exports exactly one well-known entry point (e.g., hal_backend_register); everything else is hidden. The dlsym of that symbol is the only coupling.
  • Use fixed-width types (int32_t, not int) and explicit enums with assigned values — int width and enum representation are platform-dependent.

Misconception: "if it compiles, the ABI is fine." No — ABI breaks happen without recompiling the other side. The backend compiled last year still dlopens fine and then reads your new struct wrong. ABI discipline is about the binaries you didn't rebuild.

Chapter 3: The HAL Pattern — Stable API + Backend Vtable + dlopen

The architecture (Lab 01 builds exactly this):

   application
        |  calls the stable API
        v
  +-----------+      hal_matmul(...), hal_alloc(...), hal_select_backend(...)
  |  HAL core |  <-- ONE stable API the app compiles against
  +-----------+
        |  dispatches through a vtable of function pointers
        v
  +------------------ backend ABI ------------------+
  | struct Backend {                                |
  |   uint32_t abi_version;                          |
  |   Capabilities caps;                             |
  |   void* (*alloc)(size_t);                        |  <-- function-pointer
  |   int   (*matmul)(const Tensor*, ...);           |      vtable: the ABI
  |   ...                                            |
  | };                                               |
  +-------------------------------------------------+
        ^            ^             ^
   cpu_ref.so    fast.so      limited.so      <-- dlopen'd at runtime

The three pieces:

  1. Stable API — what the application calls. It never changes shape; that's the promise to application authors.
  2. Backend interface (the vtable) — a struct of function pointers + a capability struct. This is the ABI the vendor plugins implement.
  3. Dynamic loading — dlopen("backend_fast.so") then dlsym(h, "hal_backend_register") returns the filled vtable. The core never links against any backend at build time — which is what "decoupled from hardware" means mechanically: you can ship a new backend as a file, no core rebuild (Lab 01 extension 1 proves this).

This is precisely how real systems work: CUDA's driver API loads cubins; OpenCL/Level Zero load ICDs (Installable Client Drivers); LLVM loads target backends; even Postgres/Python load .so extensions. The pattern is universal; the discipline (Ch. 2, 6) is what's hard.

Chapter 4: Capability Negotiation — Beating the Lowest Common Denominator

The trap that kills naive HALs: if your API only exposes what every backend supports, you deliver the lowest common denominator — no FP8 because the old backend lacks it, no Tensor Cores because the CPU backend has none. Customers on good hardware pay for capabilities they can't use.

The fix is capability negotiation: each backend declares what it supports — dtypes (FP32/FP16/BF16/FP8/INT8), ops (matmul, attention, conv), features (unified memory, async, specific fusions), and performance hints — in a capability struct. The platform then:

  • Queries capabilities at load time.
  • Routes each workload to a backend that supports it (a FP8 request goes to an FP8-capable backend; a CPU-only deployment falls back to FP32).
  • Exposes the union of capabilities to applications, with a clean failure ("no loaded backend supports FP8") rather than a silent downgrade.

This is the difference between a HAL that's a product and one that's a tax. Lab 01 implements capability routing: a request needing FP16 is steered away from the "limited" backend; the demo shows the router picking correctly. It also mirrors Phase 06's device-plugin capability advertising and Phase 07's multi-engine routing — same pattern, different layer.

Chapter 5: The Abstraction-vs-Performance Tension

The central design tension, and the honest hard part of "write once, run anywhere, fast."

  • Too abstract → leaky or slow: a single generic matmul can't express each backend's fast path (Tensor Core fragment shapes, AMD's matrix cores, a startup's systolic array), so you get correct-but-mediocre performance.
  • Too concrete → not portable: expose vendor-specific knobs and the application code becomes vendor-coupled again — you've lost the decoupling.

The resolutions real platforms use (you should be able to name all four):

  1. Escape hatches: a generic path plus an optional backend_native_op() that lets performance-critical code call a backend's fast path when present, falling back otherwise. Pragmatic; slightly leaky.
  2. High-level ops: abstract at the operation level (attention, layernorm), not the instruction level — so each backend implements the op optimally inside the abstraction (this is why serving engines expose "attention" not "matmul+softmax+matmul"). The most successful posture.
  3. A compiler layer: don't hand-write per-backend kernels — lower a portable IR to each backend (Triton/MLIR, Phase 03 Ch. 9). Highest ceiling, highest cost.
  4. Capability-gated fast paths: the abstraction picks the best implementation based on declared capabilities (Ch. 4).

The leadership judgment: abstract at the right altitude. Too low and you re-expose the hardware; too high and you can't reach peak. "Operation-level abstraction with capability-gated fast paths and an escape hatch" is the defensible default — and it's what the capstone (Phase 12) builds.

Chapter 6: Versioning a Plugin ABI

The chapter that separates a HAL that survives contact with vendors from one that breaks every release. You ship a backend ABI; vendors build .so files against it; then you need to evolve it without breaking their year-old binaries.

The techniques (Lab 01 extension 3 demonstrates them):

  • A version field, first in the struct. The core reads abi_version before anything else and refuses (cleanly) backends it can't support — never reading misaligned fields from a too-new or too-old struct.
  • Append-only struct growth. New capabilities/functions go at the end of the struct/vtable, behind a version check. An old backend (smaller struct) still has its early fields at the right offsets; the core just doesn't read the new tail for old abi_versions.
  • Reserved fields. Ship uint32_t reserved[8] so future fields don't even change the struct size — old and new backends agree on layout.
  • Capability flags over struct changes. Prefer adding a flag ("supports FP8") to an existing flags field over adding a struct field — flags are forward/backward compatible by construction.
  • Semantic versioning of the ABI: major = breaking (old backends rejected), minor = additive (old backends still load). The core supports a range.

The real-world stakes: CUDA's forward-compatibility (Phase 03/04), OpenCL ICD versioning, the Linux kernel's stable-ABI-for-userspace promise — all are this chapter at industrial scale. A platform whose ABI breaks vendor backends every quarter has no vendor ecosystem, and the ecosystem is the product (Ch. 9).

Chapter 7: Conformance Testing — What Makes an Abstraction Trustworthy

An abstraction is only as good as its guarantee that every backend behaves the same. Without that, applications must special-case backends — and you've lost portability. The mechanism is a conformance suite: one test matrix every backend must pass.

  • Correctness conformance: the same operations, same inputs, same expected outputs (within a stated tolerance — Phase 03's FMA/rounding caveat applies across vendors!) on every backend. Lab 01's demo runs one conformance test against all backends.
  • Capability conformance: a backend that declares FP16 support must actually produce correct FP16 results — declarations are tested, not trusted.
  • Performance conformance (the advanced tier, Lab 02 extension): a backend must hit X% of its roofline (Phase 01) on key ops, or it's "supported but not recommended" — preventing the "technically runs, uselessly slow" backend from poisoning the platform's reputation.

This is how Khronos (OpenCL/Vulkan conformance), the C/C++ standards (test suites), and every credible plugin ecosystem work: the conformance suite is the contract. As a leader you'd gate "certified backend" status on it — which is also a vendor-relations and commercial lever (Ch. 9).

Chapter 8: The Portability Landscape, as Backend Strategy

Revisiting Phase 03 Ch. 9 from the HAL builder's seat — these are your backend options:

  • Vendor compute libraries directly (CUDA+cuBLAS/cuDNN, ROCm+rocBLAS, oneAPI+oneMKL, Metal Performance Shaders): each backend wraps the vendor's tuned kernels. Fastest to near-peak per vendor, most code (one backend per vendor), the pragmatic start.
  • A portable kernel layer (Triton, increasingly multi-backend; or SYCL/oneAPI): write kernels once, lower to each backend. Less per-vendor code, inherits the compiler's backend coverage, ~90% of peak on supported ops (Phase 03 Ch. 9).
  • A full compiler stack (MLIR dialects → each target): maximum control, maximum cost; the moonshot.
  • SPIR-V/Vulkan compute: truly neutral, historically below peak for ML.

The platform reality (Phase 03 Ch. 10's three postures, now as architecture): blend — vendor-library backends for the hardware you must support today (time to market), a Triton/MLIR backend strategy for kernel velocity and future hardware, all behind your HAL (this phase) with capability routing (Ch. 4). The HAL is the constant; the backends are how you fill it. The AMD asymmetry from Phase 03 (no stable PTX-equivalent, per-gfx code objects) is a real backend-engineering cost your HAL must budget for.

Chapter 9: The Business Case — Why This Layer Is the Moat

Why a Head of Engineering — not just an architect — owns this:

  1. It's the commercialization mandate. The JD lists "own platform adoption and commercialisation through partnerships" and "drive integrations with OEMs, cloud providers, and infrastructure partners." Every one of those integrations is a backend behind your HAL. The HAL's quality determines how fast you can onboard an OEM's chip — a direct revenue lever.
  2. It inverts NVIDIA's moat. NVIDIA's lock-in is software; a platform that makes software hardware-agnostic sells freedom from that lock-in to every customer and every non-NVIDIA chipmaker who wants market access. Your HAL is the chipmakers' on-ramp to the software ecosystem they can't build alone.
  3. Conformance is a commercial gate. "Certified backend" status (Ch. 7) is leverage in OEM negotiations and a quality promise to customers — you control who gets to be in the ecosystem and at what performance bar.
  4. It's defensible. Anyone can wrap two SDKs in an afternoon; a performant, versioned, conformance-gated, multi-vendor HAL with an ecosystem of certified backends is years of accumulated engineering and relationships — the same compounding moat dynamic as CUDA, pointed the other way.

The capstone (Phase 12) composes this HAL with the scheduler (06), serving (07), and licensing (11) into the platform the JD describes. This phase is its spine.

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02 (build the HAL, then write the strategy around it).

Lab 01 (the HAL in C):

  1. make && ./hal_demo; read hal.h first — the ABI is the contract, study the Backend struct and Capabilities against Ch. 2–3.
  2. Read hal.c (loader/router) then the three backends; trace one hal_matmul call from API → vtable → backend.
  3. Reproduce: capability routing (FP16 request avoids the "limited" backend); ABI-version rejection of a stale backend.
  4. Do extension 1 — add a 4th backend as a new .so without recompiling the core. When it loads and passes conformance, you've felt what "decoupled from hardware" means. Then extension 3 (ABI evolution) for Ch. 6.

Lab 02 (strategy ADR + conformance):

  1. Read STRATEGY-ADR.md; it's a worked decision record — study its structure.
  2. python conformance.py; understand the support matrix + pass/fail logic (Ch. 7).
  3. Write your own ADR for the multi-vendor scenario — costed options, a phased rollout, the conformance gate. This is a Phase 12 leadership artifact in miniature.

Success Criteria

  • You can state the API/ABI difference and the rules that keep an ABI stable (Ch. 2)
  • You can draw the HAL pattern (API + vtable + dlopen) and explain why the core never links a backend (Ch. 3)
  • You can explain capability negotiation and why it beats LCD (Ch. 4)
  • You can name the four resolutions to the abstraction-vs-performance tension and pick the default (Ch. 5)
  • You can version a plugin ABI so a year-old backend still loads (Ch. 6) — and your Lab 01 extension proves it
  • You can explain why conformance testing makes the abstraction trustworthy (Ch. 7)
  • You can argue the portability-posture blend and the business case (Ch. 8–9)

Interview Q&A

Q1: Design a hardware-abstraction layer for an AI platform. Walk me through the API/ABI boundary. A: Three pieces. A stable API the application compiles against (hal_alloc, hal_matmul, hal_select_backend) — its shape never changes, that's the promise to app authors. A backend ABI — a struct of function pointers (the vtable) plus a capability struct — that vendor plugins implement; this is the binary contract, so it's governed by ABI rules: version field first, fixed-width types, append-only struct growth, reserved fields, one exported registration symbol. And dynamic loading — the core dlopens each backend .so and dlsyms its registration function to get the filled vtable; the core links no backend at build time, which is the mechanical meaning of "decoupled from hardware" — a new chip is a new file. The API/ABI split is the crux because backends are separately compiled binaries sharing only the ABI, so a year-old vendor .so must still load correctly — which drives every versioning decision.

Q2: How do you avoid a lowest-common-denominator abstraction? A: Capability negotiation. Each backend declares what it supports — dtypes, ops, features, perf hints — in a capability struct read at load time. The platform routes each workload to a backend that supports it (FP8 work to an FP8-capable backend), exposes the union of capabilities to applications, and fails cleanly ("no backend supports FP8") rather than silently downgrading. Combined with operation-level abstraction (expose attention, not matmul+softmax) so each backend implements ops with its fast path inside the abstraction, plus optional native escape hatches for the last mile — you get portability without forcing everyone down to the weakest backend's feature set. The anti-pattern is an API that only exposes the intersection of all backends; that makes good hardware pay for the worst backend's limits.

Q3: How do you version a plugin ABI so a year-old vendor backend still loads? A: Put an abi_version field first and check it before reading anything else, so an incompatible backend is rejected cleanly instead of read at wrong offsets. Grow the interface append-only — new functions/fields at the end of the vtable/struct, gated by version, so an old (smaller) backend's early fields stay at correct offsets and the core simply doesn't read the new tail for old versions. Ship reserved fields so additions don't even change struct size. Prefer capability flags over new struct fields (flags are compatible by construction). Use semantic versioning — major = breaking (old backends rejected), minor = additive (old still load) — and have the core accept a range. This is exactly how OpenCL ICDs, CUDA forward-compat, and the Linux userspace-ABI promise work; breaking vendor backends every release means no vendor ecosystem, and the ecosystem is the product.

Q4: Build vs adopt — would you write your own compiler layer or use Triton/MLIR for kernels? A: Default to adopt. MLIR exists precisely so platforms don't hand-roll an IR: dialects, pass infrastructure, and existing lowerings to NVPTX/ROCm are free; Triton gives ~90% of hand-tuned on the kernels LLMs need with Python-level authoring and multi-backend lowering. Writing a full compiler re-pays all of that for marginal fit gains and an unbounded maintenance bill, and it slows your ability to absorb the next FlashAttention-shaped innovation (ecosystem velocity). I'd build my own only for a genuinely novel execution model the dialects can't express. The pragmatic architecture is a blend: vendor-library backends for hardware I must support today (time to market), a Triton/MLIR backend for kernel velocity and future chips, all behind my own HAL with capability routing — the HAL is mine (it's the moat), the kernel compiler I adopt.

Q5: How is "decoupling software from hardware" actually a defensible business? A: NVIDIA's dominance is a software moat — CUDA plus two decades of kernels — so customers and non-NVIDIA chipmakers share a pain: switching hardware means rewriting the stack. A platform that runs software unchanged across vendors sells freedom from that lock-in to customers and market access to every AMD/Intel/ startup chip that can't build a CUDA-class ecosystem alone — they become backends behind the HAL. The defensibility compounds: a performant, versioned, conformance-gated, multi-vendor HAL with an ecosystem of certified backends and OEM relationships is years of engineering and trust, not an afternoon's wrapper. Conformance certification is a commercial gate you control; OEM integrations are revenue; and the whole thing is the inverse of CUDA's moat — equally compounding, pointed at the incumbent. That's why the JD frames it as both an engineering and a commercialization mandate, and why the Head of Engineering owns it.

References

  • Ulrich Drepper, "How To Write Shared Libraries" (the ABI/dlopen bible) — https://www.akkadia.org/drepper/dsohowto.pdf
  • man dlopen, man dlsym — the loader API you'll build on
  • Khronos OpenCL ICD + conformance model — https://www.khronos.org/conformance/
  • Linux kernel "stable API nonsense" (Greg KH) — the opposite philosophy, instructive on why userspace ABI stability matters — https://www.kernel.org/doc/html/latest/process/stable-api-nonsense.html
  • Intel oneAPI / Level Zero spec (a real cross-vendor HAL) — https://spec.oneapi.io/level-zero/latest/
  • LLVM "Writing an LLVM Backend" (the compiler-as-backend posture) — https://llvm.org/docs/WritingAnLLVMBackend.html
  • Triton + MLIR (Phase 03 references) — https://triton-lang.org , https://mlir.llvm.org
  • Architecture Decision Records (Michael Nygard) — https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions
  • Cross-track: Phase 03 WARMUP Ch. 9–10 (the compiler layer), Phase 12 (the capstone this spine supports)