« Track Overview

Platform Engineering Lab Standard

Every lab in this track builds a runnable, test-verified miniature of a real piece of AI platform infrastructure: the MCP tool registry, the A2A task machine, the LLM gateway's routing and fallback path, the PTU capacity planner, the hybrid retriever, the SHACL validator, the RFC 8693 token-exchange chain, the policy engine, the idempotent action gateway, the guardrail chain, the ISO 20022 parser, the resource graph with drift detection, the burn-rate alerting engine, the evidence graph.

You do not "add an API key to a .env and call a gateway." You build the gateway — the component that normalizes six providers behind one contract, picks a route under a policy, falls back inside a latency budget, meters tokens per tenant, and refuses the request when the tenant is over quota. That is what makes the knowledge defensible in a principal-level design review and useful at 3 a.m. when the platform is degraded and thirty agent teams are paging you.

Table of Contents

• Why miniatures
• The one trick that makes platforms testable: inject everything ambient
• Required files per lab
• Determinism rules
• The test contract
• Test taxonomy
• Doc conventions
• Definition of done

Why miniatures

A real AI platform is non-deterministic (the model samples), expensive (every step is tokens or GPU-seconds), credentialed (nothing runs without Entra, a vault, and a private endpoint), and hidden behind vendor surface (APIM policies, Bedrock's IAM, LiteLLM's config, Istio's CRDs). None of that is learnable by reading a YAML reference.

A lab that reimplements the algorithm — the JSON-RPC dispatch, the task state machine, the weighted-route selection, the token-bucket refill, the RRF fusion, the SHACL shape check, the delegation-chain construction, the Rego-style deny-overrides evaluation, the idempotency replay, the saga compensation, the multi-window burn-rate comparison — is offline, deterministic, free, and exactly the part an interviewer probes.

Each lab README ends with "How this maps to the real stack": the miniature's real-world equivalent (Azure APIM / LiteLLM / Kong AI Gateway / SPIRE / OPA / Temporal / Prometheus / Neo4j / Apache Jena / Terraform), what it faithfully reproduces, and what it deliberately simplifies.

The one trick that makes platforms testable: inject everything ambient

A platform's untestable parts are always the same four things: the model, the clock, the network, and randomness. So every lab injects all four:

Gateway(
    providers={"azure": FakeProvider(...), "bedrock": FakeProvider(...)},  # the model
    now=clock,             # Callable[[], float] — an integer tick counter in tests
    rng=random.Random(7),  # seeded; never the module-level random
    transport=FakeTransport(...),  # the network: scripted responses, injectable failures
)

The result: the platform becomes a pure function of its inputs, a test can assert the exact sequence of routing decisions, and a failure is reproducible from its seed. This is not a teaching shortcut — it is precisely the seam real platform teams use (record/replay fixtures, fake clocks in SRE simulations, httpx.MockTransport, SPIRE's in-memory plugin). It also teaches the discipline that defines this role: write code whose correctness does not depend on the model being right.

Required files per lab

Determinism rules

Platform code touches time, probability, money, and the network — all four are traps.

• Time is injected. Never time.time(), never datetime.now(). Every rate limiter, circuit breaker, TTL cache, token bucket, retry backoff, SLO window, and lease uses an injected now: Callable[[], float]. A platform component that reads the wall clock cannot be tested at a boundary, and a durable one that does is not replay-safe.
• Randomness is seeded. Any jitter, sampling decision, load-balancer choice, or chaos injection goes through an explicit random.Random(seed). Same seed → same bytes; tests assert it.
• Identifiers are derived, not generated. No uuid4(). Trace IDs, span IDs, task IDs, and idempotency keys are either passed in or derived deterministically (a counter, or a hash of the request). Real platforms use UUIDs; the lab uses a counter so the trace is diffable.
• The model is a pure function. The injected "LLM" maps its input to a scripted output with no hidden state, unless the lab is explicitly testing memory or session affinity.
• Money is integers where it can be. Token counts are integers; cost is computed in a smallest unit (micro-USD) and only formatted as a float, so accumulated cost is exact and tests can use ==. Where floats are unavoidable, compare with pytest.approx.

The test contract

import importlib, os
lab = importlib.import_module(os.environ.get("LAB_MODULE", "lab"))

Run both ways. The reference must be green; your lab.py goes green as you fill the TODOs:

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # the reference — must pass
python solution.py                          # the worked example

Test taxonomy

Every flagship lab covers all seven:

1. Happy path — the mechanism works on a well-formed input.
2. Malformed input — a tool call with wrong types, an unknown JSON-RPC method, a token with a bad signature, an ISO 20022 message missing a mandatory element. Result is a structured error, never a crash and never a silent pass.
3. Boundary — the off-by-one an interviewer probes: a quota at exactly the limit, a circuit breaker at exactly the threshold, a token expiring on the same tick, an empty registry, a cache at capacity, a burn rate at exactly 1.0.
4. Security — the case that makes it a bank platform: a tenant id read from the request body is ignored in favour of the token; an injected instruction does not escape the trust boundary; a delegation chain with a broken link is rejected; an expired SVID fails mTLS; a policy denies by default when no rule matches.
5. Invariants — properties that must hold for every input: the audit log's hash chain is unbroken; a saga either completes or fully compensates; idempotent replay returns the first result; the token bucket never goes negative; the error budget never exceeds 100%; the delegation chain length is bounded.
6. Failure & recovery — a provider times out and the fallback fires within budget; a downstream 5xx opens the breaker and half-open probes recover it; a partially applied change is detected as drift.
7. Determinism — same seed + same clock → byte-identical output.

Doc conventions

• WARMUP.md opens with a Table of Contents of working anchor links, kept in sync with the headings. mdBook lowercases, replaces spaces with hyphens, and strips punctuation to form anchors (an em dash inside a heading contributes nothing to the anchor).
• Every internal link to a chapter root uses index.md, never README.md. mdBook renames each chapter's README.md to index.html on disk, so an in-page link written foo/README.md gets a naive .md→.html swap and 404s. SUMMARY.md is the exception — it must reference the source name README.md.
• MathJax for math: \( … \) inline, $$ … $$ block.
• No bare angle brackets in prose — wrap <like-this> in backticks. This domain is full of them: <agent-id>, notifications/tools/list_changed, spiffe://<trust-domain>/…, generic types. mdBook will eat an unfenced one as HTML.
• Code fences are language-tagged. File references are relative links.

Definition of done

A lab is done when: LAB_MODULE=solution pytest is green, the learner lab.py passes once the TODOs are filled, python solution.py prints a sensible worked trace, the README's success criteria are testable, and "How this maps to the real stack" is accurate about the production system and honest about the miniature's limits.

A phase is done when all seven teaching docs exist, the WARMUP's ToC anchors resolve, and every lab in it is done.

The track is done when mdbook build ai-platform-architect exits 0, no in-page link 404s, and every lab passes under LAB_MODULE=solution.