Interview Prep 02 — LLM Platform Engineer

Builds the internal LLM platform: gateways, routing, provider abstraction, governance, cost control — the layer other teams build on. Phases: 7, 8, 14.

1. Core concepts

  • Gateway components: provider adapters, model registry (model×provider), routing, fallback, streaming proxy, metering, caching, guardrails/PEP, key custody, audit (cheatsheet 13).
  • Routing strategies (cost/latency/quality/availability/policy).
  • Fallback, retries, circuit breakers, timeouts.
  • Key custody (real keys server-side; scoped internal keys) + BYOK.
  • Per-tenant metering, budgets, quotas; cost attribution.
  • Policy engine (PEP/PDP), fail-closed, residency routing.

2. System design questions

  1. Design an LLM gateway for 5 providers + self-hosted vLLM, multi-tenant. (Adapters, registry, routing, fallback, metering, key custody, observability, isolation.)
  2. Design model routing that protects gross margin. (Difficulty-based small↔premium, caching, cost-per-resolved-task.)
  3. Design residency-aware routing (EU data stays in EU). (Policy engine, fail-closed, region tags.)

system-design template; gateway diagrams.

3. Debugging questions

  • One provider's latency spikes; everything slows → per-provider timeouts + failover; circuit breaker.
  • Costs untraceable → add per-tenant/feature/model cost attribution (diagrams/16).
  • A tenant's traffic starves others → per-tenant rate limits/quotas (noisy-neighbor).
  • Retry storm amplifies an outage → backoff + circuit breaker + retry budget.

4. Coding exercises

  • Build a minimal gateway: 2 provider adapters + a model registry + routing + fallback + a usage meter (labs/lab-08).
  • Implement a streaming proxy that passes SSE through transparently.
  • Implement a token-bucket per-tenant rate limiter.

5. Architecture exercises

  • Draw the OpenRouter-style and LiteLLM-style gateways; contrast aggregator vs self-hosted proxy.
  • Design the model registry schema (model × provider × price × limits × capabilities).

6. Model selection questions

  • How does the platform expose model choice to app teams without chaos? (curated registry + routing policies, not raw provider keys).
  • How do you keep the registry's prices/limits current? (refresh job; cheatsheet 03).

7. Production incident questions

  • Provider outage → fail over; was the circuit breaker tuned? (template 04).
  • A leaked internal key → revoke (scoped, instant), rotate, audit usage (Phase 14.03).
  • Cross-tenant data in a shared cache → tenant in cache key; purge; test (Phase 14.04).

8. Portfolio project

A working LLM gateway: provider adapters (≥2) + model registry + routing (cost/latency) + fallback + streaming proxy + per-tenant usage metering + a simple admin/cost view. README with routing/fallback demos and a cost report.

9. 30-minute drill

  • 5 min: name all gateway components + why a gateway at all.
  • 10 min: design multi-provider routing + fallback (whiteboard).
  • 10 min: debug a provider outage + a cost-attribution gap.
  • 5 min: key custody + BYOK + per-tenant isolation.

10. 2-hour take-home

Build a mini gateway with 2 providers (one can be a mock/local), a registry, cost-based routing with fallback, SSE streaming passthrough, and per-key usage/cost metering. Writeup: routing policy, failure handling, and how you'd add residency-aware policy.

11. Senior-level expectations

  • Junior: wraps one provider behind an internal API.
  • Mid: multi-provider routing + fallback + metering.
  • Senior: designs the registry/policy/key-custody model, tunes circuit breakers and per-tenant quotas, attributes cost, and makes the platform a reliable product for app teams.
  • Principal: sets routing/governance/cost standards org-wide; designs for residency/compliance and vendor independence; the platform is a moat.

Next: 03 — AI Infrastructure Engineer · Index: interview-prep/