MLOps & Platform Questions

The lifecycle-ownership loop. Each answer maps to Phases 01–03, 08–09, 12.

Q: What belongs in CI for an ML repo? Be specific about layers. Four layers (Phase 08): (1) code tests — plain pytest, every commit, seconds; (2) data validation — contracts on schemas/ranges/volumes at every ingestion, every pipeline run (a successful training run on garbage data is the worst outcome — it produces a plausible bad model); (3) training smoke test — tiny fixture dataset in CI, asserts loss decreases, pipeline completes, artifact round-trips serialization, <2 min, no GPU; (4) quality gates on real training — metric vs pinned baseline with significance, golden-prediction diff on a fixed probe set (catches "metric same, behavior different"), latency/size budgets. CD is then registry promotion + progressive rollout + auto-rollback — models ship like code, with statistical gates where code has assertions.

Q: Design a feature store. What problem is it actually solving? Two problems wearing one name: training-serving skew (one feature definition → both online and offline stores; never two codepaths) and point-in-time correctness (training joins must reconstruct what serving would have seen at each historical moment — the as-of join over validity intervals, Phase 03's lab). Architecture: definitions-as-code in a registry; batch + streaming materialization into (a) an offline store (columnar, full history, training frames) and (b) an online store (KV, latest values, single-digit-ms p99); a continuous parity checker sampling both. The deeper answer: "log-and-wait" (log served features, train on the logs) is the strongest skew defense and a legitimate alternative for request-scoped features — name when each wins.

Q: How does content-addressed caching make pipelines incremental — and what are its two preconditions? Cache key = H(code version ‖ params ‖ upstream artifact hashes). A change re-keys exactly its downstream closure — invalidation derives from the key structure, no invalidation logic exists; identical re-computed outputs give early cutoff (downstream stays cached). Preconditions: tasks are pure (declared inputs only — hidden state poisons the key) and deterministic (seeded — else identical keys produce different artifacts and the cache is incoherent). The classic homemade bug: omitting code identity from the key, serving stale artifacts after every refactor. (Phase 08's lab is this, tested.)

Q: Two training runs, identical config, different models. Debug order? (1) Data moved underneath — no snapshot pinning; check lineage first, it's the most common cause. (2) Unseeded randomness — init, shuffling, augmentation; audit the seed fan-out. (3) Environment drift — library versions; compare lock hashes. (4) Hardware nondeterminism (GPU atomics, cuDNN autotune) — real but small and last; people blame it first because it's blameless. The systemic fix: run records carrying data-hash + git SHA + lock-hash + seed (Phase 08 Ch. 9), so this question becomes a diff, not an investigation.

Q: Walk me through a model registry's state machine and why promotions are pointer moves. candidate → staging → production → archived, transitions carrying gates (offline quality gate into staging; shadow/canary verification into production; archive records why). Versions are immutable; "production" is an alias the serving fleet resolves — promotion and rollback are therefore pointer moves: instant, deployment-free, and reversible at 2am without a build. The registry is also where governance attaches (approvals, model cards — Phase 12): one artifact, one audit trail.

Q: Your nightly pipeline takes 6 hours and re-trains everything. The team wants a bigger cluster. Alternative? Measure first: how much of the 6 hours is recomputing unchanged work? Content-addressed caching typically converts nightly full-recompute into incremental runs (only changed partitions' downstream closure re-executes) — often a 5–10× wall-clock cut with zero new hardware. Then: per-task profiling (one pandas anti-pattern stage often dominates — Phase 01), parallelizing independent DAG branches, and only then hardware. The senior pattern: capacity requests follow profiling, not precede it.

Q: How do you roll out a new model when you can't A/B test (e.g., a single global pipeline)? Shadow first — full mirror, compare distributions/divergence/latency offline, weeks if the change is risky. Then time-sliced comparison (switchback if effects are short-lived) or geo-split if units exist at coarser granularity — with honest caveats about confounding vs a true A/B. Gate on the offline harness + golden diffs regardless. And state the real answer: if the decision matters enough, build the experimentation capability — most "can't A/B" situations are "haven't invested in assignment infrastructure."

Q: What observability do you wire up before a model's first production day? The five layers (Phase 12), minimally: system health (latency/error/QPS dashboards + SLO alerts); data quality (contracts on the serving inputs — null-rate and range alarms); drift (prediction-distribution PSI daily vs a pinned reference — the highest value per line of code; per-feature PSI weekly, importance-weighted); model quality (the delayed-label backfill join, scheduled from day one even though it fills in later); business KPI linkage (which dashboard does this model move — agreed with the PM in writing). Plus the retraining policy chosen before launch — "when do we retrain" decided by vibe post-incident is how drift gets laundered.

Q: A junior engineer asks why they can't just deploy from their notebook. Give the senior answer, not the bureaucratic one. Because production failures come from the parts the notebook doesn't have: nobody can re-run it (env unpinned, data un-snapshotted), nothing validates inputs when upstream changes next month, no gate stops a silent metric regression, no lineage answers "what made this prediction" during the incident, rollback means archaeology. Then the constructive half: the paved road makes the right way cheaper than the notebook (template repo, one-command pipeline, auto-wired monitoring) — platform work is making virtue the path of least resistance, not writing policy documents.