« All Roles

Senior ML Engineer — Production ML & GenAI Systems Curriculum

Target Roles:

• SAP — Senior ML Engineer, Production ML & Generative AI Systems (the reference JD — jd.md)
• Booking / Spotify / Zalando / Shopify — Senior/Staff ML Engineer, Ranking & Recommendations
• Uber / DoorDash / Careem — Senior ML Engineer, Marketplace ML (forecasting, pricing, matching)
• Databricks / Snowflake / AWS — ML Platform Engineer / Solutions Architect, ML
• Any enterprise AI team — Senior MLE owning data → features → models → serving → experimentation → governance end to end

Duration: 26 weeks core (6 months) — extendable to 12 months with capstone depth Seniority Target: Senior / Staff IC (7–9+ years equivalent depth) Goal: Make you the engineer who can own a production ML system end to end — data pipelines, feature stores, ranking/recsys/forecasting models, serving infrastructure, GenAI applications with guardrails, online experimentation, observability, and governance — and who can lead its technical direction.

Why This Curriculum Exists

The Senior ML Engineer role this curriculum targets is the full-lifecycle role: not the researcher (who hands off a model), not the pure platform engineer (who never owns a metric), but the engineer accountable for the system that moves a business number — from the Kafka topic to the A/B readout.

The JD makes the breadth explicit: data pipelines, feature stores, training workflows, model serving, online experimentation, observability — across deep learning, NLP, ranking, recommendation, forecasting, semantic retrieval, and LLM applications — plus MLOps, distributed systems, and evaluation and governance (offline benchmarking, online experimentation, hallucination analysis, model risk, responsible AI). Most engineers are deep in one or two of these. Senior means: deep in three or four, conversant and dangerous in all of them, and able to design the seams between them — because in production, the failures live at the seams (training-serving skew, eval-online metric divergence, drift nobody monitored).

This track is the seam-first curriculum. Every phase builds a working system component with tests, and every WARMUP teaches the failure modes that distinguish "it works in the notebook" from "it has run for a year."

How this track relates to the other tracks

Shared topics are cross-linked, never duplicated: this track's GenAI phase builds the production layer on the LLM track's RAG foundations; its serving phase builds the platform layer on the LLM track's inference mechanics.

What You Will Build

Every phase ships a tested, runnable artifact. By the end you will have:

1. A typed data-contract library — schema validation with composable rules and property-style tests (the production-Python entrance exam)
2. A mini stream processor — event-time windows, watermarks, late-data handling, idempotent exactly-once-effect aggregation
3. A mini feature store — point-in-time-correct joins (the leakage killer), online/offline parity checking, skew detection
4. A hybrid retrieval engine — BM25 + dense + reciprocal-rank fusion with a labeled-set evaluation harness
5. A learning-to-rank model from scratch — pairwise/LambdaRank-style training with NDCG implemented and tested by hand
6. An implicit-feedback recommender — matrix factorization with BPR, evaluated on recall@k/coverage against a popularity baseline
7. A forecasting backtesting harness — rolling-origin evaluation, seasonal-naive baselines, MASE/sMAPE, and a feature-based ML forecaster that must beat the baseline
8. A mini pipeline orchestrator — DAG execution with content-hash caching, retries, and an artifact ledger (the heart of every MLOps platform)
9. A model-serving simulator — dynamic micro-batching, latency SLO measurement (p50/p99), canary-vs-control comparison logic
10. A GenAI guardrail & evaluation harness — injection canaries, groundedness checks, refusal policy engine, structured-output validation
11. An online experimentation toolkit — power analysis, SRM detection, CUPED variance reduction, and a Thompson-sampling bandit
12. A drift & retraining engine — PSI/KS detectors, retraining policy state machine, and a model-risk report generator
13. Capstone platform projects — composing the above into end-to-end systems

Folder Structure

SeniorMLEngineer/
├── README.md                                   ← you are here
├── jd.md                                       ← target role profile (SAP Senior MLE)
├── phase-01-production-python-engineering/     ← typed, tested, profiled production Python
├── phase-02-data-engineering-streaming/        ← batch vs streaming, Kafka semantics, windows, watermarks
├── phase-03-feature-stores-training-serving-skew/ ← point-in-time correctness, online/offline parity
├── phase-04-semantic-retrieval-search/         ← BM25, dense, hybrid fusion, retrieval evaluation
├── phase-05-learning-to-rank/                  ← LTR objectives, NDCG, position bias
├── phase-06-recommendation-systems/            ← MF/BPR, two-tower, candidate-gen → rank architecture
├── phase-07-forecasting/                       ← baselines, backtesting, hierarchical & intermittent demand
├── phase-08-training-orchestration-mlops/      ← DAGs, caching, experiment tracking, registries, CI/CD for ML
├── phase-09-model-serving-platform/            ← serving patterns, batching, caching, canary, cost
├── phase-10-genai-production-rag-guardrails/   ← production RAG, guardrails, hallucination analysis
├── phase-11-online-experimentation/            ← A/B stats, SRM, CUPED, sequential tests, bandits
├── phase-12-observability-drift-governance/    ← monitoring, drift, retraining strategy, model risk, responsible AI
├── phase-13-capstone/                          ← end-to-end platform capstones
├── interview-prep/                             ← system design, fundamentals, MLOps, GenAI, behavioral
└── system-design/                              ← five full design walkthroughs

Each phase contains:

26-Week Schedule

Prerequisites

• Solid Python (3.10+) — this track sharpens it to expert level; it doesn't teach basics
• SQL fluency and comfort with pandas/numpy
• ML fundamentals: train/val/test discipline, overfitting, gradient descent (the CV track Phase 02 is the refresher if needed)
• Basic Docker and git
• For the GenAI phase: the LLM track Phase 07 WARMUP as background
• No paid cloud account needed: every lab runs locally (numpy/pandas/sklearn/torch), simulating the distributed component where necessary — the concepts are cloud-portable by design

Interview Relevance

This track maps directly to the four interview loops senior MLE candidates face:

• ML system design ("design a recommendation system / fraud detector / forecast pipeline") → phases 03–07 + system-design/
• ML fundamentals + coding → every flagship lab is a past interview question in disguise (implement NDCG; build an A/B significance checker; point-in-time join)
• MLOps/platform depth → phases 08–09, 11–12
• GenAI production judgment → phase 10 + the governance half of phase 12
• Behavioral at senior level → interview-prep/05: product judgment, ambiguity, leading without authority

About the job We help the world run better

At SAP, we keep it simple: you bring your best to us, and we'll bring out the best in you. We're builders touching over 20 industries and 80% of global commerce, and we need your unique talents to help shape what's next. The work is challenging – but it matters. You'll find a place where you can be yourself, prioritize your wellbeing, and truly belong. What's in it for you? Constant learning, skill growth, great benefits, and a team that wants you to grow and succeed.

What You’ll Build

In this role, you'll lead the design and delivery of production-grade ML and generative AI systems that solve complex product and business problems at scale. You'll own architecture and implementation across data pipelines, feature stores, training workflows, model-serving infrastructure, online experimentation, and observability.

You'll drive advanced use cases across deep learning, NLP, ranking, recommendation, forecasting, semantic retrieval, and LLM applications, including RAG, tool use, evaluation harnesses, and safety controls. Your work will directly shape performance, reliability, latency, and cost efficiency in live environments. You’ll guide technical direction on topics such as model selection, distributed training and inference, GPU utilization, model compression, prompt and retrieval optimization, drift detection, retraining strategy, and responsible AI controls. You’ll mentor other engineers and turn best practices into reusable patterns and platform capabilities.

What You Bring

You bring expert-level programming skills in Python, along with strong software engineering expertise in languages such as Java or Go, enabling you to build scalable, production-grade systems You have deep knowledge of machine learning, deep learning, and optimization techniques across structured data, NLP, search, ranking, and recommendation problems You have extensive experience designing and operating end-to-end ML systems, from data ingestion and experimentation to deployment, observability, and lifecycle management You bring strong hands-on experience with modern ML and LLM tooling (e.g., PyTorch, TensorFlow, scikit-learn), including fine-tuning, evaluation, orchestration, and model serving You have practical experience building generative AI applications using embeddings, vector databases, RAG pipelines, agent workflows, prompt engineering, and guardrails You bring deep expertise in MLOps and platform engineering, including model registries, feature stores, CI/CD, infrastructure as code, experiment tracking, and automated validation You have a strong architectural understanding of distributed systems, event-driven services, streaming data, and cloud-native ML platforms, with the ability to optimize for performance, scalability, reliability, and cost You define robust evaluation and governance strategies, including offline benchmarking, online experimentation, hallucination analysis, model risk assessment, and responsible AI practices

About You

You bring 7-9+ years of experience in software engineering and machine learning, with a track record of leading and delivering complex, production-scale AI systems You combine strong product judgment with technical depth, translating ambiguous business problems into scalable, high-impact AI solutions You thrive in complex, fast-moving environments and bring clarity, ownership, and strategic thinking to drive long-term platform success

Where You Belong

You will be part of a growing team of AI and industry experts dedicated to serving customers across the Kingdom of Saudi Arabia / United Arab Emirates. We are building a collaborative, high-impact environment that brings together deep AI expertise, strong industry knowledge, and regional understanding to help customers innovate, transform, and lead in their markets.

This team thrives on working together to turn complex business challenges into practical, scalable solutions. You will find an environment that values curiosity, ownership, and continuous learning, where new ideas are encouraged and real-world impact matters. This is a place for people who enjoy building something new. You will thrive here if you are customer-centric, motivated by meaningful outcomes, and comfortable navigating ambiguity in a fast-evolving AI landscape.

About The Team

We are a team of engineers passionate about building valuable, scalable AI frameworks that drive real business impact. We thrive on solving challenging problems, innovating

with cutting-edge technologies, and collaborating closely to deliver AI systems that are reliable, high-performing, and ready for production. Our team values ownership, technical excellence, and creating solutions that stand the test of scale and complexity.

Bring out your best

SAP innovations help more than four hundred thousand customers worldwide work together more efficiently and use business insight more effectively. Originally known for leadership in enterprise resource planning (ERP) software, SAP has evolved to become a market leader in end-to-end business application software and related services for database, analytics, intelligent technologies, and experience management. As a cloud company with two hundred million users and more than one hundred thousand employees worldwide, we are purpose-driven and future-focused, with a highly collaborative team ethic and commitment to personal development. Whether connecting global industries, people, or platforms, we help ensure every challenge gets the solution it deserves. At SAP, you can bring out your best.

We win with inclusion

SAP’s culture of inclusion, focus on health and well-being, and flexible working models help ensure that everyone – regardless of background – feels included and can run at their best. At SAP, we believe we are made stronger by the unique capabilities and qualities that each person brings to our company, and we invest in our employees to inspire confidence and help everyone realize their full potential. We ultimately believe in unleashing all talent and creating a better world.

SAP is committed to the values of Equal Employment Opportunity and provides accessibility accommodations to applicants with physical and/or mental disabilities. If you are interested in applying for employment with SAP and are in need of accommodation or special assistance to navigate our website or to complete your application, please send an e-mail with your request to Recruiting Operations Team: Careers@sap.com.

For SAP employees: Only permanent roles are eligible for the SAP Employee Referral Program, according to the eligibility rules set in the SAP Referral Policy. Specific conditions may apply for roles in Vocational Training.

Qualified applicants will receive consideration for employment without regard to their age, race, religion, national origin, ethnicity, gender (including pregnancy, childbirth, et al), sexual orientation, gender identity or expression, protected veteran status, or disability, in compliance with applicable federal, state, and local legal requirements.

Successful candidates might be required to undergo a background verification with an external vendor.

AI Usage in the Recruitment Process

For information on the responsible use of AI in our recruitment process, please refer to our Guidelines for Ethical Usage of AI in the Recruiting Process.

Please note that any violation of these guidelines may result in disqualification from the hiring process.

Requisition ID: 450218 | Work Area: Software-Design and Development | Expected Travel: 0 - 10% | Career Status: Professional | Employment Type: Regular Full Time | Additional Locations:

Phase 01 — Production Python Engineering

Difficulty: ⭐⭐⭐☆☆ Estimated Time: 2 weeks (30–40 hours) Roles Supported: every later phase — this is the engineering substrate the JD calls "expert-level programming skills in Python … enabling you to build scalable, production-grade systems"

Why This Phase Exists

Senior ML engineers are distinguished less by knowing more ML than by writing ML code that survives: typed interfaces other teams can build against, tests that catch regressions before the on-call does, and profiling instincts that find the real bottleneck. Every artifact in phases 02–13 is production-shaped Python; this phase sets the bar and the idioms once, so the rest of the track can assume them.

The flagship lab is a typed data-contract library — the kind of component that sits at every pipeline boundary in a real ML platform (and the reason tools like pydantic and pandera exist). Building one teaches the protocol/generic/dataclass machinery, error-design discipline, and property-style testing in one artifact.

Concepts

• Static typing as API design: Protocol (structural typing), generics (TypeVar, parameterized containers), dataclasses (frozen, slots), Enum, exhaustiveness
• Error design: exception hierarchies vs result types; collect-don't-raise validation; error messages as UX
• Iterators/generators as pipeline primitives; itertools; laziness and its traps
• Concurrency truthfully: the GIL, threads (I/O-bound) vs processes (CPU-bound) vs asyncio (many-waiting); what each buys for ML workloads
• Testing discipline: pytest fixtures/parametrize, property-style tests, golden files, test doubles at boundaries
• Profiling: cProfile/py-spy reading, the measure-first ethic
• Packaging and reproducibility: pyproject, pinned deps, src layout

Labs

Lab 01 — Typed Data-Contract Library (flagship, implemented)

→ Lab folder: lab-01-data-contracts/

Extension Project A — Concurrency Benchmark Harness (spec)

Measure the same workload (CPU-bound feature transform; I/O-bound URL fetch simulation) under threads / processes / asyncio; produce the table that demonstrates the GIL's consequences; write the one-page "which concurrency for which ML workload" memo.

Extension Project B — Profiling Kata (spec)

Take a deliberately slow data-prep script (pandas anti-patterns: row-wise apply, repeated concat, object dtypes); profile with cProfile + line-profiler; fix the top three; document each fix's mechanism and speedup. Target: 50×+ end-to-end.

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide for every concept above

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass
• Extension A table + memo (or equivalent from your own work)
• Extension B with before/after profiles saved

Warmup Guide — Production Python Engineering

Zero-to-expert primer for Phase 01: the Python engineering practices that make ML systems survive production — typing as design, error discipline, honest concurrency, testing that catches regressions, and profiling that finds the truth.

Table of Contents

• Chapter 1: What "Production-Grade" Means for ML Code
• Chapter 2: Typing as API Design
• Chapter 3: Dataclasses, Immutability, and Value Objects
• Chapter 4: Error Design — Collect, Don't Raise
• Chapter 5: Generators and Pipeline Composition
• Chapter 6: Concurrency Without Folklore
• Chapter 7: Testing Discipline
• Chapter 8: Profiling and the Measure-First Ethic
• Chapter 9: Packaging and Reproducibility
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What "Production-Grade" Means for ML Code

ML code has a specific pathology: it runs even when it's wrong. A web service with a type error crashes; an ML pipeline with a silent unit mismatch trains a worse model and tells no one. So production-grade ML code is defined by how loudly it fails:

1. Boundaries are guarded: every ingestion point validates its inputs against an explicit contract (the flagship lab) — because upstream will change a column's meaning without telling you.
2. Interfaces are typed: the type signature is the documentation that can't go stale, and mypy/pyright turn a class of runtime surprises into edit-time squiggles.
3. Behavior is pinned by tests: not "does it run" but "does it still compute the same NDCG on the golden file."
4. Performance claims are measured: the profiler, not intuition, names the bottleneck.

The rest of this guide is those four habits, mechanized. Everything later in the track (stream processors, feature stores, experiment analyzers) is written in this dialect.

Chapter 2: Typing as API Design

From zero: Python type hints are annotations (def f(x: int) -> str) checked by external tools (mypy, pyright), not the runtime. Their value compounds with codebase size and team size — exactly the senior-engineer regime.

The constructs that carry ML platform code:

• Protocol — structural typing: define what an object must do, not what it must inherit:

class Validator(Protocol):
    def validate(self, value: object) -> list[str]: ...

Anything with a matching validate method satisfies it — no inheritance, no registration. This is how you accept "anything model-like" (.predict()) or "anything rule-like" without coupling callers to your class hierarchy. Prefer Protocol over ABC at boundaries you don't control; ABCs still fit when you own the hierarchy and want shared behavior.

• Generics: class Result(Generic[T]) / def first(xs: Sequence[T]) -> T — containers and pipelines that preserve their element types end to end. An untyped pipeline returns Any and infects everything downstream; a generic one keeps IDE/checker leverage.
• Discriminating unions: int | None forces the None-check; Literal["mean", "sum"] makes string options checkable; assert_never() in the else branch makes enum matches exhaustive — adding a variant then fails the type check at every unhandled site (the same make-illegal-states-unrepresentable discipline the PMC track teaches for state machines).
• The pragmatic stance: type the public surface 100%, the internals as helpful; Any is a quarantine zone you shrink over time; never lie to the checker (cast/# type: ignore with a reason comment only).

Chapter 3: Dataclasses, Immutability, and Value Objects

@dataclass(frozen=True, slots=True)
class Field:
    name: str
    dtype: type
    required: bool = True

• @dataclass generates __init__/__repr__/__eq__ — value semantics for free.
• frozen=True: instances are immutable — hashable, safely shareable across threads, usable as dict keys, and immune to the spooky-mutation-at-a-distance bugs that plague config objects passed through five layers. Schemas, configs, and events should be frozen by default; accumulators and builders stay mutable.
• slots=True: no per-instance __dict__ — smaller, faster attribute access, and typo-proof (obj.requierd = ... raises instead of silently creating an attribute — a real bug class).
• The value-object pattern: parse/validate raw input once at the boundary into a frozen, typed object; everything downstream trusts it ("parse, don't validate" — the same principle, third track running). The flagship lab is this pattern as a library.

Chapter 4: Error Design — Collect, Don't Raise

The exception model serves control flow; data validation needs a different shape. If a record has five problems, raising on the first reports one and hides four — and in a batch of a million records, fail-on-first turns data triage into whack-a-mole.

The pattern (the lab's core):

@dataclass(frozen=True)
class ValidationIssue:
    path: str          # "user.age" — where
    rule: str          # "range(0, 150)" — what
    message: str       # human-readable why

@dataclass(frozen=True)
class ValidationReport:
    issues: tuple[ValidationIssue, ...]
    @property
    def ok(self) -> bool: return not self.issues

Rules: validators return issues (empty = pass), never raise for data problems; exceptions are reserved for programming errors (schema misconfigured, contract violated by the caller); paths make issues actionable at scale (group by path+rule and you have a data-quality dashboard — which is exactly what Phase 02's pipeline does with this library's output). Error messages are UX: include the offending value (truncated), the expectation, and the path — the person reading it is debugging at 2am with no context.

Chapter 5: Generators and Pipeline Composition

Data work is stream-shaped; generators are Python's native streams:

• A generator function (yield) produces lazily — constant memory over any input size, and composable: clean(parse(read(path))) builds a pipeline that pulls one record at a time through all stages.
• The traps that bite in production: single consumption (a consumed generator is silently empty — the "second epoch is empty" bug; materialize or re-create), late binding in closures (loop-variable capture), and exhaustion hiding errors (a generator that raises mid-stream has already yielded results — partial output without a clear failure marker; wrap stages with error channels when completeness matters).
• itertools literacy (islice, chain, groupby — sorted input!, tee — memory caveat) replaces a hundred hand-rolled loops.
• This chapter is why Phase 02's stream processor will feel natural: windows and watermarks are generator pipelines with timestamps.

Chapter 6: Concurrency Without Folklore

The truths, stated plainly:

• The GIL: one thread executes Python bytecode at a time per process. Threads therefore do not speed up CPU-bound pure-Python code — at all. They do speed up I/O-bound work (the GIL releases during blocking I/O) and C-extension work that releases it (numpy ops, many model runtimes).
• The decision table:

• asyncio in one paragraph: cooperative single-threaded concurrency — await yields control to the event loop; anything that blocks without awaiting freezes everything (the classic bug: a synchronous model call inside an async handler — Phase 09 and the CV track's serving phase both hit this). It buys massive I/O concurrency, not parallelism.
• Process pools' fine print: arguments and results are pickled (cost and pickleability constraints); prefer chunked work over per-item tasks; on Linux fork vs spawn semantics differ (copied state surprises).
• The senior reflex: name the bound (CPU? I/O? memory bandwidth?) before choosing the tool — the same diagnosis-first ethic as every performance chapter in this curriculum.

Chapter 7: Testing Discipline

The testing stack for ML systems code, in order of leverage:

1. Example tests (pytest, parametrize): the bread and butter; table-driven cases keep them dense and readable.
2. Property-style tests: instead of hand-picked cases, assert invariants over generated inputs — round-trips (decode(encode(x)) == x), idempotence (normalize(normalize(x)) == normalize(x)), oracle equivalence (optimized vs naive implementation on random inputs — the differential-testing pattern used throughout this curriculum's labs). The hypothesis library industrializes this; a seeded loop over random inputs gets you 80% with zero dependencies (the lab does the latter).
3. Golden tests: freeze a known-good output artifact (a validation report, a metric value); fail on any diff. Cheap regression armor for numeric code — with the discipline that golden updates are reviewed, not reflexive.
4. Fixtures for shared setup; test doubles only at true boundaries (network, clock, randomness — inject the clock and the RNG; code that calls time.time()/random() inline is untestable by construction, a lesson Phases 02 and 11 enforce structurally).
5. The ML-specific layer — data validation as tests, model-quality gates, golden-prediction files — arrives in Phase 08; it stands on this chapter.

Chapter 8: Profiling and the Measure-First Ethic

• cProfile (python -m cProfile -s cumtime script.py): deterministic, whole-program, function granularity. Read cumtime for "where does time go," tottime for "who is doing the work."
• py-spy (py-spy top --pid / record for flamegraphs): sampling, attachable to running production processes, near-zero overhead — the production instrument.
• line_profiler for the one hot function cProfile names.
• The pandas/numpy reality: most data-code slowness is one of four shapes — row-wise apply where a vectorized op exists, repeated concat in a loop (quadratic), object dtypes where native ones fit, or accidental copies of large frames. Extension B is a guided hunt for exactly these.
• The ethic (this curriculum's most repeated sentence): profile before optimizing; re-measure after. An optimization without a before/after number is a rumor.

Chapter 9: Packaging and Reproducibility

The minimum that makes your work installable and re-runnable:

• pyproject.toml + src layout (src/contracts/): the import-path hygiene that prevents "works from the repo root only."
• Pinned environments: a lock file (pip-tools/uv/poetry — pick one) for applications; looser ranges for libraries. "It worked last month" bugs are unpinned-dependency bugs ~half the time (the CV track's Pillow-resize incident is the canonical ML example).
• Determinism hooks: seed everything seedable (random, numpy, torch) through one function; surface the seed in logs/configs. Reproducibility is a feature you build, not a property you hope for — and it's a prerequisite for Phase 11's experimentation discipline.

Lab Walkthrough Guidance

Lab 01 — Typed Data-Contract Library, suggested order:

1. Value objects first: Field, ValidationIssue, ValidationReport (frozen, slotted). Get report.ok and issue aggregation right before any rules exist.
2. Built-in rules as small classes satisfying the Rule protocol — type check, range, regex, enum, nullability. Each returns a list of issues (Ch. 4).
3. Schema.validate(record) — walk fields, run rules, aggregate with paths; unknown-key and missing-key policies explicit.
4. Coercion ("42" → 42 where the schema says int, with issues on failure) — note how coercion-then-validation ordering is a design decision; document yours.
5. Composition: schema.extend(...), schema.subset(...) — frozen objects make these return new schemas naturally.
6. The property-style loop: generate random valid records from the schema, assert they validate clean; mutate one field at a time, assert exactly one issue at the right path. This test finds rule bugs nothing else does.

Success Criteria

You are ready for Phase 02 when you can, from memory:

1. Argue Protocol vs ABC and write a Protocol from scratch.
2. Justify frozen+slots dataclasses for configs/schemas with two concrete bug classes they prevent.
3. State the collect-don't-raise pattern and where exceptions still belong.
4. Recite the concurrency decision table and explain the GIL's exact consequence.
5. Name the four pandas slowness shapes and the profiler that finds each.
6. Write a property-style test (round-trip or oracle) without reference.

Interview Q&A

Q: Your feature pipeline silently produced garbage for three days. What engineering practices would have caught it in three minutes? Boundary contracts (schema validation at ingestion — unknown column, dtype drift, and range violations all fire on record one), golden tests on the pipeline's output statistics, and typed interfaces so the upstream rename couldn't have silently mapped to Any. The meta-answer: ML code runs-when-wrong, so the discipline is making wrongness loud — validation, types, goldens, in that order of leverage.

Q: Why doesn't threading speed up your pure-Python feature transform, and what are your actual options? The GIL serializes Python bytecode across threads — CPU-bound pure Python gains nothing. Options: vectorize into numpy/pandas (the usual 10–100× answer — the GIL releases inside C), ProcessPoolExecutor with chunked work (true parallelism, pickling costs), or move the hot loop to a compiled layer (numba/Cython). And measure first — if the transform is actually I/O-bound on reads, threads do help and the question was a trap.

Q: How do you test code that depends on time, randomness, and a database? Inject all three: a clock object (or freezegun-style patching) instead of time.time(), a seeded random.Random/numpy.random.Generator passed in, and a repository interface with an in-memory fake at the boundary. The design rule that makes testing possible is dependency injection of nondeterminism — code that reaches for global time/randomness is untestable by construction, which is why the track's later labs (stream windows, bandits) all take clocks and RNGs as parameters.

References

• mypy docs and typing module docs — Protocol, generics, Literal
• Fluent Python (Ramalho, 2nd ed.) — ch. 5 (dataclasses), 17–18 (iterators/context), 19–21 (concurrency)
• pytest docs — fixtures, parametrize; Hypothesis docs — when you graduate from hand-rolled property loops
• Alexis King, Parse, don't validate — the boundary-pattern essay
• py-spy and cProfile docs
• pydantic and pandera — the industrial versions of the flagship lab; read their docs after building yours
• Brett Slatkin, Effective Python (2nd ed.) — items on generators, concurrency, and testing

Lab 01 — Typed Data-Contract Library

Phase: 01 — Production Python Engineering | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

Every production ML pipeline has boundaries where data arrives from systems you don't control. This lab builds the boundary guard — a miniature pydantic/pandera — and in doing so drills the production-Python toolkit: protocols, frozen dataclasses, collect-don't-raise error design, and property-style testing.

What you build

• ValidationIssue / ValidationReport — frozen value objects; issues carry path + rule + message so a million-record batch aggregates into a dashboard
• Built-in rules — TypeRule (with the bool-is-not-int trap), RangeRule, RegexRule (fullmatch!), EnumRule — each a frozen dataclass satisfying a structural Rule protocol
• Schema.validate — collects every issue: unknown fields, missing required, nullability, coercion failures, rule failures
• Coercion — "42" → 42 where declared, failures as issues, never exceptions
• Composition — extend / subset returning new schemas (immutability pays off)

Key concepts

Files

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # the reference

Success criteria

• All 15 tests pass — especially test_property_single_field_mutation_localizes_issue (mutating exactly one field must produce issues at exactly that path)
• You can explain why validate returns a report while extend raises — the data-error vs programmer-error line, drawn correctly
• You can write a new rule (e.g., LengthRule) in under two minutes without touching the library

Extensions

• Nested schemas (Field(dtype=Schema)) with dotted paths (user.address.zip)
• A pandas adapter: validate a DataFrame column-wise, returning issue counts per rule (the data-quality dashboard primitive Phase 02 uses)
• Compare your design to pydantic v2's — what does compiled-core validation buy, and what did you not need?

References

• pydantic v2 docs and pandera — the industrial versions
• Alexis King, Parse, don't validate
• WARMUP.md chapters 2–4 — the design rationale in full

Phase 02 — Data Engineering & Streaming

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "data pipelines … event-driven services, streaming data" — the layer every ML system stands on

Why This Phase Exists

Models are downstream of data, and senior MLEs are accountable for the data system: where events come from, what ordering and delivery guarantees they carry, how batch and streaming views reconcile, and how quality is enforced before features are computed. The hardest production bugs in ML (silent feature corruption, double-counted events, training on data that arrived late) are streaming-semantics bugs — so this phase builds a stream processor by hand, because watermarks and exactly-once-effect only become real when you've implemented them.

Concepts

• Batch vs streaming as views of the same log; the Kafka mental model (topics, partitions, offsets, consumer groups, keys → ordering)
• Event time vs processing time — the distinction everything else hangs on
• Windows: tumbling, sliding, session; watermarks and allowed lateness
• Delivery semantics: at-most/at-least/exactly-once-effect (idempotency + dedup)
• Schemas as contracts on topics (the Phase 01 library, applied); schema evolution
• Data quality gates and quarantine paths; backfills and reprocessing

Labs

Lab 01 — Mini Stream Processor (flagship, implemented)

→ Lab folder: lab-01-stream-processor/

Extension Project A — Kafka Hands-On (spec)

Run single-node Kafka (docker); produce keyed events; demonstrate per-key ordering, consumer-group rebalancing, and replay-from-offset; wire your Phase 01 contracts as a validating consumer with a quarantine topic.

Extension Project B — Backfill Design Memo (spec)

Design (1 page + diagram) the reprocessing story for your lab's aggregates: a bug in the aggregation logic shipped for a week — how do you recompute without double-counting, what's versioned, and how do consumers cut over?

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 13 tests pass
• Extension A demo script + notes (or equivalent experience)
• Extension B memo

Warmup Guide — Data Engineering & Streaming

Zero-to-expert primer for Phase 02: logs and streams as the substrate of ML systems — Kafka's model, event time and watermarks, delivery semantics, and the data-quality machinery that keeps features trustworthy.

Table of Contents

• Chapter 1: The Log — One Abstraction Under Everything
• Chapter 2: The Kafka Model in Working Detail
• Chapter 3: Event Time vs Processing Time
• Chapter 4: Windows
• Chapter 5: Watermarks and Lateness
• Chapter 6: Delivery Semantics — What Exactly-Once Actually Means
• Chapter 7: Schemas, Contracts, and Evolution on Streams
• Chapter 8: Data Quality, Quarantine, and Backfills
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Log — One Abstraction Under Everything

From zero: an append-only, ordered sequence of records, each addressed by an offset. That's the whole abstraction — and it unifies the field: Kafka topics are logs; database replication streams are logs; a data warehouse table is a log compacted by key; "batch" is reading a bounded slice of a log, "streaming" is reading its unbounded tail. The senior framing (Kleppmann's): batch and streaming are not two systems but two consumption modes of one log — which is why architectures that treat them as separate pipelines (the old Lambda architecture's dual codebases) rot, and why "Kappa-style" (one streaming pipeline, replay the log for backfills) won.

For ML specifically, the log is the source of truth for training data: features and labels are derived views, and every "we can't reproduce last month's training set" incident traces to deriving views without retaining or versioning the log positions they came from. Hold that; Phase 03 builds on it directly.

Chapter 2: The Kafka Model in Working Detail

The vocabulary you must own (transferable to Pulsar/Kinesis/PubSub — names differ, model rhymes):

• Topic = a named log, split into partitions for parallelism. Ordering is guaranteed within a partition only — total order across a topic does not exist at scale, by design.
• Keys: a record's key hashes to a partition (the PMC track's partitioner lab is literally this code) — so per-key ordering is the guarantee you engineer with: all events for user_42 land in one partition, in order. Choosing the key is choosing your ordering and your skew (hot keys → hot partitions).
• Offsets + consumer groups: each group tracks its position per partition; partitions are divided among the group's consumers (rebalancing on join/leave). Replay = rewind offsets — the property that makes reprocessing and backfills possible at all.
• Retention/compaction: time/size-based deletion, or log compaction (keep latest per key — a changelog that doubles as a table).
• Producer acks (acks=all + min ISR — durability), idempotent producers, and transactions exist and matter — Chapter 6 places them.

Chapter 3: Event Time vs Processing Time

The distinction the entire phase hangs on:

• Event time: when the thing happened (timestamp in the record, set at origin).
• Processing time: when your system sees it.

They diverge constantly: mobile clients buffer offline for hours; a partition lags; a backfill replays last week at full speed (processing time = now, event time = last week). Any computation defined in processing time changes its answer depending on infrastructure weather — "orders per hour" computed by arrival time double-counts during a replay and undercounts during an outage.

ML's stake: features like "purchases in the last 24h" are event-time statements; computing them in processing time creates training data that doesn't match reality and — worse — training/serving skew when the serving path's timing differs from the batch path's (Phase 03's central topic). The rule: aggregate in event time; use processing time only for operational metrics about the pipeline itself.

Chapter 4: Windows

Unbounded streams need bounded questions. The window taxonomy:

• Tumbling: fixed-size, non-overlapping ([12:00, 12:05), [12:05, 12:10)). Assignment is arithmetic: start = ts - (ts % size). The default for counts/sums.
• Sliding/hopping: fixed size, overlapping starts (size 1h, hop 5min) — one event belongs to multiple windows; smoother but size/hop × the state.
• Session: gap-based (a window closes after N min of silence per key) — dynamic boundaries, merge logic when a late event bridges two sessions; the natural shape for user behavior features.

Window state is the cost center: every open window per key holds an accumulator; state size = keys × windows-in-flight — the reason streaming engines obsess over state backends and why your lab keeps explicit window lifecycle (open → closeable → expired) rather than "keep everything forever."

Chapter 5: Watermarks and Lateness

The hard question: with out-of-order arrival, when is a window done? Wait forever = no results; close at the boundary = drop every straggler.

The watermark is the answer: a monotonic estimate flowing with the stream that asserts "no events with timestamp ≤ W are still coming." Common construction: W = max_event_time_seen − bounded_delay (the heuristic: we believe disorder is bounded by, say, 2 minutes). When W passes a window's end, the window emits.

Lateness policy completes the design:

• Events older than W but within allowed lateness of their window: the window re-fires with an updated (corrected) result — consumers must tolerate updates.
• Events beyond allowed lateness: routed to a side output (late-data topic) — never silently dropped; late data is information (about client clocks, pipeline health) and sometimes must be reconciled in batch.

The trade is explicit and product-owned: delay (larger watermark lag = more complete first results) vs completeness vs state size (longer lateness = more retained windows). Saying that sentence with numbers is the difference between using Flink and understanding it. The lab implements exactly this machinery, minus the distribution.

Chapter 6: Delivery Semantics — What Exactly-Once Actually Means

The honest ladder:

• At-most-once: fire and forget — losses on failure. Rarely acceptable.
• At-least-once: retry until acked — duplicates on failure. The practical default; the system must then handle duplicates.
• "Exactly-once" is precisely: at-least-once delivery + idempotent or transactional processing = exactly-once effect. Nothing transmits each message exactly once over a faulty network (the Two Generals say hi); what's engineerable is that retries don't change the result.

The two implementation patterns:

1. Idempotency: processing is a no-op the second time — dedup by a stable event id (the lab's approach: a seen-ids set with TTL), or naturally idempotent sinks (upsert by key, set-union accumulators).
2. Transactions: atomically commit (output + consumer offset) together — Kafka's transactional producer/read_committed consumers, or the classic outbox pattern at service boundaries.

The interview-grade summary: "exactly-once is a property of the pipeline's effect on state, purchased with idempotency or atomic offset+output commits — not a network guarantee."

Chapter 7: Schemas, Contracts, and Evolution on Streams

A topic without a schema is an outage on a delay timer. The machinery:

• Schema registry pattern: producers register schemas (Avro/Protobuf/JSON-schema); consumers resolve by id; the registry enforces compatibility rules on evolution — backward (new readers read old data), forward (old readers read new data), full. This is the PMC track's compatibility discipline (its Phase 11 skew matrix applies verbatim to topic evolution: writers and readers upgrade at different times, always).
• Safe evolution: add optional fields with defaults; never repurpose/rename in place; deprecate then remove over announced versions. Breaking change = new topic (the major-version analog).
• Where Phase 01's library plugs in: contract validation at the consumer edge — schema-valid but semantically wrong records (negative prices, impossible ages) go to quarantine (Ch. 8), with the path+rule aggregation becoming your data-quality metrics.

Chapter 8: Data Quality, Quarantine, and Backfills

• The quarantine pattern: validation failures route to a dead-letter/quarantine topic with the reason attached — the pipeline keeps flowing, the bad data is preserved for diagnosis, and quarantine rate becomes the alert (a step change in rejects = upstream broke something; you found out in minutes, not at model-metric time — the entire point).
• Quality dimensions to monitor per stream: volume (events/min vs historical), schema-validity rate, semantic-validity rate (per rule!), freshness (event-time lag), key cardinality drift. These are Phase 12's drift detectors aimed at inputs — same math, earlier in the pipe, cheaper to act on.
• Backfills/reprocessing (Extension B's subject): the log makes recomputation possible; correctness requires versioned outputs (write agg_v2 alongside agg_v1, cut consumers over, then retire), deterministic logic (event-time windows — Ch. 3's payoff: a replay produces identical results because nothing depends on arrival time), and idempotent sinks (Ch. 6). The test of a well-built pipeline is literally "can you replay it" — your lab's order-independence test is the miniature of this property.

Lab Walkthrough Guidance

Lab 01 — Mini Stream Processor, suggested order:

1. Value objects: Event(key, ts, value, event_id), WindowKey(key, start, end), WindowResult — frozen (Phase 01 habits).
2. Window assignment (tumbling arithmetic) — unit-test boundaries (ts exactly at a window edge belongs to the next window: [start, end)).
3. The processor loop without lateness: ingest → update accumulator; advance watermark (max_ts − delay); emit windows whose end ≤ watermark.
4. Allowed lateness: emitted windows stay in state until end + allowed_lateness ≤ watermark; a late-but-allowed event re-emits a corrected result (mark it as an update). Then: too-late events → side output.
5. Dedup: event_id seen-set consulted before accumulation — then run the duplicate-delivery test.
6. The big test (write it before the code if you dare): shuffle a fixed event set (bounded disorder), process, and assert identical final results to in-order processing — order-independence is the property everything above exists to buy.

Success Criteria

You are ready for Phase 03 when you can, from memory:

1. Explain the log abstraction and why batch/streaming are consumption modes of it.
2. State Kafka's ordering guarantee precisely and what key choice controls.
3. Give an ML-relevant example where event-time vs processing-time changes the answer.
4. Define watermark, allowed lateness, and side output, and articulate the delay/completeness/state triangle.
5. Define exactly-once-effect and both purchase methods (idempotency, transactional offset+output).
6. Describe the quarantine pattern and name four stream-quality metrics.

Interview Q&A

Q: Your "orders per hour" feature doubled during a pipeline replay. What went wrong and how do you fix it permanently? The aggregation was keyed by processing time (arrival), so replayed events counted in today's windows — and possibly counted twice without dedup. Fix: event-time windows (replay lands in the original windows), idempotent accumulation by event id, and versioned output for the backfill cutover. The follow-up trap is "just don't replay" — wrong: replayability is the system's most valuable property; the pipeline was at fault, not the replay.

Q: Design the lateness policy for a fraud-features stream. What questions do you ask first? Product questions, not tech ones: what's the cost of an incomplete-but-fast feature vs a complete-but-slow one (fraud scoring wants fast — small watermark delay, accept corrections); can downstream consume updates (re-emission) or only finals; what's the actual disorder distribution (measure event-time lag percentiles before choosing the delay); and what happens to too-late events (for fraud: they're a signal in themselves — route to investigation, never drop). Policy = numbers chosen against those answers, monitored thereafter.

Q: The team says "Kafka gives us exactly-once, so we're fine." Correct them precisely. Kafka's idempotent producer gives exactly-once append per partition under retries; transactions give atomic offset+output commits within Kafka. The moment your consumer writes to an external system (a feature store, a DB), you're back to designing idempotency yourself — upserts, dedup keys, or an outbox. "Exactly-once" is an end-to-end property of effects, re-established at every boundary — and the boundary your ML pipeline cares about (the feature store) is exactly where the slogan stops applying on its own.

References

• Kleppmann, Designing Data-Intensive Applications — ch. 11 (streams) is this phase's textbook; ch. 4 (encoding/evolution) for Ch. 7
• Kreps, The Log: What every software engineer should know (2013) — the Chapter 1 essay
• Akidau et al., The Dataflow Model (VLDB 2015) — windows/watermarks/lateness, from the source; or Streaming Systems (Akidau, Chernyak, Lax) for the book form
• Kafka documentation: design section — delivery semantics, transactions
• Flink docs: event time and watermarks — the production implementation of your lab
• Confluent Schema Registry docs — compatibility rules
• PMC track Phase 07 WARMUP (partitioning) and Phase 11 (the skew matrix — applies to schema evolution verbatim)

Lab 01 — Mini Stream Processor

Phase: 02 — Data Engineering & Streaming | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Watermarks, allowed lateness, and exactly-once-effect are folklore until you've implemented them. This lab builds the core of Flink's event-time machinery in ~150 lines — and proves it with the property that justifies the whole design: shuffled delivery produces identical results to ordered delivery.

What you build

• assign_window — tumbling-window arithmetic ([start, end) — boundary events belong to the next window; a test enforces it)
• StreamProcessor — per-key window accumulators with the full lifecycle: open → closeable (watermark passes the end: emit once) → retained for allowed lateness (late events re-emit corrected is_update=True results) → expired (further events go to the side output, never silently dropped)
• Dedup by event_id — at-least-once delivery in, exactly-once effect out
• flush — bounded-input end-of-stream semantics

Key concepts

Files

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 13 tests pass — test_order_independence_under_bounded_shuffle and test_duplicates_under_shuffle_still_idempotent are the ones that certify understanding
• You can explain why watermark_delay and allowed_lateness are different knobs (tolerated disorder before first emission vs tolerated corrections after)
• You can state what the consumer of this stream must handle (updates! late topic!)

Extensions

• Sliding windows (one event → multiple windows) and session windows (gap-based, with merge-on-late-bridge — significantly harder; do it to feel why)
• Bound seen_ids with a TTL keyed to the watermark (real systems can't keep every id forever — what does expiring dedup state cost in guarantees?)
• Replace the dict state with an interface and add a "spill to disk" implementation — the state-backend abstraction of real engines

References

• Akidau et al., The Dataflow Model (VLDB 2015) — this lab is its §2–3
• Flink: Timely Stream Processing concepts
• WARMUP.md chapters 3–6

Phase 03 — Feature Stores & Training-Serving Skew

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "feature stores" named capability — and the home of ML's most expensive bug class (leakage and skew)

Why This Phase Exists

Two failure modes silently destroy more production ML value than any modeling choice:

1. Temporal leakage: training features computed with information that wasn't available at prediction time — offline metrics soar, production disappoints, and nobody knows why for months.
2. Training-serving skew: the offline feature pipeline and the online one compute almost the same thing — different defaults, different windows, different staleness — and the model meets features at serving time that it never saw in training.

The feature store is the architectural answer to both: one feature definition, point-in-time-correct offline materialization for training, and a low-latency online store for serving — with parity between them measured, not assumed. The flagship lab builds all three pieces.

Concepts

• Feature definitions as code; entities, feature views, timestamps (event vs created)
• Point-in-time (as-of) joins — the leakage killer, in exact semantics
• Offline store (training materialization) vs online store (serving lookup); freshness/staleness budgets
• Training-serving skew taxonomy: logic skew, data skew, temporal skew
• Online/offline parity testing and skew detection (the lab implements both)
• Feature freshness SLOs, TTLs, defaults-on-miss (and their skew implications)
• Build-vs-adopt landscape: Feast/Tecton/SageMaker FS — what they actually provide

Labs

Lab 01 — Mini Feature Store (flagship, implemented)

→ Lab folder: lab-01-feature-store/

Extension Project A — Feast Hands-On (spec)

Define the lab's feature views in Feast (local provider); materialize, fetch historical features and online features; compare its as-of semantics and TTL handling to yours; write the one-page "what Feast gives you / what you still own" memo.

Extension Project B — Skew Post-Mortem (spec)

Write the post-mortem (PMC-track template) of a realistic skew incident: offline imputed missing days_since_last_order with the median, online defaulted to 0. Reconstruct the metric impact mechanism, the detection that should have existed, and the parity test that prevents recurrence.

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 12 tests pass
• Extension A memo (or equivalent Feast/Tecton experience)
• Extension B post-mortem

Warmup Guide — Feature Stores & Training-Serving Skew

Zero-to-expert primer for Phase 03: why feature platforms exist, the exact semantics of point-in-time correctness, the taxonomy of training-serving skew, and the parity machinery that keeps offline and online worlds honest.

Table of Contents

• Chapter 1: The Problem Feature Stores Solve
• Chapter 2: The Anatomy — Definitions, Offline, Online
• Chapter 3: Temporal Leakage — the Expensive Bug
• Chapter 4: Point-in-Time Joins, Exactly
• Chapter 5: The Online Store — Freshness, TTL, Defaults
• Chapter 6: Training-Serving Skew — a Taxonomy
• Chapter 7: Parity Testing and Skew Detection
• Chapter 8: Build vs Adopt, and Operating Realities
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Problem Feature Stores Solve

From zero: a feature is a model input computed from raw data — "user's order count, last 30 days." Without shared infrastructure, every team computes features twice: once in a batch job for training, once in service code for serving. The twice is the disease:

• The two implementations drift (different languages, different engineers, different bug fixes) → skew (Ch. 6).
• Training-set construction re-solves the hardest problem — what was this feature's value at each historical label's moment? — per project, usually wrongly (Ch. 3).
• Features aren't discoverable or reusable; the same "30-day order count" exists five times with five definitions.

A feature store is the deduplication of this work: define once (feature view as code), materialize twice (offline history for training, online latest for serving), guarantee the seams (point-in-time correctness offline, freshness online, parity between them). The JD names it because owning this layer is what "end-to-end ML systems" means in practice.

Chapter 2: The Anatomy — Definitions, Offline, Online

The standard architecture (Feast/Tecton/SageMaker all rhyme):

• Entity: the join key — user_id, (user_id, merchant_id).
• Feature view: a named set of features over an entity, sourced from a timestamped table/stream, with metadata: event timestamp (when the fact became true — Phase 02's event time), optional TTL (how long a value stays valid), owner, version.
• Offline store: the full timestamped history (warehouse/lake) — the substrate for point-in-time training joins.
• Online store: key→latest-value (Redis/DynamoDB-class) — single-digit-ms reads at serving time, populated by a materialization job (batch or streaming from Phase 02's pipelines).
• The two consumption APIs that define the contract: get_historical_features(entity_df_with_timestamps) → training frame (PIT-correct), and get_online_features(entity_keys) → serving vector (fresh-as-materialized).

Everything else (registries, UIs, monitoring) decorates these five pieces — and the lab builds exactly the five.

Chapter 3: Temporal Leakage — the Expensive Bug

The setup: training data is constructed retrospectively. Each row is (entity, label_timestamp, label) and needs feature values. The naive join — "latest feature value per entity" — uses values computed after the label event: information from the future.

Why it's so damaging: the leaked feature is often causally downstream of the label ("orders_30d" computed after the churn date encodes the churn), so offline metrics inflate spectacularly; serving can't access the future, so production delivers the un-inflated reality; and the gap reads as "model degradation," sending teams chasing drift for months. It is the CV track's leakage chapter (Phase 02) in its most virulent form — every feature is a potential leak vector because every feature has a timestamp.

The discipline: training joins are as-of joins (Ch. 4), always; and the test suite contains an adversarial case — a feature table constructed so that a naive join produces detectably different output than a correct one (the lab's test_leakage_adversarial). A feature pipeline without that test is unverified.

Chapter 4: Point-in-Time Joins, Exactly

The semantics, precisely (the lab implements these to the letter):

For each entity row with label timestamp $t$, the joined feature value is the one with the largest event timestamp $\tau$ such that $\tau \le t$ and $t - \tau \le \text{TTL}$ — else null/default.

The decisions hiding in that sentence (each is a real design choice — defend yours):

• ≤ vs <: can a feature whose event timestamp equals the label instant be used? If the "feature event" is the same business event that generates the label (an order that updates order_count and also triggers the label), ≤ self-leaks. Common resolution: ≤ but with feature timestamps set to availability time (when the value was queryable), not occurrence time — which also absorbs pipeline delay honestly. Know both conventions; state which you use.
• TTL: without it, a 4-year-old value joins happily ("most recent" ≠ "relevant"). TTL turns staleness into explicit missingness — which the model then handles by a declared default (Ch. 5's skew tie-in).
• Created vs event time: if values can be backfilled/corrected, the join must use what was known at the time (event time + created time both ≤ t) to reproduce the serving view truthfully — the bitemporal wrinkle; at minimum know it exists.
• Implementation shapes: sort-merge per entity (pandas merge_asof — the lab's reference oracle), or point-in-time window functions in SQL (ROW_NUMBER() OVER (PARTITION BY entity ORDER BY τ DESC) filtered to τ ≤ t).

Chapter 5: The Online Store — Freshness, TTL, Defaults

The serving side's physics:

• Freshness = now − event_timestamp of the served value. Its budget is a product decision per feature ("30-day order count" tolerates hours; "items in cart now" tolerates seconds → streaming materialization, Phase 02).
• The materialization job is a lag source: batch-hourly materialization means online values are 0–60+ min stale — and training data joined as-of exactly will be systematically fresher than serving reality. The honest fix: train against the feature values as serving would have seen them (lag-adjusted as-of joins) or accept and measure the gap. This subtlety is senior-interview gold.
• Miss policy: key absent (new user) or TTL-expired → serve what? The default must be the same value the training pipeline used for missingness — the median-vs-zero mismatch is the classic skew incident (Extension B). Defaults are part of the feature definition, not the serving code.
• Read-path engineering: batch lookups per request (one round trip for all entities/views), p99 budgets, hot-key caching — ordinary low-latency engineering (the serving phase deepens it).

Chapter 6: Training-Serving Skew — a Taxonomy

"Skew" is three distinct diseases; name them separately because the fixes differ:

1. Logic skew: two implementations of "the same" feature differ — language ports, library versions, different null handling, different window arithmetic (calendar-month vs rolling-30d). Fix: one definition executed by both paths (the feature store's reason to exist), plus parity tests (Ch. 7).
2. Temporal skew: training joins at label time; serving reads materialized-lagged values (Ch. 5) — same logic, different effective timestamps. Fix: lag-aware training joins; freshness monitoring.
3. Distribution skew: the live entity/feature distribution drifts from the training frame's (new market launches, marketing campaign changes the user mix) — nobody's bug, still your problem. Fix: detection (Ch. 7's PSI) wired to Phase 12's retraining policy.

The Google-coined umbrella ("training-serving skew") hides this trichotomy; the taxonomy is what makes incidents debuggable.

Chapter 7: Parity Testing and Skew Detection

The two instruments the lab builds:

• Parity testing (logic+temporal skew): for a sample of (entity, timestamp) pairs, compute the feature offline (as-of join into history) and fetch it online (as materialized at that moment — in tests, materialize-then-lookup) and assert equality within tolerance. Run it in CI on every feature-definition change and continuously on a trickle of production traffic (log online values; compare against the offline recomputation nightly). A feature without parity coverage is an unverified rumor — the same epistemics as every "export-verify" ritual in this curriculum.
• Distribution-skew detection (PSI — Population Stability Index): bin the training distribution; compare serving-log proportions:

$$\text{PSI} = \sum_b (p_b^{serve} - p_b^{train}) \cdot \ln\frac{p_b^{serve}}{p_b^{train}}$$

Folklore thresholds: <0.1 stable, 0.1–0.25 watch, >0.25 act. It's KL symmetrized by use, not statistics-sacred — calibrate per feature against incident history. Phase 12 generalizes this into the full drift/monitoring system; here it guards one seam.

Chapter 8: Build vs Adopt, and Operating Realities

• What the platforms actually give you: registry + definition language, the PIT join implemented correctly, materialization orchestration, online-store adapters (Feast: open-source, bring-your-own infra; Tecton: managed + streaming transforms; cloud-native ones: integration over features). What you still own: feature quality (Phase 02's gates), defaults/TTL decisions, parity-in-CI, skew monitoring, and cost (online stores at high QPS are real money).
• Adopt when: multiple teams share features, PIT correctness is currently hand-rolled per project, or online serving exists at all. Don't adopt when: one batch model, no online serving — a warehouse table + merge_asof discipline is the right size (resisting platform-for-platform's-sake is senior judgment).
• Operating gotchas with scars attached: backfilling a corrected feature (version the view; never mutate history in place — Phase 02 Ch. 8), entity-key cardinality explosions (cross-product entities), and on-call for materialization lag (it's a freshness SLO, treat it like one).

Lab Walkthrough Guidance

Lab 01 — Mini Feature Store, suggested order:

1. FeatureView registration over a timestamped DataFrame; validate inputs with Phase 01 instincts (required columns, timestamp dtype).
2. The as-of join — the heart. Implement per-entity sort + searchsorted (or verify against pd.merge_asof as your oracle — differential testing). Get the ≤ boundary and the TTL exactly right; the tests probe both edges.
3. The adversarial leakage test: construct data where naive-latest ≠ as-of and watch your implementation pass while the naive join (provided as a foil) fails.
4. Online store: materialize(as_of_time) (latest value per key within TTL) + get_online(keys) with declared defaults on miss.
5. Parity checker: offline value at time T == online value materialized at T, over a random sample.
6. PSI skew detector + the deliberate-skew test (inject a shifted serving log; detector must fire; un-shifted must not).

Success Criteria

You are ready for Phase 04 when you can, from memory:

1. State the as-of join semantics in one precise sentence including TTL, and defend the ≤-with-availability-timestamps convention.
2. Explain why leakage inflates offline metrics specifically (causal downstream-ness of leaked features) and what the adversarial test looks like.
3. Recite the three-way skew taxonomy with one fix each.
4. Explain the materialization-lag subtlety and the lag-aware training answer.
5. Write the PSI formula and its use-not-sacred thresholds.
6. Argue build-vs-adopt for two scenarios (multi-team online recsys; single batch churn model).

Interview Q&A

Q: Offline AUC is 0.92; the deployed model performs like 0.71. Walk through your investigation. Leakage first (the magnitude says so): audit every feature's timestamp semantics — recompute the training frame with a strict as-of join and re-measure offline; a big drop confirms it (the classic: a feature computed post-label). If offline survives: skew taxonomy in order — parity-test logic (sample entities, compare offline recompute vs logged online values), check temporal skew (materialization lag vs training-join freshness), then distribution skew (PSI on serving logs vs training frame). The structure — leakage, then logic, then temporal, then distribution — is the answer.

Q: Why isn't get_latest(entity) an acceptable training join even with "fresh" data? Because training rows have different timestamps: latest-as-of-now joins tomorrow's feature values onto last month's labels — future information by construction. Each label gets the value as of its own moment. The only case where latest is fine is when feature values never change — at which point the timestamp discipline costs nothing anyway. (Then sketch the merge_asof.)

Q: Your online store misses 8% of lookups (new users). What's the full story of handling that correctly? Declare the default in the feature definition (not service code); ensure the training pipeline produced the identical value for missing history (the median-vs-zero incident is exactly this failure); consider a "missingness indicator" feature so the model can learn the new-user pattern explicitly (CV-track Phase 02's imputation lesson); monitor miss rate as a first-class metric (a step change means a materialization outage or an entity-key bug, not suddenly more new users).

References

• Feast documentation — read "point-in-time joins" and TTL semantics against your implementation
• pandas merge_asof docs — your oracle's exact semantics
• Sculley et al., Hidden Technical Debt in ML Systems (2015) — the pipeline-jungle and skew sections
• Uber Michelangelo and [Palette feature store] writeups — the original production accounts
• Tecton blog: What is a Feature Store — vendor-flavored but precise on anatomy
• Breck et al., The ML Test Score — the parity-testing rubric items
• CV track Phase 02 WARMUP — the leakage taxonomy this chapter specializes

Lab 01 — Mini Feature Store

Phase: 03 — Feature Stores & Training-Serving Skew | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

Temporal leakage and training-serving skew are ML's most expensive bug class. This lab builds the machinery that prevents both: point-in-time-correct joins, an online store with declared defaults, parity verification, and PSI skew detection — with an adversarial leakage test where the naive join provably leaks and yours provably doesn't.

What you build

• FeatureView / FeatureStore.register — feature definitions as code, with TTL and default as part of the definition (not the serving code — that's how the median-vs-zero skew incident happens)
• get_historical — the as-of join: largest τ ≤ label_ts within TTL, validated against pandas.merge_asof as a differential oracle
• materialize / get_online — latest-within-TTL per key; defaults on miss
• parity_check — online value == offline recomputation at the same instant
• psi — distribution-skew detection between training frame and serving log

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 12 tests pass — test_leakage_adversarial_naive_join_differs_from_asof is the certificate
• You can defend the ≤ boundary convention and say when it self-leaks (availability vs occurrence timestamps)
• You can explain what breaks if online TTL ≠ the TTL used in training joins

Extensions

• Lag-aware historical joins: join as-of label_ts − materialization_lag to train against what serving actually sees (WARMUP Ch. 5's subtlety, implemented)
• Multi-view get_historical returning a full training frame
• A streaming materializer fed by your Phase 02 stream processor's window results

References

• Feast: point-in-time joins
• pandas merge_asof — the oracle
• WARMUP.md chapters 3–7

Phase 04 — Semantic Retrieval & Search

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "search … semantic retrieval" — and the candidate- generation stage of every ranking/recsys/RAG system in phases 05, 06, and 10

Why This Phase Exists

Retrieval is the universal first stage: search ranks what retrieval found, recommenders rank retrieved candidates, RAG generates from retrieved passages. A senior MLE owns retrieval as an engineered system — lexical and dense scorers, fusion, indexes, and above all an evaluation harness with labeled queries, because retrieval quality upper-bounds everything downstream and is invisible without measurement.

This phase is the search-engineering counterpart of the LLM track's RAG phase (its Phase 07 covers RAG assembly and ANN internals): here you build the scorers and the harness from scratch and own the evaluation methodology.

Concepts

• The retrieval problem: queries, documents, relevance; recall-oriented first stage
• BM25 from first principles: IDF, term-frequency saturation (k1), length normalization (b) — implemented, not imported
• Dense retrieval: embeddings, cosine/dot equivalence under normalization, bi-encoder asymmetry (the LLM track covers training; here we operate them)
• Hybrid fusion: reciprocal rank fusion (RRF) and why rank-based fusion wins
• Index structures at a working level: inverted index, HNSW/IVF trade-offs
• Retrieval evaluation: qrels, recall@k, MRR, precision@k; building a labeled set
• Query understanding lite: normalization, analyzers, synonyms — where lexical systems are actually won

Labs

Lab 01 — Hybrid Retrieval Engine (flagship, implemented)

→ Lab folder: lab-01-hybrid-retrieval/

Extension Project A — ANN Benchmark (spec)

Index 100K+ real embeddings (any open set) with hnswlib and FAISS-IVF; measure recall@10-vs-QPS curves across efSearch/nprobe; reproduce the recall-latency trade curve and write the "which index when" memo.

Extension Project B — Query-Set Construction (spec)

Build a 50-query labeled set for a corpus you care about (docs, wiki dump): sample real-ish queries, label relevant docs, compute inter-annotator agreement on a subset — then measure your lab's engine on it. The artifact every retrieval team needs and few have.

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 12 tests pass
• Extension A curves + memo
• Extension B labeled set + baseline numbers

Warmup Guide — Semantic Retrieval & Search

Zero-to-expert primer for Phase 04: retrieval as an engineered system — BM25's mechanics derived term by term, dense retrieval operated honestly, hybrid fusion, index structures, and the evaluation harness that makes any of it improvable.

Table of Contents

• Chapter 1: Retrieval's Place in Every ML System
• Chapter 2: The Inverted Index
• Chapter 3: BM25, Derived Term by Term
• Chapter 4: Analyzers — Where Lexical Search Is Won
• Chapter 5: Dense Retrieval, Operated
• Chapter 6: Hybrid Fusion
• Chapter 7: Index Structures and Their Trades
• Chapter 8: Evaluating Retrieval
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Retrieval's Place in Every ML System

Retrieval answers: given a query, find the small set of candidates worth scoring carefully. It is the first stage of a universal two-stage architecture:

millions of items → [RETRIEVAL: cheap, recall-oriented] → hundreds
                  → [RANKING: expensive, precision-oriented] → ten

Search engines, recommenders (Phase 06's candidate generation), ads systems, and RAG (Phase 10) are all instances. Two consequences define the engineering:

1. Recall@k of the first stage upper-bounds the whole system — the ranker cannot rank what retrieval didn't return. Measuring first-stage recall separately (Ch. 8) is therefore non-negotiable; end-to-end metrics can't localize the loss.
2. Retrieval must be cheap per item (it touches everything) — which is why its models are constrained (bag-of-words scoring; bi-encoders with offline-computable document vectors) and its data structures matter (Ch. 2, 7).

Chapter 2: The Inverted Index

From zero: scoring every document per query is O(N). The inverted index makes lexical retrieval sublinear: a map from each term to its postings list — the documents containing it (plus term frequencies/positions):

"watermark" → [(doc3, tf=2), (doc17, tf=1), ...]

Query time: union/intersect the postings of the query's terms; only documents sharing at least one term are touched — for selective terms, thousands instead of millions. Everything Lucene/Elasticsearch does is elaboration: compressed postings, skip lists for fast intersection, segment files with merges (an LSM structure — append new segments, merge in background — the same write-pattern logic as Kafka's logs and LSM databases). You build the dict-of-lists version in the lab; the production versions differ in constants, not concept.

Chapter 3: BM25, Derived Term by Term

The scoring function (per query term t, summed):

$$\text{score}(q, d) = \sum_{t \in q} \underbrace{\ln!\left(\frac{N - df_t + 0.5}{df_t + 0.5} + 1\right)}{\text{IDF}} \cdot \underbrace{\frac{tf{t,d} ,(k_1 + 1)}{tf_{t,d} + k_1\left(1 - b + b,\frac{|d|}{\text{avgdl}}\right)}}_{\text{saturating TF, length-normalized}}$$

Each piece, justified from a failure of the simpler thing:

• Why IDF: raw term counts let "the" dominate. Rarity ≈ informativeness; the log damps (a term in 1 doc vs 10 is a big deal; 100K vs 1M is not); the +0.5s smooth zero/all-docs cases. (Probabilistic-relevance derivation exists — Robertson/Spärck Jones; for engineering, "log-damped rarity weight" is the load-bearing intuition.)
• Why TF saturates (the $k_1$ term): the 2nd occurrence of "watermark" is strong evidence; the 20th adds little — relevance isn't linear in count. The hyperbola $tf(k_1{+}1)/(tf{+}k_1)$ rises to an asymptote; $k_1 \approx 1.2$ sets how fast. ($k_1 = 0$ → pure presence/absence; large $k_1$ → near-linear TF.)
• Why length normalization (the $b$ term): long documents match more terms by bulk, not by aboutness. Dividing TF's denominator by the doc's relative length ($|d|/\text{avgdl}$) penalizes proportionally; $b \in [0,1]$ interpolates between no normalization ($b{=}0$) and full ($b{=}1$); 0.75 is the folklore default. Corpora of uniform length (tweets) want low b; mixed corpora want it high.

The lab's property tests pin each behavior independently — rarity ordering, saturation curvature, length penalty — which is how you know your implementation rather than believe it.

Chapter 4: Analyzers — Where Lexical Search Is Won

The unglamorous truth: production lexical search quality is mostly analysis — the pipeline that turns text into terms:

• Tokenization (Phase 01 of the LLM track at the word level: unicode, punctuation, CJK segmentation), lowercasing, stemming/lemmatization (matching "watermarks" to "watermark" — recall vs precision knob), stopwords (mostly obsolete with BM25's IDF — removing them saves index size, costs phrase queries), synonyms (domain dictionaries: "k8s" = "kubernetes" — the highest-ROI relevance work in enterprise search), and exact fields for IDs/codes (never stem a SKU).
• The cardinal rule (same as the feature store's parity rule): index-time and query-time analysis must match — a stemmed index queried unstemmed silently returns nothing. Same disease as training-serving skew; same cure (one definition, two call sites, a parity test).

Chapter 5: Dense Retrieval, Operated

The LLM track's Phase 02 and Phase 07 cover how embedding models are trained; here is the operator's checklist:

• Normalization: L2-normalize at index AND query time; dot product then equals cosine and rankings coincide — the lab asserts this property.
• The asymmetry trap: retrieval embedders use different prefixes/instructions for queries vs passages; omitting them at one side silently costs recall (the #1 production retrieval bug — worth repeating across tracks).
• What dense buys: paraphrase robustness ("how do I reset my password" matches "credential recovery"); what it loses: exact identifiers, rare entities, negations — precisely lexical's strengths (the complementarity that motivates Ch. 6).
• Version discipline: an index built with embedder v1 queried with v2 is silent garbage — embedder version is part of the index's schema; reindex-on-upgrade is a planned migration (Phase 11 of the PMC track's skew matrix, once again).

Chapter 6: Hybrid Fusion

Run lexical and dense in parallel; merge. The fusion question: BM25 scores (unbounded, corpus-dependent) and cosines ([−1,1]) are incommensurable — score arithmetic is meaningless without calibration. Reciprocal Rank Fusion sidesteps scores entirely:

$$\text{RRF}(d) = \sum_{r \in \text{retrievers}} \frac{1}{k + \text{rank}_r(d)} \qquad (k = 60 \text{ by convention})$$

Rank-based → scale-free, robust, zero tuning; $k$ damps the top-rank dominance. Items found by both retrievers get two contributions — agreement is evidence. RRF is embarrassingly simple and embarrassingly hard to beat; weighted score fusion with per-retriever normalization can edge it with tuning data, and learning-to-rank over both scores (Phase 05) is the principled ceiling. Engineering note: over-retrieve from each parent (k=50–100 each) before fusing — fusion can only promote what was retrieved.

Chapter 7: Index Structures and Their Trades

The operator's map (build intuition here; the LLM track details HNSW/IVF mechanics):

Two operational facts that bite: metadata filtering interacts badly with ANN (filter-during-graph-traversal degrades recall unpredictably — engines differ; ask the question of any vector DB), and the brute-force comparison on a sample is the only honest measure of your ANN recall — run it routinely (the lab's pattern, and the Extension A benchmark's whole point).

Chapter 8: Evaluating Retrieval

The harness (the lab's second half) — small, labeled, decisive:

• qrels: (query → set of relevant doc ids). Even 30–50 labeled queries transform development from vibes to measurement (the recurring curriculum claim; here it's literal — Extension B has you build one).
• Metrics, chosen by stage: recall@k for first-stage retrieval (did the answer make it into the candidate set? — the bound from Ch. 1); MRR (mean reciprocal rank of the first relevant — right when one answer suffices); precision@k (density of good stuff — right for browse). NDCG (graded relevance, position-discounted) belongs to the ranking stage — Phase 05 implements it from scratch.
• Methodology (Phase 02-of-CV's ethics, applied): fixed query set, paired comparisons between systems, never tune on the test queries, slice by query class (the lab's lexical-gap vs paraphrase-gap cases are exactly such slices — aggregate recall hides which retriever fails where; the slice tells you).

Lab Walkthrough Guidance

Lab 01 — Hybrid Retrieval Engine, suggested order:

1. Tokenizer (lowercase, alphanumeric split — keep it simple and identical for index and query: Ch. 4's rule) and the inverted index.
2. BM25 scoring; then the three property tests (rarity, saturation, length) — write them to fail first if you want the full lesson.
3. Dense index: normalized matrix + argpartition top-k; the cosine==dot assertion.
4. RRF; then the two constructed query classes: an ID-lookup query (dense fails, BM25 wins) and a paraphrase query with hand-set vectors (BM25 fails, dense wins) — hybrid must survive both. This pair of tests is the phase's thesis in executable form.
5. The eval harness: recall@k and MRR against qrels, verified on a hand-computable miniature.

Success Criteria

You are ready for Phase 05 when you can, from memory:

1. Draw the two-stage architecture and state what first-stage recall bounds.
2. Write BM25 and explain k1 and b from the failures they fix; predict the effect of b=0 on a mixed-length corpus.
3. State the analyzer parity rule and its skew analogy.
4. List dense retrieval's three operator traps (normalization, prefixes, version).
5. Justify RRF's rank-based design and the over-retrieve-then-fuse rule.
6. Pick metrics per stage (recall@k vs MRR vs NDCG) with reasons.

Interview Q&A

Q: Search works for product IDs but fails for "cheap red running shoes". Or the reverse. Diagnose each. IDs work / natural language fails: pure lexical system — no semantic matching; add a dense retriever and fuse (RRF), and check the analyzer isn't stemming IDs into collisions. Natural language works / IDs fail: pure dense system — embeddings blur exact identifiers; add lexical with exact-match fields (never stemmed), fuse. Both diagnoses end the same way: hybrid is the default for heterogeneous query traffic, and query-class slicing in the eval harness is how you saw it.

Q: Your ANN index reports great latency but product metrics sagged after the migration from brute force. What happened? Recall loss: ANN is approximate, and recall@k vs brute force on a labeled sample is the only honest measure — measure it (likely culprits: efSearch/nprobe tuned for the benchmark not the workload; metadata filtering degrading graph traversal; deletes accumulating as tombstones). The senior lesson: latency is visible by default, recall is invisible by default — instrument the invisible one before migrating.

Q: Why is RRF so hard to beat, and what eventually beats it? It's scale-free (no score calibration across incommensurable retrievers), robust to a bad parent (a garbage retriever adds noise but rank-damping limits damage), and parameter-light (k=60 works broadly). What beats it: learned fusion — a ranker (Phase 05) trained on (query, doc, both-scores, features) with real labels, which can learn query-dependent weighting (lexical matters more for ID-shaped queries). That is, the thing that beats heuristic fusion is the next stage of the architecture doing its job — with labels and an eval harness, which RRF never needed and LTR does.

References

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) — the derivation behind Ch. 3
• Manning, Raghavan, Schütze, Introduction to Information Retrieval — ch. 1–2 (indexes), 6 (scoring), 8 (evaluation); free online
• Cormack, Clarke, Buettcher, Reciprocal Rank Fusion outperforms… (SIGIR 2009)
• Elasticsearch: analysis docs — Ch. 4 industrialized
• BEIR benchmark — heterogeneous retrieval evaluation, the public version of Ch. 8
• LLM track Phase 07 WARMUP — DPR training, HNSW/IVF mechanics, RAG assembly
• hnswlib and FAISS wiki — Extension A's tools

Lab 01 — Hybrid Retrieval Engine

Phase: 04 — Semantic Retrieval & Search | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

BM25 and dense retrieval fail on opposite query classes — IDs vs paraphrases. This lab builds both from scratch, fuses them with RRF, and proves the complementarity with adversarial tests where each parent fails and the hybrid survives.

What you build

• BM25Index — inverted index + the full scoring function, with property tests pinning each term's behavior: IDF rarity ordering, TF saturation (6× occurrences scores < 3× — sublinearity verified), and length normalization (b=0.75 penalizes padding; b=0 provably doesn't)
• DenseIndex — L2-normalized vectors, exact cosine top-k, with a scale-invariance test (cosine's defining property)
• rrf_fuse — rank-based fusion: scale-free (a test rescales scores 10,000× and asserts identical output), agreement-rewarding
• The eval harness — recall@k, MRR, macro-averaged evaluate() over qrels, verified on hand-computed cases

Key concepts

Files

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 12 tests pass — test_hybrid_survives_both_adversarial_query_classes is the thesis
• You can predict the effect of k1→0 and b→0 before re-running the property tests with those values (then verify)
• You can state what RRF's c=60 does and what happens at c=0

Extensions

• Weighted RRF with per-retriever weights tuned on a dev query set — measure whether it beats plain RRF on held-out queries (the tuning-data honesty check)
• Phrase/exact-field support for the ID class (never stem a SKU)
• Swap the toy vectors for a real sentence-transformer and re-run the paraphrase slice (mind the query/passage prefixes!)

References

• Robertson & Zaragoza (2009) — BM25's derivation
• Cormack et al. (SIGIR 2009) — RRF
• WARMUP.md chapters 2–6, 8

Phase 05 — Learning to Rank

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "ranking" — the precision stage on top of Phase 04's retrieval, and the modeling core of search, recommendations, and ads

Why This Phase Exists

Ranking is the highest-leverage ML problem in industry — small NDCG gains move revenue at companies whose product is an ordered list — and it is methodologically its own discipline: the unit of prediction is a list, the metrics are position-discounted, the labels come from biased clicks, and the objectives (pointwise/pairwise/listwise) differ in what they actually optimize. Senior MLE interviews probe ranking depth specifically because it can't be faked from classification knowledge.

The flagship lab implements NDCG from scratch (an interview classic) and a pairwise ranker with LambdaRank-style NDCG-weighted gradients — the idea that powers LambdaMART, still the strongest tabular ranker in production twenty years on.

Concepts

• The LTR data model: queries, candidate lists, graded relevance, query groups
• Metrics: DCG/NDCG (derived, not recited), MRR, MAP; why position discounts
• Objectives: pointwise (regression on labels — ignores list structure), pairwise (order pairs correctly — RankNet's logistic loss), listwise (LambdaRank/LambdaMART — pairwise gradients weighted by metric impact |ΔNDCG|)
• Features for ranking: query, document, and query-document interaction features (BM25 score as a feature — Phase 04 composes in)
• Position bias: clicks are biased labels; examination hypothesis, counterfactual LTR / inverse propensity weighting at concept level
• Evaluation discipline: query-grouped splits (leakage otherwise!), per-query metric distributions

Labs

Lab 01 — Learning to Rank from Scratch (flagship, implemented)

→ Lab folder: lab-01-ltr-ndcg/

Extension Project A — LambdaMART on a Public LTR Set (spec)

Train LightGBM's lambdarank on MSLR-WEB10K (or the small LETOR sets): proper query-grouped CV, NDCG@10 vs your linear ranker, feature importance reading, and the "which features carry ranking" memo.

Extension Project B — Position-Bias Simulation (spec)

Simulate click logs with an examination-probability model over a known-relevance corpus; show that naive click-trained ranking learns the bias; implement inverse propensity weighting and recover the true ordering. The counterfactual-LTR intuition, built by hand.

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 12 tests pass
• Extension A numbers + memo
• Extension B simulation notebook

Warmup Guide — Learning to Rank

Zero-to-expert primer for Phase 05: ranking as its own ML discipline — NDCG derived from first principles, the pointwise/pairwise/listwise ladder, LambdaRank's central trick, click bias, and the evaluation discipline ranking demands.

Table of Contents

• Chapter 1: Why Ranking Is Not Classification
• Chapter 2: The LTR Data Model
• Chapter 3: NDCG, Derived
• Chapter 4: The Objective Ladder — Pointwise, Pairwise, Listwise
• Chapter 5: LambdaRank — Optimizing the Unoptimizable
• Chapter 6: Features and the Two-Stage Composition
• Chapter 7: Clicks Are Biased Labels
• Chapter 8: Ranking Evaluation Discipline
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Ranking Is Not Classification

A classifier predicts a property of one item; a ranker produces an ordering of a list, and the differences cascade:

• Only relative order matters: a scorer that outputs 0.9/0.8 ranks identically to one outputting 5.2/1.1 — calibrated probabilities are not the goal (until ads, where price × pCTR makes calibration matter again — know both regimes).
• Position is everything: users see the top; an error at rank 1 costs orders of magnitude more than at rank 50. Metrics must discount by position (Ch. 3) and objectives should concentrate effort at the top (Ch. 5).
• The query is the unit: items are only comparable within a query's candidate list. This reshapes the loss (pairs within groups), the split (by query — Ch. 8), and the metric (per-query, then averaged).

The economic stakes explain the interview weight: search, recommendations, ads, and feeds are ranking systems, and at list-product companies a 1% NDCG-correlated gain is a revenue line.

Chapter 2: The LTR Data Model

The canonical dataset row: (query_id, doc_id, feature_vector, relevance_label) — grouped by query_id into lists.

• Graded relevance: 0 (bad) … 4 (perfect) by convention (binary is the degenerate case). Sources: editorial judgment (expensive, clean) or behavioral signals (clicks/conversions — abundant, biased: Ch. 7).
• Query groups are sacred: every operation — pair generation, batching, splits, metric computation — respects group boundaries. The single most common LTR bug is leaking across them (a random row-level split puts the same query's docs in train and test: the model memorizes query-specific feature quirks and the offline number lies — the CV track's group-leakage taxonomy, ranking edition).
• Candidate lists come from Phase 04's retrieval — typically 100–1000 candidates per query, of which the ranker orders the top tens. The label distribution is wildly skewed (most candidates irrelevant) — pair-based objectives handle this naturally (only cross-label pairs train).

Chapter 3: NDCG, Derived

Build it from requirements, not memory:

1. Graded gains: more-relevant documents should contribute more. Convention: $\text{gain} = 2^{rel} - 1$ — exponential, so a perfect (4 → 15) is worth ~5× a fair (2 → 3); this encodes "one great answer beats several mediocre ones."
2. Position discount: contribution decays with rank. Convention: $1/\log_2(\text{rank}+1)$ — gentle (heavy tails of attention), normalized so rank 1 has discount 1.

$$\text{DCG@k} = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i + 1)}$$

3. Comparability across queries: a query with three perfect docs has higher attainable DCG than one with none — raw DCG can't be averaged. Normalize by the ideal DCG (the DCG of the relevance-sorted ordering):

$$\text{NDCG@k} = \frac{\text{DCG@k}}{\text{IDCG@k}} \in [0, 1]$$

The properties your lab's tests pin: perfect ordering → 1.0; swapping a relevant doc downward strictly decreases it; it's invariant to what happens below k; all-zero relevance is defined as 0 (a convention — state it). Cousins for the toolbox: MRR (first-relevant only — navigational queries), MAP (binary labels, averaged precision — older IR). Knowing when NDCG is wrong (e.g., diversity isn't rewarded — ten near-duplicates of one perfect answer score brilliantly) is the senior layer.

Chapter 4: The Objective Ladder — Pointwise, Pairwise, Listwise

How do you train for a list metric with gradient methods? Three generations:

• Pointwise: regress/classify each (query, doc) label independently. Simple, reuses everything you know — and ignores list structure entirely: it spends capacity getting absolute label values right when only within-query order matters, and weights every doc equally regardless of position impact.
• Pairwise (RankNet): for documents $i \succ j$ within a query, model $P(i \succ j) = \sigma(s_i - s_j)$ and minimize the logistic loss; the gradient on each score is the elegantly simple $\lambda_{ij} = -\sigma(-(s_i - s_j))$ pushing $s_i$ up and $s_j$ down. Now the model optimizes order — but all pairs are equal: fixing a rank-49/50 inversion earns as much as fixing rank-1/2. The metric says otherwise.
• Listwise in effect (LambdaRank): Chapter 5's move — keep the pairwise gradient, scale it by how much the metric cares about that pair.

Chapter 5: LambdaRank — Optimizing the Unoptimizable

The obstruction: NDCG depends on ranks, ranks come from sorting, and sorting is piecewise-constant in the scores — the gradient is zero almost everywhere. You cannot backprop through a sort (usefully).

LambdaRank's insight: you don't need the loss — you only need its gradients. Define them directly: for a pair $(i \succ j)$,

$$\lambda_{ij} = -\sigma\big({-(s_i - s_j)}\big) \cdot \big|\Delta \text{NDCG}_{ij}\big|$$

where $|\Delta \text{NDCG}_{ij}|$ is the NDCG change if you swapped $i$ and $j$ in the current ranking. Pairs straddling the top of the list get large weights; deep-list inversions get nearly none — the model's effort lands where the metric pays. Empirically (and later with some theory — it approximately optimizes NDCG), this works remarkably well; LambdaMART = these lambdas fed to gradient-boosted trees (Phase CV-02's boosting + this gradient), and it remains the production standard for tabular ranking (LightGBM's lambdarank objective). Modern alternates (softmax/ ListNet losses, differentiable sorting relaxations) exist; the lambda trick is the load-bearing idea to own — your lab implements it on a linear scorer where its effect is isolated and testable.

Chapter 6: Features and the Two-Stage Composition

The ranker's feature vector, by provenance:

• Query features: length, type/intent class, historical CTR of the query.
• Document features: quality/popularity priors, freshness, length.
• Query-document interaction features — where ranking quality actually lives: BM25 score (Phase 04's scorer becomes a feature), dense similarity, field-level matches (title vs body), historical query-doc engagement.
• The composition pattern to internalize: first-stage scores are second-stage features — retrieval, fusion, and ranking aren't competitors but stages, each consuming the previous one's outputs. (This is also why LTR "beats RRF" in Phase 04's framing: it includes it.)
• Tabular rankers (LambdaMART) want engineered interactions; neural rankers (two-tower for scale — Phase 06; cross-encoders for quality — LLM track Phase 07) trade feature engineering for representation learning and serving cost. The pragmatic production stack is still: retrieval → GBDT ranker → (sometimes) a neural re-ranker on the top handful.

Chapter 7: Clicks Are Biased Labels

The label problem that defines industrial LTR: editorial labels are scarce; clicks are infinite — and clicks conflate relevance with examination. The examination hypothesis: $P(\text{click}) = P(\text{examined at its position}) \cdot P(\text{relevant})$. Users don't see rank 50, so it never gets clicked, so it looks irrelevant, so it stays at rank 50: training on raw clicks freezes the current ranking's prejudices (a feedback loop — Phase 06 meets its recsys twin).

The counterfactual-LTR response, at working level:

• Estimate examination propensities $p_{pos}$ (position-bias curves — via result randomization/swap experiments, or jointly with relevance).
• Inverse propensity weighting: weight each click by $1/p_{pos}$ — an unbiased (if noisy) estimate of relevance-driven clicks. Extension B builds the simulation that makes this visceral: naive click training provably learns the bias; IPW provably recovers.
• The operational guards either way: exploration traffic (small randomization budgets), and never evaluating a new ranker only on logs collected under the old one (the same counterfactual sin — Phase 11's experimentation discipline is the real answer).

Chapter 8: Ranking Evaluation Discipline

• Split by query group, always (Ch. 2's bug). Time-based query splits when the corpus/behavior drifts (it does).
• Report per-query distributions, not just means: NDCG@10 = 0.71 hides whether hard queries (the head of complaints) are at 0.2; slice by query class (the Phase 04 lesson — navigational vs informational vs long-tail).
• Compare paired (same queries, two rankers) with the significance machinery of Phase 11/the model-accuracy track — per-query NDCG deltas are the paired samples.
• Offline ≠ online: NDCG on editorial labels and CTR online can disagree (presentation effects, bias in labels, diversity). The offline metric selects candidates for the A/B test (Phase 11); it doesn't replace it. Saying this sentence in interviews — with why — is a seniority marker.

Lab Walkthrough Guidance

Lab 01 — LTR from scratch, suggested order:

1. dcg_at_k / ndcg_at_k exactly to Chapter 3's formulas; verify against a fully hand-computed 4-doc example (do the arithmetic on paper first — the test contains the numbers).
2. Pair generation within query groups: all (i, j) with $rel_i > rel_j$ — and the group-boundary test that ensures no cross-query pairs.
3. RankNet step on a linear scorer: per pair, $\lambda = -\sigma(-(s_i - s_j))$, accumulate ±λ·x into the gradient. Train on synthetic data; NDCG must rise.
4. LambdaRank: multiply each pair's λ by |ΔNDCG| computed from the current ranking (swap i, j; recompute the two discount terms — you don't need full re-evaluation; deriving the two-term shortcut is part of the exercise).
5. The discriminating experiment (provided as a test): data constructed so top-of- list errors and deep-list errors trade off — plain pairwise splits its effort, lambda concentrates at the top, NDCG@5 shows the gap.

Success Criteria

You are ready for Phase 06 when you can, from memory:

1. Derive NDCG (gains, discount, normalization) and compute it by hand for 4 docs.
2. State what each objective generation ignores (pointwise: order; pairwise: position impact) and write RankNet's pair gradient.
3. Explain the zero-gradient obstruction and LambdaRank's gradient-first answer with the |ΔNDCG| factor.
4. Explain why splits must be query-grouped and what breaks otherwise.
5. Write the examination hypothesis and the IPW correction, and describe the feedback loop raw-click training creates.
6. Argue why offline NDCG selects A/B candidates rather than replacing the A/B.

Interview Q&A

Q: Implement NDCG@k. (The actual coding question.) Narrate while writing: gains $2^{rel}-1$ (exponential by convention — top-heavy), discounts $1/\log_2(i{+}1)$, DCG@k as their dot product over the predicted order, IDCG from sorting relevance descending, ratio with the 0/0 → 0 convention, and the test cases you'd write: perfect order = 1, downward swap strictly decreases, invariance below k. The narration of conventions-as-choices is what separates senior answers.

Q: Your ranker improved NDCG offline but CTR dropped in the A/B. Hypotheses? (1) Label-objective mismatch: editorial labels reward relevance; users click on freshness/snippets/diversity the labels don't encode. (2) The offline eval was biased: logs from the old ranker (position bias) — the new ranker's wins were on queries the logs mis-labeled. (3) Presentation interaction: better documents, worse snippets at top. (4) Distribution: offline queries ≠ live mix (slice both). Actions: slice the A/B by query class, inspect rank-1 changes manually, check diversity metrics, and audit label provenance. Concluding "trust the online metric, fix the offline proxy" is the right posture.

Q: Why does everyone still use LambdaMART when neural rankers exist? Tabular interaction features (BM25, CTR priors, engagement counts) carry most industrial ranking signal, and GBDTs exploit them better per unit of compute and tuning than neural nets (the tabular-supremacy result, ranking edition); lambdas give trees a listwise objective; serving is microseconds on CPU. Neural wins arrive with representation problems (raw text/image relevance, cold start, semantic matching) — which is why production stacks are hybrid: GBDT on features that include neural similarity scores (Ch. 6's composition), not either/or.

References

• Burges, From RankNet to LambdaRank to LambdaMART: An Overview (MSR-TR-2010-82) — the lineage paper; read it after the lab
• Järvelin & Kekäläinen, Cumulated Gain-Based Evaluation of IR Techniques (2002) — NDCG's origin
• Joachims, Optimizing Search Engines using Clickthrough Data (KDD 2002) and Joachims et al., Unbiased LTR with Biased Feedback (WSDM 2017) — Ch. 7's foundations
• LightGBM lambdarank docs and MSLR-WEB10K dataset
• Liu, Learning to Rank for Information Retrieval (2009) — the survey/book
• Wang et al., Position Bias Estimation for Unbiased LTR (WSDM 2018)

Lab 01 — Learning to Rank from Scratch

Phase: 05 — Learning to Rank | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

"Implement NDCG" is a real interview question; "why does LambdaRank work" is its senior follow-up. This lab does both: NDCG to the letter, then RankNet and LambdaRank gradients on a linear scorer — with a constructed dataset where the |ΔNDCG| weighting measurably wins (≈0.52 → ≈0.74 NDCG@5 on the reference run).

What you build

• dcg_at_k / ndcg_at_k / ndcg_of_scores — exact conventions ($2^{rel}-1$ gains, $\log_2(i{+}1)$ discounts, 0/0→0), pinned by hand-computed tests and invariance tests (downward swap strictly decreases; below-k changes are invisible)
• make_pairs — within-group cross-label pairs only (the cross-group pair is LTR's classic leakage bug; a test hunts it)
• train_ranker — RankNet gradients ($-\sigma(-(s_i - s_j))$), then LambdaRank ($\times|\Delta NDCG@k|$, computed via the two-term swap shortcut you derive)
• grouped_train_test_split — query-grouped, never row-level

Key concepts

Files

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py     # see the gap

Success criteria

• All 12 tests pass — test_lambdarank_beats_plain_pairwise_on_topheavy_task is the thesis
• You can compute NDCG@3 for a 4-doc list on paper in under two minutes
• You can explain why the constructed dataset separates the two objectives (read make_topheavy_dataset until the trap is obvious)

Extensions

• Replace the linear scorer with a 2-layer MLP (the gradients don't change — that's the point of the lambda formulation)
• LightGBM lambdarank on the same data — verify it finds f0 too
• NDCG with alternative gain (linear) and discount (1/rank) — measure how convention choices move the metric (they're choices, not laws)

References

• Burges, From RankNet to LambdaRank to LambdaMART (MSR-TR-2010-82)
• Järvelin & Kekäläinen (2002) — NDCG
• WARMUP.md chapters 3–5

Phase 06 — Recommendation Systems

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "recommendation" — the canonical ML-system-design interview and the archetype of personalization at scale

Why This Phase Exists

"Design a recommendation system" is the single most common senior-MLE system-design prompt, and recsys is where every theme of this track converges: two-stage retrieval+ranking (Phases 04–05), feature freshness (Phase 03), feedback loops and position bias (Phase 05), online experimentation as the only truth (Phase 11).

The modeling core this phase implements by hand is implicit-feedback matrix factorization trained with BPR — the pairwise-ranking view of collaborative filtering that connects directly to Phase 05's machinery — evaluated honestly against the baseline every recsys paper must beat and many products don't: popularity.

Concepts

• Explicit vs implicit feedback; why "no interaction" ≠ "dislike" (the missing- not-at-random problem)
• Collaborative filtering: neighborhood methods → matrix factorization; user/item embeddings as learned representations
• BPR (Bayesian Personalized Ranking): pairwise loss over (user, seen, unseen) triples — Phase 05's pairwise idea, recsys edition
• The production architecture: candidate generation (two-tower / ANN — Phase 04's indexes) → filtering → ranking (Phase 05) → re-ranking (diversity, business rules)
• Cold start (users and items), popularity bias, feedback loops
• Evaluation: leave-last-out / time-based splits, recall@k / NDCG@k, coverage and popularity-slice metrics; why offline recsys metrics are weak proxies

Labs

Lab 01 — Implicit-Feedback Recommender (flagship, implemented)

→ Lab folder: lab-01-bpr-recommender/

Extension Project A — Two-Tower on MovieLens (spec)

Train a two-tower model (torch) on MovieLens-100K with in-batch negatives; export item embeddings into your Phase 04 dense index; measure candidate-gen recall@100; compare against the BPR embeddings.

Extension Project B — Feedback-Loop Simulation (spec)

Simulate a recsys serving its own training data for 20 "days" (users click what's shown, model retrains on clicks): measure popularity concentration (Gini) over time; add an exploration epsilon; measure again. The degenerate-loop intuition, by hand.

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 12 tests pass
• Extension A recall table BPR vs two-tower
• Extension B concentration curves

Warmup Guide — Recommendation Systems

Zero-to-expert primer for Phase 06: implicit feedback's epistemology, matrix factorization and BPR derived, the production candidate-gen→rank architecture, cold start, feedback loops, and evaluation that doesn't fool you.

Table of Contents

• Chapter 1: The Recommendation Problem, Honestly Stated
• Chapter 2: Implicit Feedback and What Absence Means
• Chapter 3: Matrix Factorization — Embeddings Before Deep Learning
• Chapter 4: BPR — Ranking, Not Rating
• Chapter 5: The Production Architecture
• Chapter 6: Cold Start and the Content Bridge
• Chapter 7: Feedback Loops and Popularity Bias
• Chapter 8: Evaluating Recommenders
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Recommendation Problem, Honestly Stated

Given a user's interaction history, rank a catalog by predicted value of showing — which already contains the field's three honest difficulties: the label is behavioral (what users did under what they were shown — Ch. 7), the objective is contested (clicks? watch time? long-term retention? — metric choice is product strategy), and the catalog is huge while attention is ~10 slots (ranking economics, Phase 05, at their sharpest). The senior posture from the first sentence: a recommender is not a model but a system with a loop in it — model → exposure → behavior → training data → model — and most production pathologies live in the loop, not the matrix algebra.

Chapter 2: Implicit Feedback and What Absence Means

Explicit ratings (1–5 stars) are scarce and biased toward the opinionated; production runs on implicit signals: views, clicks, purchases, dwell. Two structural facts:

• Positive-only: you observe engagement, never dislike. An unobserved (user, item) pair conflates "unaware" with "uninterested" — missing not at random, since exposure was itself chosen by the previous model (Ch. 7's loop, already present in the data).
• Consequence for objectives: treating unobserved as 0 and regressing (the naive port of ratings MF) optimizes the wrong thing. The two classic responses: weighted regression with confidence weights (implicit-ALS — observed pairs get high confidence, unobserved low) or — cleaner — stop predicting values and rank (BPR, Ch. 4): only claim that seen things should rank above unseen things, the weakest statement the data actually supports.

Chapter 3: Matrix Factorization — Embeddings Before Deep Learning

The model: user $u$ and item $i$ get $d$-dimensional vectors; affinity is their dot product, $\hat{x}_{ui} = p_u^\top q_i$ (+ biases in the ratings era). Stack them and you've claimed the interaction matrix is approximately low-rank — preferences are combinations of a few latent factors (genre-ness, price-tier, novelty appetite…) discovered, not designed.

Why this 2009-vintage idea is still the right first model: it is representation learning (the original embedding model — today's two-tower nets are MF with feature towers, the LLM track's word2vec is MF of a co-occurrence matrix [its Phase 02 chapter says so explicitly], and your trained $q_i$ vectors drop straight into Phase 04's dense index for candidate generation); it's data-efficient, trains on a laptop, and remains an embarrassing-to-lose-to baseline. Training: alternating least squares (closed-form per row, parallelizes — the implicit-ALS classic) or SGD on sampled entries (your lab, because it composes with Ch. 4).

Chapter 4: BPR — Ranking, Not Rating

The objective: for user $u$, an interacted item $i$ should outscore a non-interacted item $j$:

$$\mathcal{L}{BPR} = -\sum{(u,i,j)} \ln \sigma\big(\hat{x}{ui} - \hat{x}{uj}\big) + \lambda |\Theta|^2$$

— Phase 05's RankNet logistic pair loss, with recsys's pair source: $(u, i, j)$ triples sampled as (user, positive, random-unseen-negative). The gradients (your lab implements exactly these):

$$\delta = 1 - \sigma(\hat{x}{ui} - \hat{x}{uj}); \quad p_u \mathrel{+}= \eta,(\delta,(q_i - q_j) - \lambda p_u); \quad q_i \mathrel{+}= \eta,(\delta,p_u - \lambda q_i); \quad q_j \mathrel{-}= \eta,(\delta,p_u + \lambda q_j)$$

Notes with production weight: random negatives are mostly easy (popular-item or hard-negative sampling speeds learning — same lesson as DPR's hard negatives in the LLM track); sampled pairwise ranking optimizes AUC-like objectives, not top-of-list specifically (WARP and lambda-weighting exist for that — Phase 05's idea returning); and the output that matters is the embedding spaces, reusable far beyond the scoring rule.

Chapter 5: The Production Architecture

No system scores the whole catalog per request. The canonical four stages:

catalog (10^6–10^9)
  → CANDIDATE GENERATION (10^2–10^3): ANN over two-tower/MF embeddings (Phase 04's
    indexes!), plus heuristic sources (co-visitation, same-author, trending, recency)
  → FILTERING: availability, already-seen/bought, policy, region
  → RANKING (10^1–10^2): the heavy model — GBDT/neural with full features
    (user, item, context, *freshness-sensitive* — Phase 03's online store serves this)
  → RE-RANKING: diversity (MMR-style), business rules, exploration slots

Design notes that interviews probe: multiple candidate sources are the norm (union + dedup — robustness over elegance; each source covers another's blind spot); the ranker's features are where Phase 03's feature store earns its keep (counts, recencies, CTRs — all freshness-budgeted); seen-item filtering is a product decision (re-recommending consumables is correct; re-recommending a watched movie isn't); and exploration slots (Ch. 7's antidote) are reserved here, in re-ranking, where their cost is controlled.

Chapter 6: Cold Start and the Content Bridge

Collaborative signals need history; new users and new items have none.

• New items: content features are the bridge — text/image embeddings (the other tracks' encoders) place a zero-interaction item near similar items in candidate space; engagement data then refines. Two-tower models do this natively (item tower consumes content features); pure-ID MF cannot (its $q_i$ is untrained) — the architectural reason two-tower replaced ID-MF at companies with item churn.
• New users: popularity-by-context (region/device/hour) as the fallback ranker, onboarding preference capture, and fast session-based adaptation (recommend from the last 3 clicks — sequence models shine here).
• The operational must: cold-start is a measured segment — recall@k sliced by user/item age (Ch. 8), never just the global average, because the global average is dominated by warm entities and will hide a broken cold path entirely.

Chapter 7: Feedback Loops and Popularity Bias

The loop stated plainly: the model decides exposure; users can only engage with what they see; engagement becomes training data; the next model amplifies the last one's choices. Consequences, each with a name:

• Popularity bias: popular items gather more interactions regardless of per-exposure quality; naive training compounds it; catalogs effectively shrink (the Gini concentration your Extension B measures rising over simulated days).
• Filter bubbles / interest narrowing: per-user loops — show sci-fi → clicks are sci-fi → show more sci-fi. Diversity re-ranking and exploration are the levers.
• Degenerate counterfactuals: you never learn what users would have done with what you never showed — the recsys twin of Phase 05's position bias, with the same family of answers: exploration budgets (small ε of traffic on non-greedy slates — the data they generate is what keeps future models honest; frame it as paying the data-collection bill, not losing revenue), propensity logging (record why each item was shown and with what probability — Phase 11's off-policy evaluation needs it), and popularity-debiasing in training (downweight popular positives / popularity-matched negatives).

Chapter 8: Evaluating Recommenders

• Split by time, not randomly: leave-last-out (each user's final interaction is the test target) or a global time cutoff — random splits leak the future (Phase 03's lesson; recsys edition) and overstate everything.
• Metrics beyond accuracy (recall@k/NDCG@k on held-out interactions): coverage (share of catalog ever recommended — the health metric for Ch. 7's concentration), novelty/popularity-slices (recall on tail items), per-segment slices (cold users!). A model that wins recall by recommending bestsellers to everyone loses coverage catastrophically — the lab measures both because the trade is the point.
• The baseline discipline: popularity (global, or per-segment) is the mandatory baseline; it's embarrassingly strong on accuracy metrics (Ch. 2's bias guarantees it) and embarrassingly bad on coverage — quoting both numbers for both models is the honest comparison.
• Offline is a weak proxy, weaker than in search: held-out clicks were generated under the old policy (the loop again); offline wins select A/B candidates (Phase 11), and long-term metrics (retention, not session clicks) are the real objective — the gap between them is where recsys product judgment lives.

Lab Walkthrough Guidance

Lab 01 — BPR Recommender, suggested order:

1. Data plumbing: interactions → per-user sets; leave-last-out split (each user's last interaction by timestamp is test).
2. Baselines first (the discipline): random and popularity top-k with seen-item masking; record their recall@10 and coverage.
3. BPR: embeddings ~ Normal(0, 0.1/√d); triple sampling (user with ≥2 train interactions, one of their positives, one unseen negative); the four-line gradient update; λ≈0.01, η≈0.05, a few epochs over n_interactions samples.
4. recommend(u, k): scores = $p_u Q^\top$, mask seen, top-k. Cold users (no train history) → popularity fallback (the test checks you don't crash and don't return garbage).
5. Eval: recall@10 (BPR must beat popularity on the synthetic structured data — the test's bar) and coverage (BPR must beat popularity's by construction; notice why).

Success Criteria

You are ready for Phase 07 when you can, from memory:

1. Explain missing-not-at-random and why it kills naive regression on implicit data.
2. Write the BPR loss and all four gradient updates.
3. Draw the four-stage architecture with cardinalities and say where Phases 03/04/05 each plug in.
4. Give the cold-start answer for items (content bridge / two-tower) and users (context popularity + session adaptation), plus the sliced-metrics must.
5. Tell the feedback-loop story with its three named consequences and three levers.
6. Defend leave-last-out + coverage + popularity-baseline as the minimum honest eval.

Interview Q&A

Q: Design a recommender for an e-commerce marketplace. (The classic.) Structure beats brilliance: clarify the objective (purchase? GMV? repeat rate?) and constraints (catalog size, item churn — determines cold-start weight). Then the four stages with numbers; candidate sources (two-tower ANN + co-purchase + trending); ranker features by freshness tier (feature store); business re-ranking; the loop-hygiene paragraph (exploration slots, propensity logging) — and the evaluation plan: offline leave-last-out for iteration, A/B on the true metric for truth, coverage and cold-segment dashboards for health. Interviewers grade the loop awareness and the eval plan more than the model choice.

Q: Your new model beats popularity by 8% recall@10 offline but the A/B shows nothing. Most likely explanations? (1) The offline gain was concentrated where it doesn't matter online — held-out clicks under the old policy reward predicting the old policy's exposure (the loop); (2) recall@10 isn't the product metric — position-weighted engagement may be flat while raw recall rises; (3) serving-path divergence — feature freshness differs between offline eval and online serving (Phase 03's temporal skew, the recsys incident); (4) the test lacked power for the realized effect (Phase 11's math). The investigation order: parity-check features, slice the A/B by segment, check the offline metric's correlation with the online one historically.

Q: Why does everyone use a two-stage (or four-stage) architecture instead of one great model? Economics and physics: scoring 10⁸ items with a 50-feature ranker per request is impossible at latency; candidate generation buys recall cheaply (ANN dot products), ranking buys precision expensively on 500 items. The split also decouples concerns — candidate sources are robust, recall-oriented, and individually simple; the ranker absorbs the business objective and feature complexity. The cost: the ranker can't recover what candidates missed (first-stage recall bounds all — Phase 04's law), so each stage gets its own metric and its own iteration loop. That sentence is the architecture, understood.

References

• Rendle et al., BPR: Bayesian Personalized Ranking from Implicit Feedback (UAI 2009)
• Hu, Koren, Volinsky, Collaborative Filtering for Implicit Feedback Datasets (ICDM 2008) — implicit-ALS, the confidence-weighting alternative
• Koren, Bell, Volinsky, Matrix Factorization Techniques for Recommender Systems (IEEE Computer 2009) — the Netflix-era summary
• Covington, Adams, Sargin, Deep Neural Networks for YouTube Recommendations (RecSys 2016) — the canonical two-stage production paper; read with Ch. 5 open
• Yi et al., Sampling-Bias-Corrected Neural Modeling for Recommendations (RecSys 2019) — two-tower with in-batch negatives, corrected
• Chaney et al., How Algorithmic Confounding in Recommendation Systems Increases Homogeneity (RecSys 2018) — Ch. 7, formalized
• Microsoft Recommenders repo — reference implementations to read after building yours

Lab 01 — Implicit-Feedback Recommender (BPR)

Phase: 06 — Recommendation Systems | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

Personalization, measured honestly: BPR matrix factorization built from four gradient lines, evaluated leave-last-out against the baseline most products never beat. On the reference run: popularity recall@10 = 0.18, BPR = 0.73 — and BPR's coverage is 1.0 where popularity's is 0.28. Both numbers are the lesson.

What you build

• leave_last_out — the time-respecting split (random splits leak the future)
• Baselines first — popularity (with seen-item masking) and random; the discipline every recsys claim is measured against
• BPRModel — user/item embeddings, (user, positive, negative) triple sampling, the four-line SGD update; recommend with seen-masking and a popularity fallback for cold users
• recall_at_k and coverage — accuracy and catalog health, because winning recall by showing everyone the bestsellers is a measured failure here
• A representation test: items from the same latent cluster end up closer in embedding space — the "MF is embedding learning" claim, asserted

Key concepts

Files

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py   # the table

Success criteria

• All 12 tests pass — test_bpr_beats_popularity_on_clustered_data and test_embeddings_cluster_by_latent_group are the certificates
• You can write the four BPR update lines from memory and say what δ → 0 means (the pair is already ordered; learning stops — saturation, by design)
• You can explain why popularity's coverage is so low and what that predicts about feedback loops (WARMUP Ch. 7)

Extensions

• Popularity-weighted negative sampling — measure the recall and tail-recall change
• Export model.Q into your Phase 04 DenseIndex: item-to-item recommendations and the candidate-generation pattern, literally connected
• AUC evaluation per user; compare its ranking of models against recall@10's

References

• Rendle et al., BPR (UAI 2009)
• Covington et al., Deep Neural Networks for YouTube Recommendations (RecSys 2016)
• WARMUP.md chapters 2–8

Phase 07 — Forecasting

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "forecasting" — demand, capacity, and finance planning; the ML domain with the strictest temporal discipline and the strongest baselines

Why This Phase Exists

Forecasting is where ML engineers embarrass themselves most reliably: the baselines (naive, seasonal-naive) are brutally strong, temporal leakage is one careless feature away, and the standard accuracy metrics (MAPE!) are booby-trapped. It is also a core enterprise workload — demand planning, capacity, staffing, finance — and a standard senior-MLE interview domain precisely because it tests discipline over modeling flair.

The flagship lab builds the artifact every forecasting team actually lives in: a rolling-origin backtesting harness with honest metrics (MASE, sMAPE), seasonal- naive baselines, and a leakage-proof feature-based ML forecaster that must earn its complexity by beating the baseline inside the harness.

Concepts

• The forecasting data model: series, horizon, frequency, hierarchy; point vs probabilistic forecasts
• Baselines first: naive, seasonal-naive, drift — and why they're hard to beat
• Classical decomposition vocabulary: trend, seasonality, ETS/ARIMA at working level
• The ML approach: direct multi-horizon forecasting with lag/rolling/calendar features — and the leakage rules for constructing them
• Backtesting: rolling-origin (expanding window) evaluation; why one split lies
• Metrics: why MAPE fails (zeros, asymmetry), sMAPE's partial fix, MASE (scaled by in-sample naive error — the cross-series honest metric)
• Intermittent demand, cold-start series, hierarchy reconciliation (concept level)

Labs

Lab 01 — Backtesting Harness & ML Forecaster (flagship, implemented)

→ Lab folder: lab-01-backtesting-forecaster/

Extension Project A — M5/Retail Slice (spec)

Run your harness on a slice of the M5 (Walmart) dataset: seasonal-naive vs your GBM vs LightGBM with richer features; per-series MASE distribution (not just the mean); the intermittent-series segment called out separately (Croston's method as the reference there).

Extension Project B — Probabilistic Forecast Memo (spec)

Add quantile forecasts (GBM with quantile loss) to your harness; evaluate with pinball loss; write the one-pager on why the capacity-planning consumer needs P90, not the mean.

Guides in This Phase

• WARMUP.md

Deliverables Checklist

• Lab 01 implemented; all 12 tests pass
• Extension A per-series distribution plots
• Extension B quantile harness + memo

Warmup Guide — Forecasting

Zero-to-expert primer for Phase 07: time series as a discipline of restraint — baselines that humble you, metrics that don't lie, backtesting that respects time, and the ML forecaster that earns its complexity.

Table of Contents

• Chapter 1: What Makes Forecasting Different
• Chapter 2: The Anatomy of a Series
• Chapter 3: Baselines — the Humbling Chapter
• Chapter 4: Classical Models at Working Level
• Chapter 5: Metrics — MAPE's Traps, MASE's Fix
• Chapter 6: Backtesting — Rolling-Origin Evaluation
• Chapter 7: The ML Forecaster — Features Without Leakage
• Chapter 8: Production Forecasting Realities
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What Makes Forecasting Different

Three structural facts separate forecasting from ordinary supervised learning:

1. Time is not exchangeable: rows are ordered and autocorrelated; everything built on i.i.d. assumptions (random splits, standard CV) is invalid by default. The entire phase is consequences of this sentence.
2. The future is genuinely uncertain, and the easy part of the future is carried by trivial models: tomorrow ≈ today, next Monday ≈ last Monday. Whatever your model adds must be measured above that floor (Ch. 3).
3. The consumer is an operational decision: inventory orders, staffing levels, capacity buys — usually needing a quantile (order to P90 demand), a horizon (lead time), and a granularity (SKU × store × day) determined by the decision, not the data scientist. Asking "what decision does this forecast feed?" first is the senior reflex.

Chapter 2: The Anatomy of a Series

The decomposition vocabulary (the lens for looking at a series before modeling):

• Trend: long-run level change. Seasonality: fixed-period patterns (weekly, yearly — multiple can coexist). Cyclical: irregular-period swings (business cycles). Irregular: what's left.
• Additive ($y = T + S + R$) vs multiplicative ($y = T \cdot S \cdot R$) composition — retail demand is usually multiplicative (holiday lift is a percentage); log transform turns multiplicative into additive (and stabilizes variance — the workhorse preprocessing move).
• The practical specs of a forecasting task: frequency (daily/weekly), horizon $h$ (how far ahead — set by the decision's lead time), history length, and the hierarchy (SKU → category → store → region: forecasts at every level that must reconcile — Ch. 8).
• Calendar structure is signal, not noise: day-of-week, month, holidays (and holiday proximity), paydays, promotions — Ch. 7's features.

Chapter 3: Baselines — the Humbling Chapter

The mandatory floor, in ascending sophistication:

• Naive: $\hat{y}_{t+h} = y_t$. Optimal for random walks — which many business series approximately are.
• Seasonal-naive: $\hat{y}{t+h} = y{t+h-m}$ (same point last season, $m$ = season length). The one to beat for anything with weekly/yearly pattern; on M-competition-class data it routinely lands mid-pack against sophisticated entries.
• Drift: naive + the average historical slope.

Why this chapter exists: decades of forecasting competitions (M1–M5) repeat one finding — simple methods are startlingly competitive, and complex methods win only with care (and in M5, with gradient boosting + good features rather than exotic architectures). The engineering posture that follows: every model claim in this phase is MASE vs seasonal-naive inside the backtest harness — your lab encodes the posture as a passing test.

Chapter 4: Classical Models at Working Level

What a senior MLE should know about, even when shipping GBMs:

• Exponential smoothing (ETS): forecasts as exponentially-weighted recent history; Holt adds trend, Holt-Winters adds seasonality. State-space form gives prediction intervals. Strengths: tiny data appetite, robust, fast over millions of series — the default for the long tail.
• ARIMA: model the series' autocorrelation structure — AR (regression on own lags), I (difference away trend), MA (regression on past errors); SARIMA adds seasonal terms. Strength: principled, interpretable correlation handling; weakness: per-series fitting/identification at scale, exogenous variables are bolt-ons.
• The honest modern summary: ETS/ARIMA remain excellent per-series univariate tools; the ML approach (Ch. 7) wins when there are many related series (cross-learning), rich covariates (promos, prices, weather), or cold-start series — which is precisely the enterprise demand-forecasting regime. Neural forecasters (DeepAR/N-BEATS/TFT-class) extend the same cross-series logic; reach for them after GBMs plateau, not before.

Chapter 5: Metrics — MAPE's Traps, MASE's Fix

The metric chapter matters because the standard one is broken:

• MAPE $= \text{mean},|y - \hat{y}| / |y|$: explodes at $y \to 0$ (intermittent demand → infinite/undefined), and is asymmetric — over-forecasting a small actual costs more than under-forecasting a large one, so optimizing MAPE systematically biases forecasts low (a real inventory incident, not a curiosity).
• sMAPE (divide by the average of $|y|$ and $|\hat{y}|$): bounded, still awkward at zero, still not symmetric in the ways its name implies — know it as "the M-competition convention," not as a fix.
• MASE — the one to internalize:

$$\text{MASE} = \frac{\text{MAE}(\text{forecast})}{\text{MAE}(\text{in-sample naive})}$$

The denominator is the one-step (or seasonal) naive error measured on the training history. So MASE reads as "error relative to the dumb method": < 1 beats naive, > 1 loses to it. It's defined at zero actuals, symmetric, and — the killer feature — comparable across series of different scales, which is what lets you average over a 10,000-SKU portfolio meaningfully (the seasonal variant scales by seasonal-naive in-sample error; your lab implements that one).

• Probabilistic forecasts (Extension B): pinball/quantile loss per quantile; coverage checks (does the P90 cover ~90%?) — because Ch. 1's consumers need quantiles.

Chapter 6: Backtesting — Rolling-Origin Evaluation

One train/test split lies in time series: a single cutoff date is one draw from a nonstationary world (one holiday season, one promo calendar). The honest protocol — rolling-origin (a.k.a. time-series CV, expanding window):

fold 1: train [0 .. T1]      → forecast (T1, T1+h]
fold 2: train [0 .. T1+s]    → forecast (T1+s, T1+s+h]
fold 3: train [0 .. T1+2s]   → ...

Each fold's metrics are computed on its forecast window; report the distribution across folds (mean ± spread — the Phase 02-of-CV uncertainty ethic, time edition). Rules your lab's splitter enforces by test: train always ends strictly before the forecast window; windows never overlap a fold's training data; the feature builder runs inside the fold (computing features once on the full series then splitting is the subtle leak — rolling means computed over the future). Backtesting is the forecasting analog of the experiment ledger: the harness, not the notebook, is where claims live.

Chapter 7: The ML Forecaster — Features Without Leakage

The reduction: forecasting → tabular regression. For each (series, date), build features from the past only, target = value at date (+ horizon handling):

• Lag features: $y_{t-1}, y_{t-7}, y_{t-14}, y_{t-28}$ (align lags to the seasonality), and for horizon h, lags must be ≥ h — predicting $t{+}7$ with lag-1 uses a value you won't have at forecast time; this is THE leak, and the lab's feature builder takes horizon as an argument to enforce it.
• Rolling statistics: mean/std/min/max over trailing windows — shifted by the horizon for the same reason.
• Calendar features: day-of-week, month, holiday flags — known in advance, always safe.
• Known-future covariates (promo calendar, prices): safe if truly known ahead; observed-only covariates (weather): must use forecasts-of-the-covariate at inference, so train on forecast-vintage values or accept the optimism (a classic hidden leak — saying this in interviews lands).
• One global model over all series (series id / category as features) is the cross-learning move that makes ML beat per-series classics — more data per parameter, cold-start handled by relatives (the M5-winning recipe: LightGBM + exactly these features).
• Multi-horizon strategies: recursive (feed predictions back — error compounds), direct (a model per horizon, or horizon-as-feature — the lab's choice: simpler to keep honest).

Chapter 8: Production Forecasting Realities

• Scale shape: forecasting is many-small-models/one-global-model over 10³–10⁶ series on a schedule (nightly/weekly) — an orchestration problem (Phase 08) more than a serving problem; "inference" is a batch job whose output lands in a database.
• Hierarchy reconciliation: SKU forecasts must sum to category forecasts that planners also see; naive independent forecasts won't. Approaches: bottom-up, top-down, or optimal reconciliation (MinT) — know the problem exists and that "whose number is official" is as much governance as math.
• Intermittent demand (mostly zeros, occasional spikes — spare parts, long-tail SKUs): standard metrics and models mislead; Croston's method / zero-inflated approaches; segment these series out in evaluation (Extension A) or they poison the averages.
• Monitoring (Phase 12 hooks): forecast-vs-actual error tracked per series per cycle is the drift detector; structural breaks (COVID, a new competitor) demand intervention features or regime resets — a model quietly extrapolating a dead regime is the expensive failure.
• The judgment layer: planners override forecasts; log overrides and measure their accuracy too — half the value of a forecasting platform is making the human-vs-model ledger visible.

Lab Walkthrough Guidance

Lab 01 — Backtesting Harness & ML Forecaster, suggested order:

1. Metrics first, hand-verified: mase (seasonal in-sample scaling) and smape on tiny arrays you compute on paper (the tests carry the numbers).
2. Baselines: naive_forecast, seasonal_naive_forecast — pure indexing; get the horizon arithmetic right.
3. rolling_origin_splits(n, initial, step, horizon) — yields index pairs; the no-leak property is tested (train end < every test index).
4. Feature builder: lags (≥ horizon!), trailing rolling mean (shifted), day-of-week — built per fold's training slice + frozen continuation, never on the full series.
5. The GBM direct forecaster (sklearn GradientBoostingRegressor) and the harness loop producing the comparison table: on the synthetic trended-seasonal-with- covariate data, seasonal-naive MASE ≈ 1 by construction and the GBM must land clearly below.

Success Criteria

You are ready for Phase 08 when you can, from memory:

1. Name the three structural differences from i.i.d. ML and their consequences.
2. Define naive/seasonal-naive/drift and state the competition finding that makes them mandatory.
3. Explain MAPE's two failures, what sMAPE is, and derive why MASE is cross-series comparable.
4. Draw rolling-origin evaluation and name the two subtle leaks (features computed on the full series; lags < horizon).
5. List the four feature families with their safety conditions.
6. Sketch the production shape: batch orchestration, hierarchy reconciliation, intermittent segmentation, forecast-vs-actual monitoring.

Interview Q&A

Q: Your demand model shows 12% MAPE; the planner says the old spreadsheet was better. How do you investigate? First suspect the metric: MAPE is asymmetric and explodes near zero — check the intermittent SKUs' contribution and whether optimizing it biased forecasts low (stockouts would confirm). Re-evaluate both methods in the same rolling-origin harness with MASE per series; segment (fast-movers vs intermittent, promo vs regular weeks); and check the planner's complaint window for a structural break the model extrapolated through. The likely truth: both are right — different metrics, different segments — and the harness adjudicates.

Q: Why can't you use standard k-fold CV for time series, and what do you do instead? Random folds train on the future of their test points — autocorrelation makes neighboring points near-duplicates, so the model effectively memorizes the answer; offline scores inflate and production disappoints (temporal leakage, the same disease as Phase 03's feature-store joins). Instead: rolling-origin evaluation — expanding train windows, forecast the next h steps, repeat — with features rebuilt inside each fold. Bonus point: even hyperparameter tuning must respect this (nested rolling-origin), or the tuning itself leaks.

Q: When would you choose ETS over your gradient-boosting forecaster? Few series with decent individual history and no covariates (nothing for cross-learning to exploit); very short series (GBM features eat history); the long-tail regime where fitting cost per series matters (ETS is closed-form fast); and when calibrated prediction intervals are first-class (state-space form gives them natively). The deployment answer is usually a portfolio: ETS for the tail, the global GBM for covariate-rich heads, seasonal-naive as the floor everywhere — chosen per segment by the backtest harness, not by ideology.

References

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed., free at otexts.com/fpp3) — the field's textbook; ch. 5 (toolbox), 7–9 (ETS/ARIMA)
• Hyndman & Koehler, Another look at measures of forecast accuracy (2006) — MASE's source
• Makridakis et al., The M5 Accuracy Competition (2022) — what actually won (LightGBM + features) and why
• Tashman, Out-of-sample tests of forecasting accuracy (2000) — rolling-origin methodology
• Croston (1972) / Syntetos & Boylan — intermittent demand
• Wickramasuriya et al., Optimal Forecast Reconciliation (MinT) (2019)
• sktime and statsforecast — the libraries whose internals mirror this lab

Lab 01 — Backtesting Harness & ML Forecaster

Phase: 07 — Forecasting | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours

The forecasting lab is really a methodology lab: rolling-origin backtesting, scale-free metrics, baselines that must be beaten, and the lag-≥-horizon rule that keeps ML forecasters honest. The model is the easy part.

What you build

• smape / mase — the two metrics worth carrying: bounded symmetric error and the beats-naive ratio (MASE < 1 = better than the naive baseline; > 1 = your model is worse than doing nothing clever)
• naive_forecast / seasonal_naive_forecast — the baselines every forecast must clear, with the wrap-around indexing done correctly
• rolling_origin_splits — expanding-window time-series CV (random K-fold on a time series is leakage by construction)
• build_features — lag/rolling/calendar features under the lag ≥ horizon rule: when forecasting 7 days ahead, lag-1 doesn't exist at prediction time
• GBMForecaster — a direct (non-recursive) gradient-boosting forecaster over those features, with a runtime assertion that no feature reached into the future
• backtest — the harness scoring all methods on identical folds

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference
python solution.py                          # the comparison table

Reference output on the synthetic demand series:

gbm             MASE = 0.476
seasonal_naive  MASE = 0.848
naive           MASE = 2.336

Suggested TODO order

1. smape, mase — hand-verify against the worked numbers in the tests
2. Baselines — the seasonal wrap-around indexing is the classic off-by-one
3. rolling_origin_splits — check the no-leak property test first
4. build_features — write the lag rule down before coding it
5. GBMForecaster.fit/predict — the NaN assertion is your leak alarm
6. backtest — same folds, same metric, all methods

Success criteria

• All 12 tests pass
• Your GBM beats seasonal-naive by ≥30% MASE on the demand series
• You can explain why lag-1 is illegal at horizon 7, and what the backtest would (falsely) show if you used it anyway
• You can say why MASE beats MAPE (zero/near-zero actuals; asymmetry) in one sentence

Extensions

• Recursive forecasting + error-compounding comparison vs direct
• Prediction intervals via quantile GBM (loss="quantile"); calibration check
• Intermittent demand: Croston's method vs the GBM on a sparse series
• Hierarchical reconciliation: forecast SKU and store level, reconcile bottom-up vs MinT, measure coherence

Interview Q&A

Q: Your model's backtest MASE is 0.4 but production forecasts are garbage. Top suspects? Leakage in features (a lag < horizon, or a covariate that's actually observed-after — audit build_features against the deployment timeline); backtest folds that don't match the production cadence (retraining weekly in backtest, monthly in prod); or the series regime-shifted and the expanding window is dominated by stale history. The lag-≥-horizon audit is always first.

Q: Why is random cross-validation invalid for time series? Training folds would contain the future of test folds — autocorrelation means the model effectively memorizes the neighborhood of each test point. Rolling-origin keeps every train set strictly before its test window, matching how the model will actually be used: trained on the past, scored on the future.

Q: When does seasonal-naive beat your ML model, and what do you do about it? Short histories (features can't be estimated), strong stable seasonality with little else (nothing left to learn), or regime changes the lags haven't caught up with. The response is not "more model" — it's ensembling toward the baseline (shrink predictions to seasonal-naive when fold-level MASE ≈ 1) and investing in covariates (promos, holidays) that carry genuinely new information.

References

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed., free online) — ch. 5 (the toolbox), ch. 13 (practical issues)
• Hyndman & Koehler, Another look at measures of forecast accuracy (2006) — the MASE paper
• Makridakis et al., The M5 Accuracy Competition (2022) — where GBMs-with-good-features beat everything
• sktime and statsforecast — the industrial versions of this harness

Phase 08 — Training Orchestration & MLOps

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "deep expertise in MLOps and platform engineering, including model registries, feature stores, CI/CD, infrastructure as code, experiment tracking, and automated validation"

Why This Phase Exists

Every ML platform — Airflow+MLflow stacks, Kubeflow, Vertex, SageMaker Pipelines — is the same five ideas wearing different YAML: a DAG of tasks, content-addressed caching so unchanged work is never redone, retries because infrastructure flakes, an artifact ledger answering "where did this model come from," and gates that block bad artifacts from promotion. Engineers who have only used these platforms configure them by superstition; engineers who have built one configure them by mechanism. The flagship lab builds one in ~300 lines.

The WARMUP covers the surrounding practice: experiment tracking, model registries and promotion lifecycles, CI/CD for ML (what "test" even means when the artifact is a model), and reproducibility discipline.

Concepts

• DAGs: topological execution, cycle detection, fan-in/fan-out
• Content-addressed caching: hash(code, params, upstream artifact hashes) → skip; cache invalidation as dependency propagation
• Idempotency and retries; transient vs permanent failure
• Artifact lineage: model → data → code provenance, queryable
• Experiment tracking: params/metrics/artifacts per run; what makes runs comparable
• Model registry: stages (candidate → staging → production), promotion gates
• CI/CD for ML: data validation, training smoke tests, quality gates, golden predictions; the train-on-merge pattern
• Reproducibility: seeds, pinned environments, data snapshots/versions

Labs

Lab 01 — Mini Pipeline Orchestrator (flagship, implemented)

→ Lab folder: lab-01-mini-orchestrator/

Extension Project A — Experiment Tracker (spec)

A track(run) context manager logging params/metrics/artifacts to a SQLite ledger; a compare(run_a, run_b) diff report; a leaderboard query. Then map every feature to its MLflow equivalent and write the one-pager "what MLflow actually stores."

Extension Project B — Model CI Pipeline (spec)

A GitHub Actions (or local make ci) pipeline for one of the earlier phases' models: data-contract validation (Phase 01's library!) → training smoke test (tiny data, must converge) → quality gate (metric ≥ baseline from a stored golden) → package artifact with lineage metadata. Deliberately break each stage; verify each gate fires.

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide

Deliverables Checklist

• Lab 01 implemented; all 13 tests pass
• Extension A tracker or documented MLflow equivalent
• Extension B pipeline with all gates demonstrated firing

Warmup Guide — Training Orchestration & MLOps

Zero-to-expert primer for Phase 08: the five mechanisms inside every ML platform — DAGs, content-addressed caching, retries, lineage, and promotion gates — plus the surrounding practice: experiment tracking, registries, and CI/CD for models.

Table of Contents

• Chapter 1: What MLOps Actually Is
• Chapter 2: Pipelines as DAGs
• Chapter 3: Content-Addressed Caching
• Chapter 4: Idempotency and Retries
• Chapter 5: Artifact Lineage
• Chapter 6: Experiment Tracking
• Chapter 7: Model Registries and Promotion
• Chapter 8: CI/CD for ML
• Chapter 9: Reproducibility as a System Property
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What MLOps Actually Is

Strip the vendor marketing and MLOps is one sentence: applying software delivery discipline to artifacts that depend on data as well as code. The "as well as data" clause is what makes it a distinct field:

• Code changes are diffable and reviewable; data changes are silent and continuous — so validation must be runtime, not review-time (Phase 01's contracts, Phase 12's drift detection).
• A binary either passes tests or doesn't; a model is statistically good — so gates are metric thresholds with baselines, not assertions (Phase 09 of the model-accuracy track built exactly this for accuracy; this phase generalizes it).
• Builds are seconds; training runs are hours — so caching and incremental recomputation are economics, not conveniences (Chapter 3).

The platform zoo (Airflow, Dagster, Kubeflow, MLflow, SageMaker, Vertex) all implement the same five mechanisms this phase teaches. Learn the mechanisms; the YAML dialects follow in an afternoon.

Chapter 2: Pipelines as DAGs

From zero: a pipeline is tasks + dependencies = a directed graph; requiring it be acyclic (a DAG) is what makes execution decidable — a cycle means "A needs B needs A," which is a specification bug, and your orchestrator must detect and reject it, not hang (the lab tests this).

The mechanics:

• Topological sort produces a valid execution order: repeatedly emit a node whose dependencies are all emitted (Kahn's algorithm: maintain in-degrees, pop zeros). Multiple valid orders exist; independent tasks are the parallelism opportunity.
• Fan-out/fan-in: one upstream feeding N independent tasks (per-country training) then merging — the shape that makes DAGs strictly more useful than scripts.
• Tasks are functions of their inputs — the discipline that makes everything else (caching, retries, lineage) possible. Hidden state (a task reading a global path another task mutates "by convention") is the original sin of pipeline design; every input must be an explicit, declared edge.
• Real-system mapping: Airflow tasks declare >> dependencies (scheduling-only edges — data passing is on you); Dagster's assets and Kubeflow's typed inputs/outputs make the data edges first-class, which is the modern direction and the lab's design.

Chapter 3: Content-Addressed Caching

The economic core of every serious pipeline system (also: Bazel, Nix, DVC, dbt):

The key insight: a task's output is fully determined by (its code, its parameters, its inputs). So cache by content:

$$\text{cache_key} = H(\text{task code version} ,|, \text{params} ,|, H(\text{input}_1) ,|, \cdots)$$

Key consequences, each of which the lab turns into a test:

• Hash, not timestamps: timestamps (Make's method) lie under re-runs, clock skew, and re-materialized-but-identical data. Content hashes are ground truth: same bytes, same key, skip.
• Invalidation propagates automatically: change one param in task B → B's key changes → B re-runs → B's output hash changes → downstream keys change → exactly the affected subgraph re-executes. Nothing upstream re-runs. This is the whole feature, derived from the key structure — no invalidation logic exists anywhere.
• The code-version term: if the key omits the task's code identity, editing the logic silently serves stale artifacts — the most common homemade-cache bug (production sightings: weekly). Real systems hash the function source, the container digest, or demand an explicit version= bump (the lab's approach).
• Nondeterminism poisons caching: a task with an unseeded RNG produces different outputs for the same key — downstream caching becomes incoherent. Determinism (Ch. 9) is a caching requirement, not just a science one.

Chapter 4: Idempotency and Retries

Infrastructure fails transiently (OOM, spot preemption, network); orchestrators retry. Retrying is only safe if tasks are idempotent — running twice has the same effect as once:

• Write outputs to a staging location, then atomically rename/commit — a task killed mid-write must not leave a half-artifact that a retry then treats as done (or that downstream reads). This is the same atomic-commit pattern as the PMC track's release staging repos and Kafka's log: stage, verify, publish.
• Classify failures: transient (timeout, 5xx, OOM-on-noisy-neighbor) → retry with backoff; permanent (validation error, missing input, code bug) → fail fast, no retry. Retrying a permanent failure wastes hours and hides the bug in retry noise; the lab's retry policy takes an is_transient classifier for exactly this reason.
• Bounded retries with backoff + jitter — unbounded retries turn one incident into a queue flood (the thundering-herd lesson every distributed-systems curriculum repeats because every on-call relearns it).

Chapter 5: Artifact Lineage

The question lineage answers is the one every incident review asks: "this model is misbehaving — what data, code, and params produced it?" And its inverse: "this upstream table was corrupted Tuesday — which models are tainted?"

The mechanism (the lab's ledger): every task execution appends an immutable record — (artifact_hash, task, code_version, params, input_artifact_hashes, timestamp). Lineage queries are then graph walks: upstream closure of a model = everything it was made from; downstream closure of a dataset = everything made from it (the taint query). Append-only matters: ledgers you can edit are ledgers you can't trust during an incident — the same audit-trail discipline as Phase 10's vote parser in the PMC track.

Production mapping: MLflow's run/artifact metadata, OpenLineage events, SageMaker/Vertex lineage tracking — all this record, at enterprise scale.

Chapter 6: Experiment Tracking

Training is experimentation; untracked experiments are unrepeatable anecdotes. The minimum viable record per run:

• Params (all of them — including the ones you "didn't change": defaults drift), metrics (logged over steps, not just finals — curves diagnose what finals hide), artifacts (the model, the eval report, the confusion matrix PNG), environment (git SHA, package lock hash, data version), seed.
• The discipline that makes runs comparable: same eval data + same metric code = comparable; anything else is two different experiments. Pin the eval set by content hash in the run record (the Phase 09 model-accuracy lesson, recurring).
• What separates tracking from a CSV: queryability ("all runs on dataset X with lr<0.01, by AUC") and artifact linkage (click from metric to model). That's the whole product surface of MLflow/W&B — Extension A has you build it in 100 lines to demystify it.

Chapter 7: Model Registries and Promotion

A registry is a state machine over model versions (the PMC track's issue-FSM pattern, applied to artifacts):

candidate → staging → production → archived

• Transitions carry gates: candidate→staging requires the offline quality gate (metric ≥ baseline on the pinned eval set, fairness slices pass); staging→ production requires online verification (shadow or canary — Phase 09 of this track); any→archived records why (superseded / regressed / deprecated).
• Versions are immutable; promotion moves pointers, never rewrites artifacts — rollback is then a pointer move too (the deployment-safety property that matters at 2am).
• The registry is the deployment interface: serving infrastructure pulls "production" by alias, not by version number — which is what makes promotion and rollback deployment-free operations.

Chapter 8: CI/CD for ML

What "CI" means when the artifact is a model — the four test layers (Extension B builds them):

1. Code tests — ordinary pytest; fast; on every commit.
2. Data validation — contracts on training inputs (schema, ranges, volumes, class balance); on every pipeline run. A training run on garbage data that succeeds is the worst outcome — it produces a plausible-looking bad model.
3. Training smoke test — train on a tiny fixture dataset in CI (<2 min): does the loss decrease, does the pipeline run end-to-end, does the artifact serialize/deserialize? Catches the 80% of training bugs that need no GPU.
4. Quality gates — full training (scheduled or on-merge): metric vs pinned baseline with significance (Phase 11's statistics), golden-prediction diffs on a fixed probe set (catches "metric same, behavior different"), latency/size budget checks.

CD then = registry promotion + progressive rollout (canary/shadow, Phase 09) + automated rollback on guardrail breach (Phase 12). The pattern sentence for interviews: models ship like code, but with statistical gates where code has assertions, and with data validation where code has type checks.

Chapter 9: Reproducibility as a System Property

Reproducibility is engineered, layer by layer, in descending order of pain caused when missing:

1. Data: snapshot or version every training input (DVC, lakehouse time-travel, or plain immutable dated paths). "The table changed since training" is the #1 irreproducibility cause and the #1 lineage query.
2. Code: git SHA in every run record; no uncommitted-state training (the tracker should refuse or loudly tag dirty runs).
3. Environment: lock files + container digests (Phase 01 Ch. 9).
4. Seeds: one seed in the config, fanned to every RNG; logged.
5. Hardware nondeterminism (GPU atomics, cuDNN autotuning): bound it — deterministic flags where affordable, or tolerance-based golden tests where not. Bit-exactness is sometimes the wrong goal; statistically indistinguishable with a stated tolerance is the honest one.

Lab Walkthrough Guidance

Lab 01 — Mini Orchestrator, suggested order:

1. Task + Pipeline.add_task + topological sort with cycle detection — get ordering tests green first; Kahn's algorithm, and a cycle raises with the cycle named (error messages as documentation).
2. Execution without caching: run tasks in order, pass declared inputs, record results. The execution-counter test fixture shows you exactly what runs.
3. Cache keys per Chapter 3's formula (the provided stable_hash handles hashing; your job is what goes in the key). Make the hit/miss tests pass, then the propagation test — if propagation fails, your key forgot upstream output hashes.
4. Retries: the flaky-task fixture fails N times then succeeds; implement bounded retries with the transient/permanent classifier; the give-up test wants the original exception surfaced, not a retry wrapper.
5. Ledger + lineage: append execution records; implement lineage(artifact) as the upstream graph walk; the taint test walks downstream.

Throughout: tasks are pure functions in the tests — notice how much machinery that purity buys you; that's Chapter 2's discipline made visceral.

Success Criteria

You are ready for Phase 09 when you can, from memory:

1. Run Kahn's algorithm on paper and state why cycles must be rejected loudly.
2. Write the cache-key formula and derive invalidation-propagation from it; name the code-version trap.
3. Define idempotency, the stage-then-commit pattern, and the transient/permanent retry split.
4. Sketch the lineage ledger schema and both closure queries (provenance, taint).
5. Recite the four CI layers for ML with what each catches.
6. Map each lab mechanism to its Airflow/MLflow/DVC equivalent in one line each.

Interview Q&A

Q: Design CI/CD for a churn model retrained weekly. Layers: code tests on PR; data contracts on the weekly snapshot (schema, volume, label rate — block training on violation); training smoke test on a fixture in PR CI; the weekly run gated by metric-vs-baseline with significance on a pinned eval set plus golden-prediction diff; artifact to registry as candidate; shadow against production for N days (Phase 09); auto-promote on parity, page on divergence; rollback = registry pointer move. Lineage recorded at every step so a bad weekly snapshot is traceable to the models it touched. Each clause of that answer is a chapter of this guide.

Q: Your nightly pipeline re-trains everything every night even when nothing changed. Fix it properly. Content-addressed caching: key each task on code version + params + input content hashes; nightly runs then skip every task whose key is unchanged — the pipeline becomes incremental automatically, and a one-table change re-runs exactly its downstream closure. The prerequisites to call out: tasks must be pure (declared inputs only) and deterministic (seeded), else the cache is incoherent — which is usually the actual engineering work hiding inside "add caching."

Q: Two runs with identical configs produced different models. Walk through it. Order of suspects: data changed underneath (no snapshot/version pinning — check lineage first), unseeded randomness (init, shuffling, dropout, augmentation — audit the seed fan-out), environment drift (library upgrade between runs — compare lock hashes), then GPU nondeterminism (atomics/cudnn autotune — bounded, small, and only credible once the first three are excluded; people blame it first because it's blameless, but it's rarely the real cause).

References

• Sculley et al., Hidden Technical Debt in Machine Learning Systems — the founding document
• Google: MLOps — Continuous delivery and automation pipelines in ML — the maturity-levels paper
• The ML Test Score (Breck et al., 2017) — the checklist behind Chapter 8
• Airflow concepts, Dagster assets, MLflow tracking & registry, DVC pipelines — read each after the lab and map mechanisms
• OpenLineage — the lineage record, standardized
• Kahn, Topological sorting of large networks (CACM 1962) — the algorithm

Lab 01 — Mini Pipeline Orchestrator

Phase: 08 — Training Orchestration & MLOps | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours

Airflow, Dagster, Kubeflow, DVC, and dbt are the same five mechanisms in different YAML. Build them once — DAG execution, content-addressed caching, retries, ledger, lineage — and you'll configure any of them by mechanism instead of superstition.

What you build

• Pipeline.topo_order — Kahn's algorithm with cycle detection that names the cycle
• cache_key — H(version ‖ params ‖ upstream artifact hashes): the formula from which incremental recomputation falls out with zero invalidation logic
• run — topological execution with cache hits, including early cutoff (re-run task produces identical bytes → downstream stays cached)
• _run_with_retries — bounded retries with a transient/permanent classifier and the original exception surfaced on give-up
• Ledger + lineage — immutable execution records; lineage() (provenance: what made this model?) and downstream_of() (taint: this input was corrupted — what's affected?)

Key concepts

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (14 tests)
python solution.py                          # caching demo: run, re-run, param change

Suggested TODO order

1. add_task + topo_order — get the DAG tests green
2. run without caching — make the execution-counter fixture show all tasks running
3. cache_key + the cache path in run — hit/miss, propagation, early-cutoff tests
4. _run_with_retries — the Flaky fixture drives all three retry tests
5. latest_record / lineage / downstream_of

Success criteria

• All 14 tests pass
• You can explain early cutoff and why it's correct (and name the build systems that do it)
• You can state what breaks if tasks are impure (undeclared inputs) or nondeterministic (unseeded RNG) — and therefore why those disciplines are caching requirements
• You can map each mechanism to Airflow/MLflow/DVC equivalents

Extensions

• Parallel execution of independent tasks (ThreadPoolExecutor over each topo "level")
• Persistent cache + ledger (SQLite) surviving process restarts
• Stage-then-commit artifact writes (atomic rename) and a kill-mid-task crash test
• --force task and --downstream-of task re-run flags (the operational verbs)

Interview Q&A

Q: How does your cache know to re-run downstream tasks when an upstream changes? It doesn't "know" — the knowledge is structural. Downstream keys include upstream output hashes; an upstream change that alters its output alters those hashes, which alters downstream keys, which miss the cache. And if the upstream re-runs but produces identical bytes, downstream keys are unchanged and correctly skip (early cutoff). All invalidation behavior derives from the key formula.

Q: A task succeeded but wrote a corrupt artifact, and three downstream models shipped before anyone noticed. What do you need? The taint query: downstream_of(task) over the lineage ledger gives the affected closure; the ledger's artifact hashes identify exactly which model versions consumed the corrupt hash; the registry (Phase 08 WARMUP Ch. 7) maps those to deployments. Then prevention: a data-contract gate (Phase 01) between that task and its consumers, so corruption fails the pipeline instead of flowing.

References

• DVC pipelines — content-addressed ML pipelines, productionized
• Bazel: correctness via content hashing and early cutoff
• Dagster software-defined assets — data edges as first-class
• Kahn, Topological sorting of large networks (CACM 1962)
• OpenLineage spec — the ledger, standardized

Phase 09 — Model Serving Platform

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "model-serving infrastructure … performance, reliability, latency, and cost efficiency in live environments"

Why This Phase Exists

Serving is where ML meets SLOs. The model is a function; the system around it decides whether p99 is 40 ms or 4 s, whether a GPU runs at 8% or 80% utilization, and whether a bad model version reaches 1% or 100% of users. The three levers that dominate every serving conversation — batching (throughput vs latency), caching (work avoidance), and progressive delivery (shadow/canary) — are exactly what the flagship lab builds, against a simulated clock so the tradeoffs are measurable and the tests deterministic.

The WARMUP covers the surrounding architecture: serving patterns, latency budgets and percentile thinking, autoscaling and cost, and the LLM-serving specifics this track inherits from the llm-inference track.

Concepts

• Serving patterns: online request/response, batch scoring, streaming, async/queue
• Latency as a distribution: p50/p95/p99, tail amplification under fan-out, budgets
• Dynamic batching: amortizing per-call overhead; max-batch-size and max-wait knobs; the latency-throughput frontier
• Caching tiers: exact-match request cache, feature cache (Phase 03's online store), embedding cache; TTLs and staleness contracts
• Progressive delivery: shadow (mirror traffic, compare offline), canary (small live fraction + guardrails), blue/green; automated rollback
• Cost: GPU utilization economics, right-sizing, autoscaling signals (QPS vs queue depth vs latency), cold starts

Labs

Lab 01 — Batching Server Simulator (flagship, implemented)

→ Lab folder: lab-01-batching-server/

Extension Project A — Real Async Server (spec)

Wrap an actual model (Phase 05's ranker) in FastAPI with an asyncio dynamic batcher; load-test with locust/hey; reproduce the simulator's frontier on real hardware; find the knee point.

Extension Project B — Cost Model (spec)

Build the QPS × latency-budget × instance-type cost calculator: given a traffic profile and SLO, output the cheapest fleet (CPU vs GPU, batch sizes, replica counts, with/without cache at assumed hit rates). Defend three scenarios in a one-pager.

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide

Deliverables Checklist

• Lab 01 implemented; all 13 tests pass
• Extension A latency-throughput frontier measured on real hardware
• Extension B cost memo

Warmup Guide — Model Serving Platform

Zero-to-expert primer for Phase 09: the system around the model — serving patterns, latency distributions, dynamic batching, caching tiers, progressive delivery, and the cost economics of inference fleets.

Table of Contents

• Chapter 1: The Four Serving Patterns
• Chapter 2: Latency Is a Distribution
• Chapter 3: Dynamic Batching
• Chapter 4: Caching Tiers
• Chapter 5: Progressive Delivery — Shadow, Canary, Rollback
• Chapter 6: Autoscaling and Cost
• Chapter 7: The Serving Stack in Practice
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Four Serving Patterns

Choose the pattern before the framework — most "serving problems" are pattern mismatches:

The senior move the JD calls "cost efficiency": precompute everything precomputable. A recommender that batch-scores candidates nightly and serves from a key-value lookup beats an online-scoring design by 10–100× in cost — and the hybrid (precomputed candidates + light online re-rank, Phase 06's architecture) is the industry default for exactly this reason.

Chapter 2: Latency Is a Distribution

• Report p50 / p95 / p99, never means: latency is right-skewed, and the mean is dominated by the tail you're trying to manage. SLOs bind on percentiles ("p99 < 100 ms") because users experience the tail.
• Tail amplification under fan-out: a request touching N services each with p99 = 100 ms has P(any one is slow) = 1 − 0.99ᴺ — at N = 10, ~9.6% of requests hit a 100 ms+ dependency. Microservice-heavy ML stacks (feature fetch + model + ranker + filters) live or die by this arithmetic; it's why feature-store reads (Phase 03) get single-digit-ms budgets.
• Measurement honesty (the Phase 07 model-accuracy lessons apply verbatim): warm up before measuring, percentiles need enough samples (p99 of 50 requests is noise), watch for coordinated omission (a load generator that waits for slow responses under-samples the tail — fixed-rate arrival schedules fix this; the lab's simulator uses arrival schedules for exactly this reason).
• The latency budget exercise: decompose the SLO end to end — network 5 ms + feature fetch 10 ms + inference 25 ms + post-processing 5 ms + headroom — and assign the budget per component. Components without budgets grow until the SLO breaks.

Chapter 3: Dynamic Batching

Why batching wins: every inference call pays fixed overhead (framework dispatch, memory movement, kernel launches — and on GPUs, underutilized parallelism). A model whose cost is c_fixed + n · c_item per call has per-item cost c_fixed/n + c_item — batching amortizes the fixed term. On GPUs the effect is dramatic (the hardware wants 10³+ parallel items — the Phase 07/08 model-accuracy bandwidth story); on CPUs it's still real via vectorization (Phase 01's numpy lesson).

The mechanism (the lab's core): requests arrive asynchronously; a batcher queues them and flushes when either:

• the batch reaches max_batch_size (the throughput knob), or
• the oldest queued request has waited max_wait (the latency knob).

The tradeoff, quantified: batching adds up to max_wait of queueing delay to buy throughput. The frontier — p99 vs sustainable QPS as you sweep the knobs — is the deliverable graph of both the lab and any real tuning exercise. Two regimes to recognize: at low traffic, batches rarely fill → latency cost ≈ max_wait, little throughput gain (consider batch size 1); at high traffic, batches fill instantly → throughput gain ≈ full, queueing delay dominated by capacity, and max_wait barely binds.

Setting max_wait: from the latency budget (Ch. 2) — if inference has a 40 ms budget and the model takes 25 ms at your batch size, max_wait ≤ ~10 ms. It's a derived number, not a vibe.

(LLM serving replaces this with continuous batching at iteration granularity — the llm-inference track Phase 09 and model-accuracy Phase 08 cover it; the queueing logic you build here is its foundation.)

Chapter 4: Caching Tiers

Work avoidance beats work optimization. The tiers, outermost first:

1. Exact-match response cache: hash(model version, canonicalized input) → response. Hit rates are workload-dependent: high for popular-item recsys and repeated searches, ~zero for unique-per-user inputs. Always key on model version — serving stale predictions from the old model after a deploy is the classic cache incident.
2. Feature cache / online store (Phase 03): the most load-bearing tier — it's why p99 feature reads stay single-digit ms.
3. Embedding cache: user/item embeddings change slowly; cache with TTL matched to the refresh cadence; pairs with the ANN index (Phase 04).
4. Intermediate caches (KV cache in LLMs, candidate lists in recsys): inside the model server itself.

The discipline for every tier: an explicit staleness contract (TTL chosen against the upstream change rate), an invalidation story (event-driven beats TTL where possible — Phase 02's streams), and hit-rate monitoring (a silently degraded cache looks exactly like a capacity problem).

Chapter 5: Progressive Delivery — Shadow, Canary, Rollback

Models are statistically validated, so deployment must be too:

• Shadow (mirroring): the new model receives a copy of live traffic; its outputs are logged, not served. Compares: output distributions, latency, error rate, divergence-vs-current on identical inputs. Zero user risk; catches engineering-level breakage and gross prediction shifts. Cannot measure outcome impact (nobody acted on its predictions) — that needs the canary/A-B.
• Canary: route a small fraction (1% → 5% → 25% → 100%) of live traffic; monitor guardrails — error rate, latency, and the business guardrail metrics — with statistical care (a 1% canary needs hours-to-days for significance on subtle metrics; Phase 11's power math applies). The lab's CanaryRouter implements the decision: promote when healthy after the soak, auto-rollback on guardrail breach — rollback must be a pointer move (Phase 08's registry), not a build.
• Blue/green: full parallel fleet, instant cutover/cutback — simpler, costlier, no gradual exposure.
• The composition that mature teams run: shadow first (engineering safety), then canary (outcome safety), then the A/B test (outcome measurement — Phase 11). Three different questions; three mechanisms.

Chapter 6: Autoscaling and Cost

• The GPU economics fact: an idle GPU costs the same as a busy one. Fleet cost ≈ replicas × instance price; replicas ≈ peak QPS / per-replica throughput — which is why batching (throughput ↑) and caching (QPS ↓) are cost levers first and latency levers second. Utilization below ~30% on a GPU fleet is a redesign signal (batch more, consolidate models, or move to CPU).
• Autoscaling signals, in order of usefulness for ML serving: queue depth / concurrency (leading, direct), latency (lagging — you scale after users already hurt), CPU/GPU utilization (misleading for memory-bound or bursty workloads), QPS (only with stable per-request cost).
• Cold starts: model loading (seconds–minutes for large models) makes scale-from-zero brutal — mitigations: min-replicas ≥ 1, model pre-warming, weights on fast local storage, smaller models (the whole model-accuracy track's reason for existing, from the platform side).
• Right-sizing heuristic (Extension B): compute per-replica sustainable QPS at the SLO from the measured frontier (Ch. 3), then fleet = peak-QPS / that, +30% headroom, then compare instance types by $/sustained-QPS — a one-afternoon exercise that routinely halves bills.

Chapter 7: The Serving Stack in Practice

Mapping the concepts to the tools you'll meet:

• Model servers: TorchServe / Triton Inference Server (dynamic batching built in — the knobs are literally max_batch_size/max_queue_delay, i.e., the lab), vLLM for LLMs (continuous batching), ONNX Runtime for portability.
• The two-service shape: a thin business-logic API (auth, feature fetch, post-processing) in front of a model server — keeps GPU fleets dense and independently scalable from CPU logic.
• Interfaces: REST for compatibility, gRPC for internal hops (latency + streaming), batched endpoints for bulk.
• What "reliability" means here: timeouts + fallbacks at every hop (a ranking timeout falls back to popularity order, not an error page — degrade, don't fail), load shedding before collapse, and the observability of Phase 12 wired in from day one.

Lab Walkthrough Guidance

Lab 01 — Batching Server Simulator, suggested order:

1. percentile + LatencyRecorder — exact interpolation method per the docstring; the hand-computed test pins it.
2. SimulatedClock + request/arrival plumbing (provided) — read it; the simulator is discrete-event, so tests are deterministic and fast.
3. DynamicBatcher.run — the flush policy. Get the two invariants right: no batch exceeds max_batch_size; no request's batch-start exceeds its arrival + max_wait. Then latency accounting: completion = batch start + model time; request latency = completion − arrival.
4. The frontier experiment (sweep) — run the provided workload across knob settings; the tests assert the qualitative shape (batching ≥5× throughput at bounded p99 cost vs batch-1).
5. CanaryRouter — deterministic hash-based routing, per-arm metrics, the guardrail decision (promote / rollback / continue).

Success Criteria

You are ready for Phase 10 when you can, from memory:

1. Match the four serving patterns to four product scenarios and name the precompute opportunity in each.
2. Do the fan-out tail arithmetic and decompose an end-to-end latency budget.
3. Explain both batching knobs, derive max_wait from a budget, and describe both traffic regimes of the frontier.
4. Recite the caching tiers with each one's staleness contract.
5. Distinguish shadow / canary / A/B by the question each answers.
6. List autoscaling signals in usefulness order and the cold-start mitigations.

Interview Q&A

Q: Your p99 is 3× the SLO but p50 is fine. Where do you look? Tail-specific causes, in order: queueing under bursts (look at queue-depth percentiles — the mean hides the bursts), batching max_wait set too high for the low-traffic regime, GC/checkpoint pauses in the server process, cache-miss path (p50 = hits, p99 = misses — check hit rate), fan-out amplification (one slow dependency), and noisy neighbors on shared infra. The p50/p99 split itself is the diagnostic: the system is healthy on the common path and pathological on a minority path — find which minority.

Q: When would you NOT add dynamic batching? Low-traffic services (batches don't fill — you pay max_wait for nothing), models where per-item cost dominates (no fixed term to amortize — check the measured cost curve first), strict-latency paths whose budget can't fund any queueing, and sequence models with high padding waste under heterogeneous lengths (bucket or use continuous batching instead). The general answer: batching buys throughput with latency — if you don't need the throughput or can't spend the latency, don't.

Q: Design the rollout for a new fraud model where false negatives cost real money. Shadow first — weeks, not days: compare score distributions and disagreement cases against the incumbent on identical traffic; have analysts review the high-disagreement set (fraud labels lag, so offline outcome comparison needs patience). Then a small canary with conservative guardrails — block rate, decline rate, manual-review queue depth — and asymmetric rollback bias (rollback on any doubt; the cost asymmetry demands it). Promote on canary parity plus the lagged label readout. And the org answer: pre-agree the guardrail thresholds and decision owner before the rollout starts — mid-rollout threshold negotiation is how bad models reach 100%.

References

• Triton Inference Server: dynamic batching docs — the lab's knobs, productionized
• Dean & Barroso, The Tail at Scale (CACM 2013) — the fan-out arithmetic and tail-tolerance patterns
• Tene, How NOT to Measure Latency — coordinated omission, canonically
• vLLM and the llm-inference track Phase 09 — continuous batching
• Kubernetes HPA + KEDA — queue-depth-driven autoscaling in practice
• Google SRE Book, ch. 21–22 — load shedding and handling overload

Lab 01 — Batching Server Simulator

Phase: 09 — Model Serving Platform | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Triton's dynamic batcher has two knobs: max_batch_size and max_queue_delay. This lab builds the mechanism behind them as a discrete-event simulator — so the latency-throughput tradeoff stops being folklore and becomes a table you computed.

What you build

• percentile / LatencyRecorder — honest tail measurement (linear-interpolated order statistics; the p50/p99 vocabulary every SLO is written in)
• DynamicBatcher — the canonical flush policy: seal on size-full OR oldest-waits- max_wait; FIFO single-server simulation with full latency accounting
• sweep — the latency-throughput frontier across knob settings
• CanaryRouter — sticky hash-based traffic splitting + guardrail decision logic (continue / promote / rollback)

The two regimes (the tests make you produce both)

Reference frontier (2 000 requests at ~1 req/ms, model cost 8 + n ms):

        config  thr (req/ms)     p50     p99  avg batch
      (1, 0.0)         0.111  8020.2 15811.3        1.0
    (32, 20.0)         0.715   425.1   769.1       20.6

(Saturated: the queue itself dominates latency — batching here reduces p99 6.4× while multiplying throughput, because the alternative is an exploding queue.)

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (16 tests)
python solution.py                          # the frontier table

Suggested TODO order

1. percentile — pin against the hand-computed test
2. DynamicBatcher.run — get the invariants first (size cap, wait cap), then the hand-computed latency test, then conservation
3. sweep — both regime tests
4. CanaryRouter — routing stickiness, then the three-way decision

Success criteria

• All 16 tests pass
• You can read the frontier table aloud: which regime each row is in and why
• You can derive max_wait from a latency budget
• You can explain why canary routing must be sticky and deterministic (same user on both arms contaminates the comparison; non-reproducible routing is undebuggable)

Extensions

• Multiple replicas (k servers) + the queue-depth autoscaling signal
• Priority lanes: interactive vs batch traffic sharing one server
• Coordinated-omission demonstration: closed-loop vs open-loop load generation on the same system, compare measured p99
• Shadow mode: mirror every request to a second model; divergence report

Interview Q&A

Q: Batching raised your p99 at low traffic. Why, and what do you do? Underfull batches wait the full max_wait — pure latency cost with no throughput benefit (the low-traffic regime test). Fixes: adaptive max_wait (shrink when queue empty), flush-on-idle policies, or traffic-aware config (batch only above a QPS threshold). The general lesson: batching knobs are workload-dependent; tune against the production arrival process, not a synthetic saturated one.

Q: Your canary shows 0.3% higher error rate. Promote or rollback? First: is it significant? 0.3% on a few hundred samples is noise — compute the two-proportion test (Phase 11) or keep soaking (continue). Second: is it the model or the infrastructure (cold cache, new container image)? Check error types. Third: pre-agreed thresholds decide — if 0.3% exceeds the agreed max_error_delta with significance, rollback without negotiation; renegotiating thresholds mid-canary is how bad models ship.

References

• Triton dynamic batching — the productionized version of this lab
• Dean & Barroso, The Tail at Scale (CACM 2013)
• Tene, How NOT to Measure Latency — coordinated omission
• Phase 11 — Online Experimentation — the statistics the canary decision borrows

Phase 10 — GenAI in Production: RAG, Guardrails & Hallucination Analysis

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "generative AI applications using embeddings, vector databases, RAG pipelines, agent workflows, prompt engineering, and guardrails" plus "hallucination analysis"

Why This Phase Exists

The llm-inference track teaches how RAG and agents work; this phase teaches what it takes to put them in front of enterprise users — which is a different discipline. Production GenAI is defined by its failure management: hallucinations measured rather than anecdotal, prompt injection treated as an attack surface, outputs validated against schemas before any downstream system trusts them, and an evaluation harness that runs on every change like a test suite.

The flagship lab builds that harness: groundedness scoring, injection canaries, a policy engine, and structured-output validation — deterministic, model-free (rule-based scorers + a mock LLM), so the system is what's tested. Swapping in a real LLM and LLM-as-judge scorers is the extension.

Concepts

• The production RAG loop: retrieve → assemble context (budgeted) → generate → validate → respond-or-fallback
• Groundedness/faithfulness: is every claim in the answer supported by the retrieved context? Sentence-level support scoring; the unsupported-claim rate as the hallucination metric for RAG
• Prompt injection: direct and indirect (poisoned documents); defense layers — privilege separation, input/output filtering, instruction hierarchy; canary-based regression testing
• Structured output: schema-constrained generation, validate-then-repair loops
• Refusal policy: what the system must not answer; topical guardrails
• Eval harnesses for GenAI: golden sets, canary suites, LLM-as-judge (and its biases), regression gating (Phase 12 wires this into CI)

Labs

Lab 01 — Guardrail & Eval Harness (flagship, implemented)

→ Lab folder: lab-01-guardrail-harness/

Extension Project A — Real-Model Harness (spec)

Swap the mock LLM for a real one (any provider); replace the rule-based groundedness scorer with an NLI model or LLM-as-judge; measure judge-vs-rule agreement on 50 hand-labeled cases; document the judge's failure modes (position bias, verbosity bias, self-preference).

Extension Project B — Indirect-Injection Red Team (spec)

Poison a document corpus with adversarial instructions; measure the attack success rate through your RAG pipeline with and without each defense layer; produce the defense-in-depth report.

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass
• Extension A judge-agreement study
• Extension B red-team report

Warmup Guide — GenAI in Production

Zero-to-expert primer for Phase 10: turning RAG demos into systems enterprises can trust — hallucination measurement, prompt-injection defense, structured outputs, refusal policy, and the evaluation harness that gates every change.

Table of Contents

• Chapter 1: Why GenAI Production Is Failure Management
• Chapter 2: The Production RAG Loop
• Chapter 3: Hallucination, Measured
• Chapter 4: Prompt Injection — the Attack Surface
• Chapter 5: Structured Output and the Repair Loop
• Chapter 6: Refusal Policy and Topical Guardrails
• Chapter 7: Evaluation Harnesses and LLM-as-Judge
• Chapter 8: Cost, Latency, and the Cascade Pattern
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why GenAI Production Is Failure Management

A classical ML model fails quantifiably — a wrong class, a bad score, measurable offline. An LLM application fails fluently: confident prose containing fabricated facts, leaked instructions, or malformed JSON that takes down the consumer service. Three engineering consequences define this phase:

1. Validation moves to the output side. You cannot test an LLM into correctness ahead of time; you wrap every response in checks (groundedness, schema, policy) and route failures to fallbacks.
2. Quality becomes a measured rate, not a property. "Does it hallucinate?" is the wrong question; "what is the unsupported-claim rate on our golden set, and did this prompt change move it?" is the right one (Ch. 3, 7).
3. Inputs are adversarial. The prompt is an API that users — and documents — can program. Injection is an attack surface with defense-in-depth, not a quirk (Ch. 4).

The JD's phrase "hallucination analysis … and safety controls" is this chapter, operationalized in the lab.

Chapter 2: The Production RAG Loop

The demo loop is retrieve → stuff → generate. The production loop:

query → [input guards] → retrieve (hybrid, Phase 04) → rerank → 
context assembly (token budget, source attribution, freshness) →
generate (schema-constrained where possible) →
[output guards: groundedness, policy, schema] → respond | fallback | escalate

The load-bearing differences from the demo:

• Context assembly is a budgeted ranking problem: which chunks, in what order (LLMs attend better to start/end — "lost in the middle"), with per-source attribution markers so groundedness checking (Ch. 3) and citation are possible. Retrieval quality bounds answer quality — most "hallucination" bugs are retrieval bugs (right answer not in context → model improvises); always check retrieval recall before blaming the generator.
• Fallbacks are designed, not improvised: low retrieval confidence → "I don't have information on this" beats a fluent guess; guard failure → retry with repair, then degrade to extractive answer (quote the chunks), then escalate to a human. The fallback ladder is a product decision encoded in the service.
• Everything is versioned: prompt templates, retrieval config, model version, guard thresholds — because Ch. 7's harness must attribute regressions to a change (the Phase 08 lineage discipline, applied to prompts).

Chapter 3: Hallucination, Measured

For RAG systems the operational definition is groundedness (faithfulness): every factual claim in the answer is supported by the retrieved context. This is measurable:

1. Decompose the answer into claims (sentence-level is the practical granularity; clause-level is better and costlier).
2. Score support for each claim against the context: token/n-gram overlap (cheap, the lab's method), NLI entailment models (better), LLM-as-judge with a support rubric (best, with Ch. 7's caveats).
3. Aggregate: groundedness = supported claims / total claims; the complement — the unsupported-claim rate — is the hallucination metric you trend, gate on, and report.

Two companion metrics complete the triangle: answer relevance (does it address the question?) and context relevance (was the retrieval any good? — the diagnostic that splits retrieval bugs from generation bugs). This triangle is the RAGAS-style evaluation core, and the lab implements its mechanics by hand so the industrial tools are legible.

The honest caveats: overlap-based scoring misses paraphrase (false alarms) and miss-attributes coincidental overlap (false passes); sentence decomposition struggles with compound claims. That's why the lab pairs the scorer with golden cases — hand-labeled supported/unsupported examples that pin the scorer's behavior — and why Extension A measures judge-vs-rule agreement before trusting either.

Chapter 4: Prompt Injection — the Attack Surface

The vulnerability class: LLMs cannot reliably distinguish instructions from data in their context. Anyone who controls text the model reads can attempt to program it.

• Direct injection: the user says "ignore previous instructions and …" — embarrassing but bounded by the user's own privileges.
• Indirect injection (the serious one): instructions hidden in retrieved documents, emails, or web pages — the attacker programs your system through your own corpus, with the system's privileges. Any RAG system over user-contributed or external content has this surface.

Defense in depth (no single layer suffices; the lab builds the testable ones):

1. Privilege separation — the architectural defense: the LLM's outputs get least privilege; tool calls are allow-listed and parameter-validated; actions with side effects require confirmation. Assume injection will sometimes succeed; bound what success buys.
2. Input/context filtering: detect known injection patterns in user input and retrieved chunks (the lab's detector); strip or quarantine suspicious content; mark untrusted spans in the prompt ("the following is data, not instructions" — weak alone, useful in depth).
3. Output filtering: detect signs of successful injection (system-prompt leakage, off-policy actions) before the response leaves.
4. Canary regression testing (the lab's signature move): maintain a suite of known attack strings; run them through the full pipeline on every change; the attack success rate is a tracked metric exactly like accuracy. Pattern detectors decay as attacks evolve — the canary suite is how you notice.

Chapter 5: Structured Output and the Repair Loop

Downstream systems need JSON, not prose. The reliability ladder:

1. Constrained decoding (where you control inference): grammar/JSON-schema enforced at the token level — malformed output becomes impossible (the llm-inference track covers the mechanics). Use when available; it converts a validation problem into a non-problem.
2. Validate-then-repair (the portable pattern, the lab's): validate against the schema (Phase 01's contract library does this verbatim!); on failure, re-prompt with the validation errors included ("your output failed: missing field X; emit corrected JSON only") — one repair round fixes the large majority of failures; bound the loop (1–2 rounds), then fallback.
3. Never json.loads + pray: every unvalidated LLM output consumed by code is a production incident on a timer.

Design note the lab encodes: the validator returns aggregated, path-specific errors (Phase 01's collect-don't-raise) precisely because those errors become the repair prompt — error-message quality is now system quality.

Chapter 6: Refusal Policy and Topical Guardrails

Enterprise assistants have a charter: topics they must not advise on (medical, legal, competitor comparisons…), data they must not reveal (PII, system prompts), and tones they must keep. Engineering form:

• Policy as data (the lab): rules with scope (input/output), matcher, action (block / safe-completion / escalate), and rationale — reviewable and versionable like any config, because policy changes need the same governance as code (Phase 12).
• Layered enforcement: system-prompt instruction (steers the model) + input classifier (blocks before spending tokens) + output classifier (catches what steering missed). Measure each layer's catch rate separately — that's how you know which layer regressed.
• The refusal-quality bar: a good refusal states what it can't do, why, and what it can do instead — refusal UX is measurable with the same golden-set machinery (over-refusal is also a failure mode: track false-refusal rate on benign prompts, or the guardrails quietly eat the product).

Chapter 7: Evaluation Harnesses and LLM-as-Judge

The harness is to GenAI what the test suite is to code — and this phase's lab plus Phase 12's CI gate make it literal:

• Golden sets: curated (question, context, reference-or-rubric) cases covering the product's real distribution plus its known failure modes; 50 good cases beat 5 000 scraped ones. Versioned, content-hashed (the Phase 09 model-accuracy discipline).
• Canary suites: the adversarial twin — injections, policy probes, format-breakers. Gate = (quality metrics ≥ baseline) AND (attack success rate ≤ threshold) AND (false-refusal rate ≤ threshold).
• LLM-as-judge: scalable scoring of fluency/relevance/support with a rubric prompt. Its measured biases — position (prefers first answer; mitigate by swapping orders), verbosity (prefers longer), self-preference (prefers its own model family) — and the discipline: validate the judge against human labels on a sample (agreement rate), re-validate when the judge model changes. A judge is a model in production; it gets the same skepticism.
• What runs when: rule-based checks on every request (ms, the lab's layer); golden+canary harness on every change (minutes); human-labeled audits monthly (the calibration anchor).

Chapter 8: Cost, Latency, and the Cascade Pattern

The economics that shape GenAI architecture:

• Token cost asymmetry: context tokens are bought on every request — context assembly discipline (Ch. 2) is a cost lever; caching (Phase 09's tiers, plus provider prompt-caching for stable prefixes) routinely cuts 30–70%.
• The cascade: route easy/common queries to a small model (or cache, or extractive path); escalate hard ones to the big model — a router classifier + confidence threshold. Most enterprise traffic is head-heavy; cascades cut cost multiples at equal quality if the router is evaluated with the same rigor as everything else (its errors are silent quality loss).
• Latency shape: TTFT (time to first token) is UX; streaming hides total latency; guards add tail latency — run input guards in parallel with retrieval, output guards on the stream (sentence-buffered) where possible.
• The Phase 09 frontier thinking applies unchanged; only the units (tokens, not requests) are new.

Lab Walkthrough Guidance

Lab 01 — Guardrail & Eval Harness, suggested order:

1. sentences + token_overlap_support — the groundedness primitives; pin against the hand-built cases first.
2. groundedness_report — claim decomposition, per-claim support, the unsupported-claim rate. The compound-claim test shows the granularity limits; read its comment.
3. InjectionDetector — patterns + the canary suite; note the suite is data (CANARY_ATTACKS), so red-teamers extend it without touching code.
4. PolicyEngine — policy-as-data evaluation, first-match-wins with severity ordering.
5. StructuredOutputGuard — Phase 01's validation shape + the bounded repair loop with the mock LLM.
6. GuardrailHarness.evaluate — compose everything into the gate verdict; the final tests run the full suite both on a healthy mock pipeline and a broken one.

Success Criteria

You are ready for Phase 11 when you can, from memory:

1. Draw the production RAG loop with both guard stages and the fallback ladder.
2. Define groundedness operationally and compute an unsupported-claim rate by hand on a 3-sentence answer.
3. Explain direct vs indirect injection and the four defense layers; say why canaries are the regression mechanism.
4. Recite the structured-output ladder and why repair prompts want path-specific errors.
5. Name the three LLM-as-judge biases and the validation discipline.
6. Sketch the cascade pattern and its router-evaluation requirement.

Interview Q&A

Q: Users report the assistant "makes things up." Walk through your response. Instrument before fixing: build/check the golden set, measure the groundedness triangle — if context relevance is low, it's a retrieval problem (fix Phase 04's stack: hybrid search, reranking, chunking); if context is fine but support is low, it's generation (tighten the prompt's grounding instructions, add citation requirements, lower temperature, add the groundedness output guard with extractive fallback). Then make the metric a CI gate so the fix can't silently regress. The answer's skeleton: measure → localize (retrieval vs generation) → fix the right stage → gate.

Q: Your RAG corpus includes user-uploaded documents. What's your injection story? Indirect injection is the threat model: a poisoned upload programs the assistant for other users. Defense in depth: least-privilege outputs (no tool side effects from document-derived instructions; allow-listed tools with validated params), context filtering on retrieved chunks (detector + quarantine), untrusted-span marking in prompts, output filtering for leakage, and a canary corpus — poisoned test documents whose attack success rate is tracked per release. And the honest statement: filters mitigate, privilege separation is the only defense that holds against novel attacks.

Q: When do you trust LLM-as-judge scores? After validating against human labels on a sample (report agreement, not vibes), with bias mitigations in place (order swapping, length normalization, judge ≠ generator family), pinned judge version, and re-validation on judge change. Use judges for relative comparisons (A/B of prompts) where biases partially cancel, not absolute quality claims. A judge is a model in production — it gets a golden set too.

References

• Lewis et al., Retrieval-Augmented Generation (2020) — arXiv:2005.11401
• Liu et al., Lost in the Middle (2023) — arXiv:2307.03172 — context-position effects
• Es et al., RAGAS: Automated Evaluation of RAG (2023) — arXiv:2309.15217 — the metric triangle
• Greshake et al., Not what you've signed up for: Indirect Prompt Injection (2023) — arXiv:2302.12173
• OWASP Top 10 for LLM Applications — the attack-surface checklist
• Zheng et al., Judging LLM-as-a-Judge (2023) — arXiv:2306.05685 — the bias measurements
• NVIDIA NeMo Guardrails and Guardrails AI — industrial versions of the lab
• llm-inference track Phase 07 — the RAG/agent foundations this phase hardens

Lab 01 — Guardrail & Eval Harness

Phase: 10 — GenAI in Production | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours

Production GenAI is failure management: hallucinations measured, injection treated as an attack surface, outputs validated before anything downstream trusts them. This lab builds that layer — deterministic and model-free (mock LLM), so the system is what's tested.

What you build

• Groundedness scoring — sentence-level claim decomposition + token-overlap support; the unsupported-claim rate as the hallucination metric
• InjectionDetector — pattern detection plus the canary suite: known attack strings whose detection rate is tracked in CI like accuracy
• PolicyEngine — refusal policy as data (scope, matcher, action, rationale), most-severe-match-wins
• StructuredOutputGuard — JSON schema validation with a bounded validate-then-repair loop; the aggregated path-specific errors are the repair prompt (Phase 01's collect-don't-raise, weaponized)
• GuardrailHarness — the release gate: groundedness ≥ threshold AND canary detection ≥ threshold AND zero policy violations

Reference behavior

healthy        groundedness=1.00 unsupported=0.00 attacks_caught=1.00 passed=True
hallucinating  groundedness=0.75 unsupported=0.25 attacks_caught=1.00 passed=False

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (17 tests)
python solution.py                          # the demo above

Suggested TODO order

1. sentences / content_tokens / token_overlap_support — pin against the hand-built supported/fabricated cases
2. groundedness_report — the mixed-answer test computes 2/3 by hand
3. InjectionDetector.scan + canary_detection_rate — then read the false-positive test: clean questions mentioning "previous quarter" must NOT flag
4. PolicyEngine.evaluate — scope separation and severity ordering are the two traps
5. StructuredOutputGuard — the repair test asserts your repair prompt contains the specific errors; that's the design point
6. GuardrailHarness.evaluate — compose; three gate tests (healthy / hallucinating / policy-violating-but-grounded)

Success criteria

• All 17 tests pass
• You can compute an unsupported-claim rate by hand and state the overlap scorer's two failure modes (paraphrase → false alarm; coincidental overlap → false pass)
• You can explain why the canary suite is data, not code, and what its detection-rate trend tells you
• You can argue why the policy-violating-but-perfectly-grounded case must still fail the gate

Extensions

• Swap MockLLM for a real provider; add an NLI-based or LLM-as-judge support scorer; measure agreement with the rule-based scorer on 50 hand-labeled cases
• Indirect-injection red team: poison a document corpus, measure attack success through full RAG with each defense layer on/off
• Over-refusal tracking: a benign-prompt suite whose false-refusal rate is gated too
• Streaming output guards: sentence-buffered groundedness on a token stream

Interview Q&A

Q: How do you measure hallucination in a RAG system? Groundedness: decompose the answer into claims (sentence level), score each for support against the retrieved context (overlap → NLI → LLM-as-judge, in increasing cost and quality), report unsupported-claim rate over a versioned golden set. Trend it per release and gate on it. And split the diagnosis: low context relevance means retrieval is the problem, not generation — most "hallucination" bugs are retrieval bugs.

Q: Your injection detector has 100% canary detection. Are you safe? No — the canary suite only proves you catch known attacks; it's a regression floor, not a security ceiling. Novel attacks bypass pattern detectors by construction. The defenses that hold against unknowns are architectural: least privilege on outputs, allow-listed tools with validated parameters, human confirmation on side effects. Detection is one layer of depth, never the story.

References

• Es et al., RAGAS (2023) — the groundedness/relevance metric triangle
• Greshake et al., Indirect Prompt Injection (2023) — the threat model
• OWASP Top 10 for LLM Applications
• Guardrails AI / NeMo Guardrails — industrial versions

Phase 11 — Online Experimentation

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "online experimentation" and "robust evaluation … strategies, including … online experimentation"

Why This Phase Exists

Offline metrics propose; online experiments dispose. A ranking model with +3% offline NDCG can lose revenue online (position effects, feedback loops, latency); the only ground truth for "did the model help" is a controlled experiment — and experiments are easy to run and very easy to run wrong. The failure modes are statistical (underpowered tests, peeking, multiple comparisons) and infrastructural (broken randomization, SRM), and they produce confident wrong answers.

The flagship lab builds the analyst's toolkit from scratch — power analysis, the two-proportion test, sample-ratio-mismatch detection, CUPED variance reduction, and a Thompson-sampling bandit — each pinned by hand-computed tests, so the statistics stop being incantations.

Concepts

• Randomized assignment: units (user vs session vs request), hash-based bucketing, stickiness; interference/network effects
• Hypothesis testing for experiments: two-proportion z-test, t-test for means, p-values honestly interpreted, confidence intervals over point estimates
• Power analysis: MDE, α/β, the sample-size formula — and what to do when traffic can't reach power
• SRM (sample ratio mismatch): the chi-square canary that catches broken randomization before it poisons conclusions
• Peeking and sequential testing: why repeated significance checks inflate false positives; group-sequential/always-valid approaches
• CUPED: variance reduction with pre-experiment covariates — the cheapest power multiplier in the industry
• Multiple comparisons across metrics/segments; guardrail vs goal metrics
• Bandits vs A/B: Thompson sampling, when regret minimization beats inference
• Interleaving for ranking (cross-ref Phase 05)

Labs

Lab 01 — Experiment Analyzer & Bandit (flagship, implemented)

→ Lab folder: lab-01-ab-toolkit/

Extension Project A — Sequential Testing (spec)

Implement a group-sequential boundary (O'Brien-Fleming-style alpha spending) and an always-valid e-value test; simulate peeking with both vs naive repeated z-tests; produce the false-positive-rate comparison table.

Extension Project B — Interleaving Simulator (spec)

Team-draft interleaving for two rankers over simulated click models (position-biased); measure sensitivity vs A/B on the same traffic: how much less traffic does interleaving need to detect the same ranker difference?

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide

Deliverables Checklist

• Lab 01 implemented; all 15 tests pass
• Extension A peeking comparison table
• Extension B interleaving sensitivity result

Warmup Guide — Online Experimentation

Zero-to-expert primer for Phase 11: how to learn the truth from live traffic — randomization, power, the tests, the traps (SRM, peeking, multiple comparisons), CUPED, and when bandits replace A/B tests.

Table of Contents

• Chapter 1: Why Online Experiments Are Ground Truth
• Chapter 2: Randomization — the Part That Does the Work
• Chapter 3: The Statistics You Actually Use
• Chapter 4: Power and the MDE Conversation
• Chapter 5: SRM — the Canary in Every Experiment
• Chapter 6: Peeking and Sequential Testing
• Chapter 7: CUPED — Variance Reduction for Free
• Chapter 8: Multiple Metrics, Guardrails, and Decisions
• Chapter 9: Bandits — When to Stop Splitting Traffic Evenly
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Online Experiments Are Ground Truth

Offline evaluation (Phases 04–07) answers "does the model predict held-out data better?" The business question is "does shipping it cause better outcomes?" — and cause is the hard word. Correlations in logged data are poisoned by confounders (the old model chose what users saw — Phase 06's feedback loop); pre/post comparisons are poisoned by time (seasonality, marketing, news). The randomized controlled experiment is the one design where the difference between groups estimates the causal effect, because randomization makes the groups exchangeable in expectation — every confounder, known and unknown, is balanced.

The senior-engineer framing the JD implies: experimentation is infrastructure (assignment, logging, analysis pipelines, dashboards — this phase + Phase 08's patterns) plus statistical hygiene (this WARMUP) plus decision discipline (Ch. 8). Most experiment failures are not math failures; they're hygiene failures.

Chapter 2: Randomization — the Part That Does the Work

• The unit: user (default — consistent experience, captures learning effects), session (more power, risks within-user contamination), request (max power, only for invisible backend changes). Choose the unit that matches where the effect lives and where interference is tolerable.
• Hash-based assignment (Phase 09's canary router, generalized): bucket = hash(salt:unit_id) / MAX — deterministic (re-computable anywhere), sticky (same user, same arm, forever), and salt-independent across experiments (a fresh salt per experiment decorrelates assignments; reusing salts couples experiments — a real and nasty bug class).
• Interference: when one user's treatment affects another's outcome (social features, marketplaces with shared inventory, shared caches!), the independence assumption breaks. Mitigations: cluster randomization (by region/ market), switchback designs (time-sliced for marketplaces). Know the names; recognize the situations.
• The engineering invariant: assignment is logged at exposure time with the experiment version. Analyzing "everyone in the bucket" instead of "everyone exposed" dilutes effects with users who never saw the change.

Chapter 3: The Statistics You Actually Use

For conversion-style (binary) metrics — the two-proportion z-test (the lab's core):

$$\hat{p} = \frac{x_A + x_B}{n_A + n_B}, \qquad z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}(1-\hat{p})(\tfrac{1}{n_A} + \tfrac{1}{n_B})}}$$

with the two-sided p-value $2(1 - \Phi(|z|))$ and the CI on the difference using unpooled standard errors. For means (revenue, latency): Welch's t-test — and for heavy-tailed metrics like revenue, expect to need winsorization or much more traffic (a handful of whales dominates the variance).

Honest interpretation (the part interviews probe):

• The p-value is P(data this extreme | no true effect) — not P(no effect | data).
• A non-significant result is not "no effect" — it's "couldn't distinguish from zero at this power"; report the CI, which says what effect sizes remain plausible.
• Always report the CI alongside the decision — "+0.4% [+0.1%, +0.7%]" carries the information; "significant ✓" does not.
• Calibration check the lab performs: under A/A simulation (no true effect), your test must reject ≈ α of the time. A test that can't pass its A/A check is miscalibrated machinery, and so is an experimentation platform that never runs A/A tests.

Chapter 4: Power and the MDE Conversation

The pre-experiment math that prevents the most common waste (underpowered tests):

$$n \text{ per arm} \approx \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2,\bar{p}(1-\bar{p})}{\delta^2}$$

for detecting an absolute difference δ at significance α and power 1−β (z's: 1.96 and 0.84 for the standard 0.05/0.80). The numbers that build intuition: baseline 5% conversion, MDE 0.25% absolute (5% relative) → ~120k users per arm. Small effects on small baselines are expensive.

The MDE conversation (the senior skill): before launch, compute the minimum detectable effect at the traffic you'll actually get. If the MDE exceeds any plausible effect size, the experiment cannot answer the question — the honest options are: a more sensitive metric (CUPED, Ch. 7; or a proxy metric closer to the mechanism), longer duration, larger traffic share, interleaving (for rankers), or don't run it and decide on other evidence. Running it anyway and reading the noise is the failure mode.

Chapter 5: SRM — the Canary in Every Experiment

Sample Ratio Mismatch: you configured 50/50; you observe 50.6/49.4 over a million users. Is that chance? The chi-square test on counts answers:

$$\chi^2 = \sum_{arms} \frac{(O_i - E_i)^2}{E_i} \sim \chi^2_{k-1}$$

If p < 0.001 (the conventional SRM threshold — deliberately strict), your assignment or logging is broken, and every other number in the experiment is untrustworthy — because whatever deleted users from one arm (a crash on the new code path, a redirect dropping trackers, a bot filter interacting with the treatment) almost certainly didn't delete them at random. Industry experience (Microsoft, booking.com report ~6–10% of experiments): SRM is common, automatic SRM checks are non-negotiable platform hygiene, and the response to SRM is stop and debug the plumbing, never "adjust for it."

Chapter 6: Peeking and Sequential Testing

The trap: check the p-value daily, stop when significant. Each look is another chance for a false positive; with daily peeks over a month, the realized false-positive rate can exceed 25% against a nominal α = 5%. The fixed-horizon test is only valid at its fixed horizon.

The legitimate options:

1. Don't peek (for decisions): fix duration in advance — full weeks (weekly seasonality!), minimum two — and look once. Dashboards can display data; the decision rule fires once.
2. Group-sequential: pre-plan k looks with adjusted boundaries (O'Brien–Fleming: very strict early, near-nominal late) — early stopping for clear winners/harms with controlled overall α (Extension A).
3. Always-valid inference (mSPRT/e-values): p-values valid under continuous monitoring — costs some power, removes the discipline burden; increasingly the platform default.

The deeper point: peeking is one instance of garden-of-forking-paths problems (unplanned subgroups, metric switching, post-hoc exclusions). The cure is the same: pre-registration of the decision rule — written before launch (Ch. 8).

Chapter 7: CUPED — Variance Reduction for Free

The cheapest power multiplier in the industry. Idea: much of the variance in your metric is pre-existing user heterogeneity (heavy users convert more), which pre-experiment data already predicts — remove it.

With $X$ = the user's pre-experiment value of the (ideally same) metric:

$$Y_{cuped} = Y - \theta (X - \bar{X}), \qquad \theta = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}$$

(θ is the OLS slope — the control-variates trick from simulation.) Properties that make it safe and effective: the adjustment is mean-zero, so the treatment-effect estimate is unbiased regardless of X's quality; variance shrinks by factor $(1 - \rho^2)$ where ρ = corr(X, Y) — at ρ = 0.6, variance drops 36%, equivalent to 36% more traffic for free; X must be pre-experiment (anything measured after assignment can be affected by treatment and biases the estimate — the one rule). New users have no X → use $\theta \cdot 0$ adjustment (their unadjusted Y), or stratify. The lab implements CUPED and measures the variance reduction against the $(1-\rho^2)$ theory.

Chapter 8: Multiple Metrics, Guardrails, and Decisions

• Metric roles, declared up front: one goal metric (the decision metric — powering and α apply to it), a small set of guardrails (latency, error rates, unsubscribes, revenue-per-user — tested for harm, often one-sided), and diagnostics (unlimited, decision-irrelevant, exploration-only).
• Multiple comparisons: 20 metrics × 5 segments at α=0.05 ≈ 5 false alarms per experiment. Corrections (Bonferroni, Benjamini–Hochberg FDR) for anything decision-relevant; the cleaner fix is the role discipline above — one goal metric needs no correction.
• Segment "wins" ("it works for Android users in Egypt!") found post-hoc are hypotheses for the next experiment, not findings — forking paths again.
• The decision document (pre-registered): goal metric + MDE + duration + guardrail thresholds + the action for each outcome (ship / iterate / abandon), agreed before launch. Experiments end in arguments exactly when this doesn't exist (the same pre-agreement discipline as Phase 09's canary thresholds).

Chapter 9: Bandits — When to Stop Splitting Traffic Evenly

A/B tests spend traffic on inference; bandits optimize traffic allocation while learning. Thompson sampling (the lab's algorithm) for Bernoulli rewards: keep a Beta(α=1+successes, β=1+failures) posterior per arm; each round, sample a value from each posterior and play the argmax. Exploration emerges from posterior uncertainty — wide posteriors win sometimes; as evidence accumulates, allocation concentrates on the best arm. It dominates epsilon-greedy (which explores at a fixed rate forever, paying linear regret on the exploration fraction — the lab measures this ordering).

Bandit vs A/B — the decision rule: bandits when the cost of serving the worse arm matters and the question is "which is best" (headlines, promos, creatives — short feedback, many variants); A/B when you need an unbiased effect estimate (ship decisions with guardrails, anything with slow/delayed outcomes, anything requiring segment analysis — bandit allocation confounds time with arm). Hybrid patterns exist (bandit for allocation + holdout for measurement). Delayed feedback and nonstationarity are the practical bandit killers — say so when proposing one.

Lab Walkthrough Guidance

Lab 01 — Experiment Analyzer & Bandit, suggested order:

1. required_sample_size — pin against the worked example (120k figure).
2. two_proportion_ztest — hand-computed case first, then the A/A calibration test (simulated no-effect experiments must reject ≈ α — if this fails, your test is broken, full stop).
3. srm_check — chi-square with the strict threshold; the sensitivity test shows a 1% imbalance at scale screaming while a 1% imbalance at n=200 stays quiet — that's the statistics working.
4. cuped_adjust — implement θ from the formula; the test checks measured variance reduction ≈ $(1-\rho^2)$ and that the effect estimate stays unbiased.
5. The bandits — EpsilonGreedy then ThompsonSampling (seeded RNG injected — Phase 01's testability discipline); the regret test asserts the ordering TS < ε-greedy < uniform on a fixed scenario.

Success Criteria

You are ready for Phase 12 when you can, from memory:

1. Explain what randomization buys (exchangeability → causal estimate) and choose a unit for three scenarios.
2. Compute a required sample size and run the MDE conversation for an underpowered proposal.
3. State precisely what a p-value is and is not; explain why CIs must accompany decisions.
4. Define SRM, its test, its threshold, and the only correct response to it.
5. Explain why peeking inflates false positives and name the three legitimate alternatives.
6. Derive CUPED's θ, state its two safety properties, and the pre-experiment rule.
7. Describe Thompson sampling in four sentences and give the bandit-vs-A/B decision rule.

Interview Q&A

Q: Your new ranker won the A/B (+2% CTR, significant) but revenue is flat. Ship? First check the pre-registered decision rule — if CTR was the goal metric and revenue a guardrail that didn't degrade, the rule says ship. But interrogate the gap: CTR up with flat revenue suggests clickbait-shaped ranking (clicks without purchases — check downstream funnel metrics), position effects, or revenue being underpowered relative to CTR (almost always true — compute revenue's MDE before calling it "flat"; the CI will likely include meaningful gains and losses). The senior answer is the decomposition, not a yes/no.

Q: An experiment shows SRM (p = 1e-7). The PM wants to "just reweight the arms." Respond. No — SRM means units were selectively lost, and selection is correlated with treatment by construction (the treatment caused the loss: a crash, a timeout, a tracker change). Reweighting fixes the counts, not the selection bias; every metric remains untrustworthy. The fix is forensic: segment the mismatch by browser/platform/date to localize what's dropping users, repair, relaunch. Quote the industry stat (≈6–10% of experiments hit SRM) to normalize stopping.

Q: When would you use CUPED and what's the one way to get it wrong? Any user-level metric with decent pre-period correlation (returning-user products: ρ 0.4–0.7 typical → 16–50% variance reduction → weeks of traffic saved). The fatal error: using a covariate measured after assignment — it can carry treatment effect, biasing the estimate. Everything else is safe: θ misestimated just reduces less variance; new users degrade gracefully to unadjusted.

References

• Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020) — the book; ch. 3 (SRM), 17 (statistics), 22 (CUPED context)
• Deng, Xu, Kohavi & Walker, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (WSDM 2013) — the CUPED paper
• Fabijan et al., Diagnosing Sample Ratio Mismatch (KDD 2019) — the SRM taxonomy
• Johari et al., Always Valid Inference: Continuous Monitoring of A/B Tests (2017) — peeking, solved properly
• Russo et al., A Tutorial on Thompson Sampling (2018) — arXiv:1707.02038
• Chapelle & Li, An Empirical Evaluation of Thompson Sampling (NeurIPS 2011)
• Spotify, Netflix, and booking.com engineering blogs on experimentation — platform war stories

Lab 01 — Experiment Analyzer & Bandit

Phase: 11 — Online Experimentation | Difficulty: ⭐⭐⭐⭐⭐ | Time: 4–6 hours

Every number an experimentation platform shows you comes from five computations. Build all five from scratch — power, the z-test, SRM, CUPED, Thompson sampling — each pinned by hand-computed cases and simulation-based calibration checks.

What you build

• required_sample_size — the power formula; the ~122k-per-arm anchor case
• two_proportion_ztest — pooled-SE z, two-sided p, unpooled-SE CI — validated by an A/A calibration test (under no effect, rejects ≈ α — the test that tests your test)
• srm_check — chi-square sample-ratio-mismatch detection at the strict 1e-3 threshold, including unequal design ratios
• cuped_adjust — θ = Cov/Var, mean-zero adjustment; the test verifies measured variance reduction ≈ ρ² and that the mean (hence the effect estimate) is preserved
• EpsilonGreedy / ThompsonSampling / run_bandit — regret ordering measured over seeds: TS < ε-greedy < uniform

Reference demo

n per arm to detect 5% → 5.25% at alpha=.05, power=.80: 122,121
A/B: effect=+0.0030 [+0.0011, +0.0049] p=0.0024 significant=True
SRM check 50600/49400: p=1.48e-04 detected=True
cumulative regret over 20k pulls: eps-greedy=35  thompson=31

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (15 tests)
python solution.py

Suggested TODO order

1. required_sample_size — anchor on 122k; check the inverse-square MDE scaling test
2. two_proportion_ztest — hand-computed case, then run the A/A test repeatedly: it is the single most diagnostic test in the suite
3. srm_check — note the three cases: scale makes 1.2% scream, n=200 stays quiet, design ratios respected
4. cuped_adjust — the mean-preservation test is the unbiasedness property
5. Bandits — seeded RNGs are injected everywhere (Phase 01's discipline); regret tests average over 8 seeds

Success criteria

• All 15 tests pass
• You can explain why the z-test uses pooled SE but the CI uses unpooled
• You can state what an SRM detection means operationally (stop; debug plumbing; never reweight)
• You can derive CUPED's variance-reduction factor (1−ρ²) and its one fatal misuse
• You can explain why Thompson sampling's exploration anneals and ε-greedy's doesn't

Extensions

• Welch's t-test for continuous metrics + winsorization for revenue-style tails
• Group-sequential boundaries (O'Brien–Fleming) — simulate peeking, compare realized α
• Always-valid e-value test; same peeking simulation
• Bonferroni / Benjamini–Hochberg over a 20-metric scorecard; false-discovery demo
• Contextual bandit (LinUCB) on synthetic user features

Interview Q&A

Q: The PM checked the dashboard daily and stopped the test on day 4 when p hit 0.03. What do you tell them? That p-value doesn't mean what it claims: each peek is another draw at α, and stopping-at-first-significance inflates the false-positive rate severalfold (simulate it — Extension A's table). Options going forward: pre-committed fixed horizon (full weeks), group-sequential boundaries if early stopping matters, or an always-valid method if the dashboard culture won't change. And re-run this test to the planned horizon before shipping.

Q: Why does an experimentation platform need A/A tests? They validate the machinery end-to-end: assignment uniformity (SRM at scale), metric-pipeline correctness, and statistical calibration (significance fires ≈ α with no effect). A platform that has never passed A/A tests produces numbers with unknown error rates — you're not measuring the product, you're measuring your bugs.

References

• Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)
• Deng et al., CUPED (WSDM 2013)
• Fabijan et al., Diagnosing Sample Ratio Mismatch (KDD 2019)
• Russo et al., A Tutorial on Thompson Sampling — arXiv:1707.02038

Phase 12 — Observability, Drift & Governance

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "observability … drift detection, retraining strategy, and responsible AI controls" and "model risk assessment"

Why This Phase Exists

Deployment is the midpoint of a model's life, not the end. The world drifts away from the training distribution, labels arrive late or never, silent feature breakage masquerades as gradual degradation, and someone — increasingly, a regulator — asks "who approved this model and what are its known limitations?" This phase builds the machinery that keeps shipped models honest: drift detection with statistical teeth, a retraining policy that's a state machine rather than a vibe, and the governance artifacts (model cards, risk tiers) that make ML auditable.

The flagship lab implements all three layers; the WARMUP supplies the monitoring taxonomy, the drift mathematics, and the responsible-AI frameworks the lab encodes.

Concepts

• The monitoring stack: system health → data quality → drift → model quality → business KPIs; why each layer catches what the previous can't
• Covariate / prior / concept drift — definitions, causes, and which monitors see which
• PSI (population stability index) and KS test — the workhorse detectors; binning discipline; thresholds and their abuse
• Proxy metrics under label delay: prediction-distribution monitoring, autoencoder/ density scores, delayed-label backfill joins
• Retraining strategy as policy: triggers (scheduled / drift / performance), validation gates, and the retraining-on-drifted-data trap
• Model risk: tiering by impact, model cards, fairness slices, human override paths
• Responsible AI in practice: documentation, monitoring of protected-group performance, the EU-AI-Act-shaped compliance reality

Labs

Lab 01 — Drift Monitor & Retraining Policy (flagship, implemented)

→ Lab folder: lab-01-drift-monitor/

Extension Project A — Label-Delay Simulator (spec)

Simulate a fraud stream where labels arrive 30 days late; compare detection lag of (a) prediction-distribution drift, (b) PSI on features, (c) delayed ground-truth metrics; produce the "what monitoring buys you" timeline chart.

Extension Project B — Fairness Slice Monitor (spec)

Add protected-attribute slicing to the Phase 03 model's evaluation: per-slice precision/recall, demographic parity and equalized-odds gaps, with the small-slice-variance caveats handled (CIs per slice); wire as a Phase 08 CI gate.

Guides in This Phase

• WARMUP.md — the full zero-to-expert guide

Deliverables Checklist

• Lab 01 implemented; all 15 tests pass
• Extension A timeline chart
• Extension B fairness gate

Warmup Guide — Observability, Drift & Governance

Zero-to-expert primer for Phase 12: keeping shipped models honest — the monitoring stack, drift mathematics (PSI/KS), retraining as policy, and the governance artifacts that make ML auditable.

Table of Contents

• Chapter 1: Why Models Rot
• Chapter 2: The Five-Layer Monitoring Stack
• Chapter 3: The Drift Taxonomy
• Chapter 4: PSI and KS — the Workhorse Detectors
• Chapter 5: Monitoring Without Labels
• Chapter 6: Retraining as Policy
• Chapter 7: Model Risk and Governance Artifacts
• Chapter 8: Responsible AI, Operationally
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Models Rot

A deployed model is a frozen snapshot of a moving world. It degrades through three distinct mechanisms, each needing different detection:

1. The world changes (drift proper): user behavior shifts, new products launch, a pandemic rewrites every prior. Gradual or sudden; inevitable.
2. The plumbing breaks (silent feature failure): an upstream schema change, a default-to-null deploy, a stale cache — the model sees garbage and produces confident, plausible garbage. This is the most common "drift" in practice and the most preventable (Phase 01's contracts + this phase's distribution monitors).
3. The model changes the world (feedback): the recommender trains on clicks it caused (Phase 06's loop); the fraud model changes fraudster behavior (adversarial drift). Monitoring must account for the model's own fingerprints in the data.

The phase's stance: rot is not an if but a when-and-how-fast; the engineering question is detection lag — how long between the world changing and you knowing.

Chapter 2: The Five-Layer Monitoring Stack

Each layer catches what the previous can't; mature teams run all five:

Two structural facts: layer 4 is gated on labels (often delayed days or forever — Ch. 5 exists because of this), and alerts must carry attribution — "PSI breach on merchant_category (0.31), top contributing bins: …" is actionable; "model drift alert" is noise that trains on-call to ignore the channel (alert fatigue is the death of monitoring programs; tune for precision first).

Chapter 3: The Drift Taxonomy

With features $X$ and target $y$, the joint $P(X, y)$ can shift three ways:

• Covariate drift: $P(X)$ changes, $P(y|X)$ stable — new traffic mix, new demographics. The model may still be correct where it's now being asked to predict — but it's extrapolating into thinner training support. Detectable without labels (layer 3 sees it).
• Prior/label drift: $P(y)$ changes — fraud base rate doubles. Calibration breaks even if ranking survives; thresholds need re-tuning. Visible in the prediction distribution before labels confirm.
• Concept drift: $P(y|X)$ changes — the same features now mean something different (a behavior that signaled churn becomes normal). The damaging one, and invisible to feature-distribution monitors — only label-based layer 4 (or its proxies, Ch. 5) sees it.

The diagnostic discipline: feature drift detected ≠ model damaged (impact depends on the feature's importance and the direction of the shift); model damaged ≠ feature drift visible (concept drift hides). Drift monitors are smoke detectors, not damage meters — they tell you where to look, which is exactly how the lab's report ranks features.

Chapter 4: PSI and KS — the Workhorse Detectors

PSI (Population Stability Index) — the credit-risk industry's standard, because it's interpretable and decomposable:

1. Bin the reference distribution into quantile bins (deciles standard) — reference quantiles, so expected mass is uniform 10% per bin; current data is then counted into those fixed bins. (Binning on current data instead is the classic implementation bug — it hides exactly the shifts you seek.)
2. With reference share $r_i$ and current share $c_i$ per bin:

$$\text{PSI} = \sum_i (c_i - r_i) \ln!\frac{c_i}{r_i}$$

3. Conventional thresholds: < 0.1 stable, 0.1–0.25 moderate, > 0.25 action. These are folklore, not theory — PSI grows with sample size sensitivity and bin count; calibrate thresholds on your metric's historical week-over-week PSI (the lab's no-shift test makes this concrete). Per-bin terms give attribution: which part of the distribution moved.

KS (Kolmogorov–Smirnov): max distance between empirical CDFs, $D = \max_x |F_{ref}(x) - F_{cur}(x)|$, with a p-value. Binning-free and principled — but at production sample sizes (millions), it flags trivially small shifts as significant (the statistical-vs-practical significance gap). Practical pattern: KS/PSI as the detector, an effect-size threshold (PSI level, or KS D itself) as the alerter. For categorical features: chi-square or PSI over category shares (with smoothing for empty cells — the lab handles this).

Prediction-distribution drift (PSI/KS on the model's scores) deserves special status: it summarizes all features through the model's own lens, catches prior drift early, and needs no labels — the single highest-value drift monitor per line of code.

Chapter 5: Monitoring Without Labels

Ground truth arrives late (fraud: 30–90 days), sampled (manual review), or never (what would the rejected loan have done?). The proxy toolbox, in order of deployment frequency:

1. Prediction-distribution monitoring (Ch. 4) — always available, immediate.
2. Feature drift with importance weighting: weight per-feature drift by the feature's model importance — drift in the top feature outranks drift in a minor one (the lab's report does this).
3. Confidence/entropy trends: rising average uncertainty often precedes measurable degradation.
4. Delayed-label backfill: when labels do arrive, join them back to the predictions of that time (event-time discipline — Phase 02!) and compute trailing metrics with their inherent lag; this is layer 4's actual implementation, and it also feeds Ch. 6's performance trigger.
5. Canary cohorts with fast labels: a small segment where ground truth is cheap (e.g., manually reviewed sample) as a leading indicator.

The honest sentence to internalize: between label arrivals, you are flying on proxies; the engineering goal is to choose proxies whose failure correlates with the model's.

Chapter 6: Retraining as Policy

"When do we retrain?" answered by vibe is answered wrong. The policy is a state machine (the lab implements exactly this):

HEALTHY → (drift breach | schedule due | perf breach) → RETRAINING
RETRAINING → (new model trained) → VALIDATING
VALIDATING → (gates pass) → PROMOTING → HEALTHY
VALIDATING → (gates fail) → INCIDENT  (human in the loop — do NOT auto-loop)

The design decisions that matter:

• Triggers are tiered: scheduled retraining (the baseline cadence — data freshness has value even without detected drift), drift-triggered (early, cheap), performance-triggered (late, definitive). Most teams run scheduled + drift-as-accelerator.
• The validation gate compares challenger vs incumbent on current data (the freshest labeled window — not the original test set, which the world has left behind). Phase 08's quality-gate machinery, pointed at a moving target.
• The trap the FSM must encode: drift fired because data is broken (Ch. 1 mechanism 2) → retraining on broken data launders the breakage into the model. Hence the data-quality gate precedes the retrain trigger in the policy: drift + contract violations = data incident, not retraining.
• Failed validation goes to INCIDENT, not auto-retry: a challenger that can't beat the incumbent on current data means something structural changed — a human decides (more data? new features? concept shift needing redesign?). Auto-looping retrains burns compute to hide a signal.

Chapter 7: Model Risk and Governance Artifacts

The JD's "model risk assessment" — the practice (from banking's SR 11-7 heritage, now spreading everywhere via AI regulation):

• Risk tiering: classify each model by decision impact (advisory → automated with human override → fully automated), affected population, and reversibility. The tier sets the process weight: a Tier-1 credit model gets independent validation and quarterly review; a Tier-3 internal search ranker gets CI gates. Proportionality is the point — governance that treats everything as Tier-1 gets routed around.
• The model card (the lab generates one): intended use and out-of-scope uses, training-data provenance (Phase 08's lineage), evaluation results including slice performance, known limitations and failure modes, monitoring plan + retraining policy, owner and approval chain. The test of a good card: an engineer who has never seen the model can decide from it whether a proposed new use is safe.
• Auditability: every production prediction traceable to (model version → training run → data snapshot → approval record) — Phase 08's ledger + registry is this; governance is where it pays off.

Chapter 8: Responsible AI, Operationally

Stripped of slogans, the engineering content:

• Fairness is measured on slices: per-protected-group performance (TPR/FPR/ precision gaps — equalized odds family; selection-rate ratios — demographic parity family). The metrics conflict mathematically (you can't satisfy all simultaneously except in degenerate cases) — choosing which matters is a product/policy decision you document, then monitor like any other metric (Extension B wires it as a CI gate). Small slices need CIs (Phase 11's statistics) or they generate false alarms.
• Human override paths for consequential decisions: the override rate is itself a monitored metric — rising overrides = the model is losing operator trust, often before metrics confirm why.
• Documentation as control: model cards + decision logs + known-limitation registers are what "responsible AI" audits actually inspect (EU AI Act high-risk requirements are, to first order: risk tiering + documentation + monitoring + human oversight — i.e., this chapter).
• The line engineers hold: no governance artifact substitutes for the Phase 11 question "did we measure the harm?" — guardrail metrics in experiments (complaint rates, slice deltas) are responsible AI practiced, not just documented.

Lab Walkthrough Guidance

Lab 01 — Drift Monitor & Retraining Policy, suggested order:

1. psi — reference-quantile binning first (the test plants a shift detectable only with correct binning), smoothing for empty bins, then the hand-computed case.
2. ks_statistic — the two-sample D; check against scipy in the test.
3. DriftMonitor.report — per-feature PSI + KS, importance-weighted ranking, the no-shift calibration case (your false-alarm rate on stable data is the threshold-calibration lesson).
4. RetrainingPolicy — the FSM with guards; the two tests to respect: the data-quality gate preempts the drift trigger, and validation failure lands in INCIDENT (no auto-loop).
5. model_card — assemble from the monitor + policy + provided metadata; the completeness test enumerates the required sections (Ch. 7's list).

Success Criteria

You are ready for Phase 13 when you can, from memory:

1. Name the three rot mechanisms and which monitoring layer catches each.
2. Define the three drift types and state which are label-free detectable.
3. Compute PSI by hand on a 3-bin example; state the binning rule and why; recite the threshold folklore and its caveat.
4. Explain the statistical-vs-practical significance gap at production sample sizes and the detector/alerter split.
5. Draw the retraining FSM with both trap-guards (broken-data laundering; no-auto-loop).
6. List the model card's sections and the risk-tiering axes.

Interview Q&A

Q: Your fraud model's AUC dropped from 0.92 to 0.87 over two months. Walk through the investigation. Timeline first: sudden drop vs slow slide (sudden = plumbing or upstream change — check data-quality layer and deploy/schema timelines; slow = world drift). Then attribution: per-feature PSI ranked by importance — a top feature with PSI 0.4 and a null-rate change is a broken join, not drift; fix the data, don't retrain. If features are healthy: prediction-distribution + prior check (fraud base rate moved? → threshold/calibration fix may suffice) before concept drift (adversarial adaptation — needs new features/labels, not just fresh training). Retraining is the last answer, after the data is known-good — retraining on broken data launders the breakage in.

Q: How do you monitor a model whose labels arrive 60 days late? Layered proxies: prediction-distribution drift (immediate), importance-weighted feature PSI (immediate), confidence trends, a fast-label canary cohort (sampled manual review), and the 60-day backfill join computing true metrics with honest lag (event-time join — predictions matched to their moment). Set the retraining schedule from drift velocity measured on the proxies; let the delayed true metric audit the proxy choices quarterly. Name the residual risk: concept drift that leaves feature distributions intact will take up to 60 days + detection to surface — that bound should be in the model card.

Q: The compliance team asks "how do you know this model is safe to keep running?" Give the systems answer. Point at artifacts, not assurances: the five-layer monitoring stack with current dashboards (data contracts green, PSI under calibrated thresholds, trailing metrics within control bands, slice gaps monitored with CIs), the retraining policy FSM with its gates and history, the model card with known limitations and out-of-scope uses, and the lineage chain from every prediction to an approved training run. Safety isn't a property we assert; it's a process whose evidence is queryable — and here's the query.

References

• Breck et al., The ML Test Score (2017) — monitoring sections
• Gama et al., A Survey on Concept Drift Adaptation (ACM CS 2014) — the taxonomy, rigorously
• Rabanser et al., Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019)
• Evidently AI docs — the industrial version of the lab; read after building
• Mitchell et al., Model Cards for Model Reporting (FAT* 2019) — arXiv:1810.03993
• Federal Reserve SR 11-7, Guidance on Model Risk Management — the governance origin document
• Hardt et al., Equality of Opportunity in Supervised Learning (NeurIPS 2016) — the fairness-metric mathematics
• EU AI Act high-risk requirements overview — the regulatory shape

Lab 01 — Drift Monitor & Retraining Policy

Phase: 12 — Observability, Drift & Governance | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Deployment is the midpoint of a model's life. This lab builds what keeps shipped models honest: PSI/KS detection with attribution, a retraining policy that's a state machine instead of a vibe, and a model card generated from the live system.

What you build

• psi — reference-quantile binning (the implementation detail that decides whether you can see shifts at all — a test plants the case that current-quantile binning hides), empty-bin smoothing, the standard thresholds
• ks_statistic — two-sample D, validated against scipy to 1e-9
• DriftMonitor.report — per-feature PSI+KS, importance-weighted ranking (drift in the top feature outranks drift in a minor one), prediction-distribution PSI as a first-class trigger
• RetrainingPolicy — the FSM HEALTHY → RETRAINING → VALIDATING → PROMOTING, with two guards the tests enforce: broken data preempts retraining (laundering guard) and failed validation lands in INCIDENT (no auto-retry loop); append-only history as the audit trail
• model_card — every governance section, generated from live monitor + policy state, not hand-maintained prose

Reference demo

feature drift (ranked):
  amount     PSI=0.499 KS=0.287 [action] weighted=0.299
  age_days   PSI=0.001 KS=0.007 [stable] weighted=0.000
prediction PSI: 0.180  any_breach=True

policy history:
  healthy -> retraining: drift breach on amount
  retraining -> validating: challenger trained
  validating -> promoting: challenger 0.8730 >= incumbent 0.8610
  promoting -> healthy: registry pointer moved

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (17 tests)
python solution.py

Suggested TODO order

1. psi — the reference-quantile test is the one to satisfy first
2. ks_statistic — scipy agreement pins it
3. FeatureDrift.level / DriftReport properties / DriftMonitor.report
4. RetrainingPolicy — happy path, then the two guard tests
5. model_card — the section checklist test

Success criteria

• All 17 tests pass
• You can explain why PSI bins on reference quantiles and what breaks otherwise
• You can defend both FSM guards in one sentence each
• You can state what drift detection does NOT tell you (impact), and how importance-weighting partially compensates

Extensions

• Categorical PSI (chi-square / share-based with smoothing) for string features
• Window management: rolling reference vs fixed reference, and when each is right
• The label-delay simulator (phase README Extension A)
• Per-slice drift: the same report computed per segment — drift that only hits one country is invisible in the aggregate

Interview Q&A

Q: PSI on your top feature is 0.4 but the model's trailing AUC is unchanged. What do you do? Investigate before retraining: PSI measures distribution movement, not damage. Check whether the shift is in a region where the model's prediction is flat (no impact), whether it's a data-quality artifact (null spike, unit change — contract layer), and what the prediction distribution did (unchanged predictions + shifted feature = likely benign or compensated). If genuinely benign, recalibrate the threshold for that feature; alert fatigue from impact-free PSI alarms is how monitoring dies.

Q: Why must the retraining validation gate compare against the incumbent on current data rather than the original test set? The original test set measures the old world — both models can score identically there while differing where it matters: the drifted region that triggered retraining. The decision being made is "which model serves today's traffic better," so the gate evaluates on the freshest labeled window (with the label-delay caveat documented). The original test set is still useful as a no-regression check on stable segments.

References

• Rabanser et al., Failing Loudly (NeurIPS 2019) — what shift detectors actually detect
• Evidently AI — the industrial version; map each lab piece to its reports
• Mitchell et al., Model Cards for Model Reporting (2019)
• Gama et al., A Survey on Concept Drift Adaptation (2014)

Phase 13 — Capstone: End-to-End ML Platforms

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2–4 weeks per capstone (pick one to ship; outline a second)

Why This Phase Exists

Phases 01–12 built components; the JD's job is systems. Each capstone composes 4–6 phase artifacts into one running platform with a single entry point, a results report, and the operational discipline (gates, lineage, monitoring) the track teaches. The capstone is the portfolio artifact: the thing you demo in interviews and the proof that the seams — where production ML actually fails — are yours.

Every capstone follows the same contract:

• A walking skeleton first: all stages connected with trivial implementations, one artifact flowing end to end, before any component is deepened.
• One entry point (make demo / python run.py): clean-machine runnable, CPU-only, minutes not hours.
• A money table: the headline results (metrics, latency, comparison vs baseline) at the top of the capstone README.
• An honest-limitations paragraph: what's simulated, what wouldn't survive contact with real scale, and what you'd build next.

The Four Capstones

Pick by target role: recsys/marketplace roles → 01; platform/infra roles → 02; GenAI roles → 03; supply-chain/fintech → 04. Ship one completely; outline a second in a design doc (the system-design/ walkthroughs show the altitude).

Guides in This Phase

• WARMUP.md — integration discipline: skeletons, seams, reports, and turning capstones into interview ammunition

Deliverables Checklist

• One capstone shipped to its full spec (entry point + money table + tests pass)
• Honest-limitations paragraph written
• A second capstone outlined as a 2-page design doc
• Resume bullets drafted from measured numbers (no invented metrics)

Warmup Guide — Capstone

Orientation for Phase 13. No new theory — the capstones test whether Phases 01–12 compose. This guide covers the integration discipline: walking skeletons, seam testing, the report contract, and converting capstones into interview ammunition.

Table of Contents

• Chapter 1: What Changes at Capstone Altitude
• Chapter 2: The Walking Skeleton
• Chapter 3: Seams — Where Integration Fails
• Chapter 4: The Report Contract
• Chapter 5: From Capstone to Interview
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What Changes at Capstone Altitude

A lab asks "can you implement X?"; a capstone asks "do X, Y, and Z work together, and can you prove it with numbers?" Three shifts:

1. Interfaces over implementations: each phase artifact becomes a component with a contract (the feature store takes events, returns point-in-time training frames; the ranker takes candidates + features, returns ordered ids). Integration failures are contract failures — an undeclared assumption (timestamp semantics, id namespaces, score scales) crossing a seam.
2. Numbers over claims: every capstone deliverable is a table — offline metrics vs baseline, latency percentiles, experiment readouts. "It works" is not a result; the money table is.
3. Operations over happy paths: the gates (Phase 08), guardrails (Phase 10), and monitors (Phase 12) are part of the build, not garnish — an injected failure that the system catches is a headline demo.

Chapter 2: The Walking Skeleton

Build the thinnest end-to-end path first: every component present, each trivially implemented (popularity recommender, pass-through guard, constant forecast), one artifact flowing from entry point to report. Then deepen components one at a time, keeping the skeleton green.

Why this beats building components-then-integrating: integration risk — the highest-variance risk — dies in day one instead of week three; the demo exists from the start (and stays demoable through every iteration); and each deepening step has a measurable before/after (the trivial implementation is the baseline your money table needs — a popularity recommender or seasonal-naive forecast is not scaffolding, it's the comparison row).

Chapter 3: Seams — Where Integration Fails

The recurring seam bugs this track has armed you against — check each explicitly:

• Time semantics (P02/P03): event time vs processing time crossing a boundary; a feature computed with as-of semantics trained on, but served with latest-value semantics — the skew you built detection for. Write the seam test: online and offline features for the same (entity, timestamp) must match.
• Leakage across stages (P03/P07): the orchestrator (P08) makes data flows explicit — use its lineage to verify nothing downstream of the label enters features.
• Id and schema drift (P01): every seam gets a data contract; the contracts are the integration documentation.
• Score-scale assumptions (P05/P06): a ranker consuming retrieval scores must not assume calibration; a canary comparing two models must compare like units.
• Clock and RNG injection (P01): the moment two components share simulated time (stream + serving simulator), a single injected clock is the difference between deterministic tests and flaky ones.

Each seam check is one test in the capstone's suite — the suite is small but aimed at the seams, because the components already carry their own phase tests.

Chapter 4: The Report Contract

The capstone README leads with results (the same shape every phase lab used):

1. Money table: headline metrics with baselines — BPR recall@10 0.142 vs popularity 0.031, p99 38 ms at 900 QPS, groundedness 0.94, attack detection 1.00, MASE 0.48 vs seasonal-naive 0.85. Real numbers from your runs; ranges honest.
2. How to run: the one entry point, expected runtime, what appears.
3. Architecture sketch: components and seams, one diagram or ASCII block.
4. Design notes: the 3–4 decisions that mattered, each with the alternative you rejected and why.
5. Honest limitations: what's simulated (single process, synthetic data), what breaks at scale (the in-memory feature store, the brute-force ANN), and the next-step you'd build. This paragraph reads as senior; its absence reads as not knowing.

Chapter 5: From Capstone to Interview

• Every system-design interview answer in this track's domain ("design a recommender / feature platform / GenAI assistant / forecasting service") is a guided tour of a capstone you built — rehearse the tour: 2 minutes end-to-end, then dive where the interviewer steers.
• Resume bullets come from the money table: built X composing Y, measured Z. Never invent scale you didn't run; "simulated at 10k QPS" is honest and still impressive.
• The behavioral bank (interview-prep/05) wants stories with stakes — the capstone's injected-failure demos (the gate that caught the broken data, the canary that rolled back) are exactly those stories in miniature.

Lab Walkthrough Guidance

For whichever capstone you pick:

1. Read its README's stage list; copy the relevant phase labs' solutions into a components/ package (they were designed to be composed — same dataclass idioms, injected clocks/RNGs throughout).
2. Day one: the walking skeleton + the entry point + a trivial money table.
3. Deepen one component at a time; after each, re-run the seam tests and update the table — the git history should read as a sequence of measured improvements.
4. Add the injected-failure demo last (broken data caught by contracts; drifted stream caught by PSI; injection caught by canaries; SRM caught by the analyzer) — one per capstone minimum.
5. Write the limitations paragraph before polishing — it tells you what not to gold-plate.

Success Criteria

The capstone — and the track — is complete when:

1. One capstone runs end to end from a single entry point on a clean machine, CPU-only, in minutes.
2. Its money table shows the deepened system beating its own walking-skeleton baseline on the primary metric.
3. The seam tests pass, and at least one injected failure is demonstrably caught by a gate/guard/monitor.
4. The README fulfills the report contract including honest limitations.
5. You can give the 2-minute architecture tour without notes, and answer "why not ?" for each major design decision.

Interview Q&A

Q: Walk me through the hardest integration bug you hit building this. Have a real one from the seam list — time-semantics mismatches and score-scale assumptions are the usual suspects. The answer's shape: the symptom (metrics looked great offline, diverged in the serving path), the localization method (seam test / lineage query / parity check), the fix, and the guard you added so it can't recur. That last clause is what separates senior answers.

Q: Your capstone is synthetic and single-process. Why should I believe it transfers? Because the failure modes it engineers against are scale-independent: point-in- time correctness, training-serving skew, SRM, drift laundering, injection — all occur identically at any scale; the synthetic setup makes them testable. What changes at scale is the infrastructure (distributed stores, real ANN, real streams) — and the components are interface-shaped precisely so those swap in. Then name the two things that genuinely don't transfer (network partitions, organizational data ownership) — the honest boundary is the credibility.

References

• The track's twelve phase WARMUPs — the actual prerequisite list
• system-design/ — the five walkthroughs, which are the capstones' design docs written in interview form
• The Pragmatic Programmer — tracer bullets (the walking skeleton's origin)
• Hidden Technical Debt in ML Systems — re-read after building; it reads differently now

Capstone 01 — Recommender Platform, End to End

Composes: Phase 03 (feature store) · Phase 05 (LTR) · Phase 06 (BPR/two-stage) · Phase 09 (serving + canary) · Phase 11 (A/B analysis) · Phase 12 (drift) Time: 2–4 weeks

The canonical ML system-design interview ("design a recommendation system"), built instead of described.

What you ship

A single-process platform over a synthetic interaction stream:

events ──► feature store (point-in-time) ──► candidate gen (BPR, P06)
                                                  │ top-200
                                                  ▼
                                            ranker (pairwise LTR, P05)
                                                  │ top-10
                                                  ▼
                            serving simulator (batching + canary, P09)
                                                  │ exposure + click logs
                                                  ▼
                       A/B analyzer (P11)  +  drift monitor (P12)

Stage list

1. Data: synthetic users/items/interactions with popularity skew, user taste clusters, and temporal drift built in (the drift is planted so the monitor has something to catch).
2. Walking skeleton: popularity recommender end to end — this is also your baseline row.
3. Candidate generation: the Phase 06 BPR factorizer; evaluate recall@200 against popularity.
4. Ranking: the Phase 05 pairwise ranker over (user, item) features from the Phase 03 store — point-in-time correct features (the seam test: offline and online features match for sampled (user, t) pairs).
5. Serving: the Phase 09 batching simulator wraps scoring; report the latency-throughput frontier for your actual ranker cost.
6. The experiment: simulate the click model over control (popularity) vs treatment (two-stage); run the Phase 11 analyzer — power check first, SRM check, then the readout with CIs.
7. Operations: Phase 12 drift monitor over feature + score distributions across the planted temporal drift; show the breach → retrain → validate → promote cycle on the candidate generator.

Money table (fill with your runs)

Injected-failure demo (required)

Break the feature pipeline mid-experiment (null out a feature); show the data contract catching it and what the drift monitor + SRM check each saw. This demo is the interview story.

Honest-limitations paragraph (required)

Address at minimum: synthetic click model realism (position bias only as modeled), in-memory stores, brute-force candidate scoring vs real ANN, single-process clock.

Capstone 02 — Streaming Feature Platform with Gated Retraining

Composes: Phase 01 (contracts) · Phase 02 (stream processor) · Phase 03 (feature store) · Phase 08 (orchestrator) · Phase 12 (drift + retraining FSM) Time: 2–4 weeks

The platform-engineering capstone: events in, point-in-time-correct features out, and a retraining loop that cannot launder broken data into a model.

What you ship

event stream (P02 generator: late events, duplicates, planted drift)
   │
   ├─► contract validation (P01) ──quarantine──► dead-letter ledger
   │
   ├─► windowed aggregation w/ watermarks (P02) ──► online store (P03)
   │                                                  │ parity check
   └─► append-only event log ──► offline store ◄──────┘
                                      │
                         point-in-time training frames (P03)
                                      │
                    orchestrated train/validate DAG (P08, cached, lineaged)
                                      │
                  retraining policy FSM (P12): drift- & schedule-triggered

Stage list

1. Stream generator: sessions of events with controlled out-of-orderness, duplicate rate, a schema-break injection point, and a planted distribution shift at a known timestamp.
2. Walking skeleton: events → one windowed feature → online store → a trivial model retrained on schedule. End to end, day one.
3. Hardening: contracts at ingestion (violations → quarantine with reasons — never silent drops); dedup + watermark policy from Phase 02; the online/offline parity test as a continuous check.
4. Training frames: point-in-time joins over the offline store; prove leak-freedom with the Phase 03 leakage test adapted to this data.
5. Orchestration: the Phase 08 DAG runs ingest-validate → build-frames → train → validate → register; show content-hash caching skipping unchanged stages and lineage answering "which model saw the broken events?"
6. The loop: Phase 12 FSM wired to the drift monitor over the planted shift; demonstrate both guard rails — (a) schema break + drift → INCIDENT, not retraining; (b) challenger that fails current-data validation → INCIDENT.

Money table (fill with your runs)

Injected-failure demo (required)

The schema-break injection: contracts quarantine it; the drift monitor flags the survivors; the FSM lands in INCIDENT with reasons; the lineage taint query names every downstream artifact. One incident, four mechanisms, fully narrated.

Honest-limitations paragraph (required)

In-memory stores vs real Kafka/Redis/warehouse; single-process watermarks vs distributed ones; exactly-once effect via dedup, not transactional semantics.

Capstone 03 — GenAI Assistant Platform with Guardrails & Eval CI

Composes: Phase 04 (hybrid retrieval) · Phase 10 (guardrails + eval harness) · Phase 09 (canary) · Phase 11 (experiment readout) · Phase 12 (governance card) Time: 2–4 weeks

The enterprise-GenAI capstone: a RAG assistant whose every prompt/retrieval change passes an evaluation gate, whose attack surface is canary-tested, and whose release is a guarded canary — the JD's GenAI paragraph as a running system.

What you ship

corpus (provided synthetic KB + a poisoned-document injection set)
   │
hybrid retrieval (P04: BM25 + dense + RRF) ──► context assembly (budgeted)
   │                                                │
   │                                          generation (mock LLM by default;
   │                                          real provider behind a flag)
   │                                                │
   └────────► guardrail layer (P10): groundedness · injection · policy · schema
                                                     │
                              eval harness as CI gate (P10 + P08 pattern)
                                                     │
                  canary release (P09) + experiment readout (P11)
                                                     │
                            model/system card (P12) generated per release

Stage list

1. Corpus + golden set: a synthetic knowledge base with known facts (so groundedness has ground truth), a golden Q/A set, a canary attack set, and a poisoned-document subset for indirect injection.
2. Walking skeleton: BM25-only retrieval → extractive answer → harness run. The extractive path is also your fallback and your baseline row.
3. Retrieval depth: Phase 04 hybrid + RRF; measure recall@k on the golden set vs BM25-only — retrieval quality bounds everything downstream; prove it.
4. The guarded loop: Phase 10's full layer wired in; the harness becomes a gate (Phase 08's pattern): groundedness ≥ threshold AND attack detection ≥ threshold AND zero policy violations AND false-refusal rate ≤ threshold.
5. Release machinery: two assistant configs (e.g., different context budgets) through the Phase 09 canary router on simulated traffic; Phase 11 analyzer for the readout — including the SRM check on the router.
6. Governance: the Phase 12 card generated per release: config versions, gate results, known limitations, attack-suite status.

Money table (fill with your runs)

Injected-failure demo (required)

Ship a deliberately degraded prompt template (drops the grounding instruction); show the eval gate blocking the release with the specific failed cases — the "hallucination regression caught in CI" story.

Honest-limitations paragraph (required)

Mock-LLM determinism vs real-model variance; rule-based groundedness scoring limits (paraphrase); synthetic corpus scale; judge-free evaluation (and what Extension A of Phase 10 adds).

Capstone 04 — Forecasting Platform with Backtest-Gated Promotion

Composes: Phase 07 (backtesting + forecaster) · Phase 08 (orchestration) · Phase 09 (batch serving pattern) · Phase 12 (monitoring + retraining FSM) Time: 2–4 weeks

The marketplace/supply-chain capstone: hundreds of series forecast nightly, every model promotion gated on rolling-origin backtests, accuracy monitored per-series in production.

What you ship

synthetic demand panel (200+ series: trend/seasonality/promo/intermittent mix,
                        planted regime change at a known date)
   │
nightly batch DAG (P08): validate ──► feature build ──► per-segment training
   │                                        │
   │                            rolling-origin backtest gate (P07):
   │                            challenger MASE must beat incumbent AND
   │                            seasonal-naive on the same folds
   │                                        │
batch scoring run (P09 batch pattern) ──► forecast store + accuracy ledger
   │
per-series monitoring (P12): trailing MASE control bands, drift on residuals,
retraining FSM with the regime change as the live-fire test

Stage list

1. The panel: heterogeneous series — smooth/seasonal, promo-driven, intermittent (zeros-heavy), short-history — plus one planted regime change. Heterogeneity is the point: one model config will not fit all.
2. Walking skeleton: seasonal-naive for every series, nightly DAG, forecast store, trailing-accuracy ledger. The baseline is the product on day one.
3. Models per segment: the Phase 07 GBM where history suffices; seasonal- naive where it doesn't; (extension: Croston's for intermittent). Segment assignment is itself backtested.
4. The gate: promotion requires beating the incumbent and seasonal-naive on identical rolling-origin folds (MASE, with the fold table archived as the approval artifact — Phase 08 lineage).
5. Operations: per-series trailing MASE with control bands; the planted regime change must (a) trip the per-series monitor, (b) trigger the FSM, (c) produce a challenger that passes the gate on post-change folds.
6. The aggregate report: fleet-level MASE distribution (median + p90 across series — a fleet is judged on its tail, not its average), per-segment breakdown, and cost of the nightly run (cached vs cold, Phase 08).

Money table (fill with your runs)

Injected-failure demo (required)

The regime change plus a broken-promo-covariate injection on a different date: show the residual-drift monitor distinguishing them (regime → retrain path; broken covariate → data INCIDENT path). Two alarms, two different correct responses — that discrimination is the demo.

Honest-limitations paragraph (required)

Synthetic series vs real demand pathologies (stockouts censoring demand!); no hierarchical reconciliation; point forecasts only (quantile extension noted); single-process nightly scale.

Interview Prep — Senior ML Engineer

The four loops a senior MLE candidacy runs through, mapped to this track:

How to use these

• Every answer here is written at the bar the JD sets (7–9+ years): structure first, numbers where they bite, tradeoffs named, limits admitted. Don't memorize prose — internalize the skeletons and regenerate in your own words.
• The coding-adjacent questions are literally the track's labs ("implement NDCG", "build an SRM check", "point-in-time join") — if you built the labs, rehearse by re-deriving, not re-reading.
• For system design, rehearse the 2-minute capstone tour (Phase 13 WARMUP Ch. 5) — interviewers steer; you need entry points, not scripts.

The senior-bar rubric (what interviewers actually grade)

1. Clarify before designing — requirements, scale, constraints, the metric.
2. Baseline first — the candidate who says "popularity / seasonal-naive / logistic regression first, here's why" outranks the one who opens with a transformer.
3. Name the seams — training-serving skew, leakage, feedback loops, label delay. Senior candidates volunteer the failure modes; juniors wait to be asked.
4. Quantify — KV-cache math, sample sizes, latency budgets, cost per 1k requests. Two or three well-chosen numbers per hour signal depth.
5. Decide — end ambiguous tradeoffs with a recommendation and its trigger for revisiting, not a survey.

ML System Design Questions

The method, then four worked questions in compressed form. Full-depth walkthroughs of five designs live in system-design/; these are the interview-time versions.

The method (use it every time)

1. Clarify (3–4 min): who uses it, what decision it makes, scale (users, QPS, items, data volume), latency budget, what "good" means (the metric — and who owns it), constraints (labels available? cold start? regulatory?).
2. Metric & baseline (2 min): name the offline metric, the online metric, and the dumb baseline you must beat. Saying "seasonal-naive" or "popularity" out loud is a senior signal.
3. Data & features (5 min): sources, labels (and their delay!), the feature groups, point-in-time discipline, the feature store split (online/offline).
4. Model(s) (5 min): start simple, justify each escalation; for retrieval- shaped problems, the two-stage candidate-gen → rank architecture.
5. Serving (5 min): pattern (online/batch/streaming), latency budget decomposition, caching, batching.
6. Evaluation & rollout (5 min): offline harness → shadow → canary → A/B with guardrails; the decision rule pre-registered.
7. Operations (5 min): monitoring stack, drift, retraining policy, the feedback loop you're creating and its mitigation.
8. Failure modes & evolution (remaining): what breaks first at 10×, what you'd build next.

The seven seams to volunteer unprompted: training-serving skew · leakage · label delay · feedback loops · cold start · SRM/experiment hygiene · drift laundering. Each maps to a phase of this track; each has a one-line mitigation.

Q1: Design a product-recommendation system (e-commerce, 10M users, 1M items)

Clarify: homepage feed vs item-page "similar items"? (Assume homepage.) Implicit feedback only. 100 ms budget. Online metric: conversion-weighted CTR with revenue guardrail.

Skeleton: two-stage. Candidate gen — BPR/two-tower retrieving ~500 from ANN index (HNSW), unioned with popularity (cold start) and recent-history neighbors; ranker — GBM/LTR over (user, item, context) features from the feature store with point-in-time training frames; business layer — diversity (MMR), exclusions, exploration slice (~2–5% — the feedback-loop tax, stated as such).

Numbers to drop: two-tower trains nightly (items churn), user embeddings computed online from session; ANN recall@500 ≥ 95% vs exact; ranker p99 ≤ 30 ms at batch 500 → feature fetch budget 20 ms → online store required.

Operations: interleaving for ranker iterations (10× sample efficiency), A/B for end-to-end with CUPED on pre-period engagement; per-feature drift with importance weighting; weekly retrain + drift-accelerated.

Evolution: sequence models for candidate gen; real-time features (session intent) — streaming pipeline (Phase 02) is the prerequisite, say so.

Q2: Design a fraud-detection system for payments (5k TPS, decision in 150 ms)

Clarify: block vs review vs allow (three-way, asymmetric costs); labels from chargebacks (30–90 days!) plus analyst reviews (fast, biased); base rate ~0.1%.

Skeleton: streaming features (velocity counters: txn count/amount per card/merchant/device over 5 min–30 d windows — the Phase 02 windowed aggregations, exactly); GBM scorer (calibrated — thresholds are cost decisions); rules layer for hard constraints and analyst agility; review queue as the human-in-the-loop tier.

The label-delay answer (they will probe): proxy monitoring meanwhile — score-distribution drift, fast-label canary cohort (analyst-reviewed sample), backfill joins computing true metrics with honest lag (event-time! predictions joined to their moment); selective-labels bias named: blocked transactions never get labels → propensity-aware evaluation or exploration slice of borderline approvals.

Operations: adversarial drift is the norm — fraud adapts to the model; retraining cadence aggressive (daily-to-weekly), with the Phase 12 guard: data-quality gate before retrain. Asymmetric rollout (Phase 09): shadow for weeks, conservative canary, rollback-biased guardrails (decline rate, review queue depth).

Q3: Design demand forecasting for 50k SKUs × 500 stores

Clarify: horizon (1–14 days), granularity (SKU-store-day), downstream consumer (replenishment — over/under costs asymmetric → quantile forecasts, not point), history depth varies, promos known in advance.

Skeleton: this is a fleet problem — segment series (smooth / seasonal / intermittent / short) and assign methods per segment: seasonal-naive floor everywhere, GBM with lag-≥-horizon features (Phase 07's rule, stated) for rich series, Croston-style for intermittent. Hierarchical reconciliation if planners need coherent totals (bottom-up first; MinT if justified).

The discipline: rolling-origin backtesting as the promotion gate — MASE vs both incumbent and seasonal-naive on identical folds; fleet judged on median AND p90 MASE (the tail is where stockouts live). Nightly batch DAG (Phase 08) with content-hash caching — most series don't change most nights.

The trap to name: stockouts censor demand (sales ≠ demand) — uncensoring or flagging is a data decision that dominates model choice. Naming it usually ends the question.

Q4: Design the ML platform itself (feature store + training + serving for 30 teams)

Clarify: this is a product question — users are ML engineers; the metric is time-to-production and incident rate, not AUC.

Skeleton: golden path over mandates — a paved road (feature definitions as code → registry; orchestrated training with lineage; registry-gated promotion; serving templates with batching/canary built in; monitoring auto-wired from the registry). Each component is a phase of this track (03, 08, 09, 12) — design by walking the lifecycle.

The senior content: what you don't centralize (modeling choices, feature logic) vs what you do (point-in-time join engine, lineage, gates, monitoring plumbing); migration strategy for the 30 teams' existing pipelines (strangler, not rewrite); the two metrics that prove the platform: median days-to-production and % of models with passing monitoring coverage.

Rehearsal protocol

For each question: 25 minutes against a wall-clock, out loud, whiteboard or paper. Then grade against the method's eight steps and the seven seams — which did you skip? The capstone you built (Phase 13) is your worked example for at least one of these; rehearse pivoting from the abstract design to "I built exactly this seam, here's what broke."

ML Fundamentals Questions — Ranking, RecSys, Forecasting, Tabular

Complete answers at the senior bar. Each maps to a track lab — if you built it, re-derive rather than re-read.

Ranking & Retrieval (Phases 04–05)

Q: Implement NDCG and explain every design choice. (This is a real coding screen — Phase 05 lab 01.) DCG@k = Σ (2^rel − 1)/log₂(i+1) over ranked positions; NDCG = DCG/IDCG where IDCG is the DCG of the ideal ordering. Choices to narrate: the log discount (smooth position decay matching attention data; base-2 convention), gain 2^rel−1 (emphasizes high grades; linear gain is the alternative for binary labels), normalization per-query (makes queries with different label counts comparable — and means NDCG can't compare absolute quality across query sets), and the edge case IDCG = 0 (no relevant docs → define 0 or skip query; say which and why it matters for averaging).

Q: Pointwise vs pairwise vs listwise LTR? Pointwise regresses absolute relevance — ignores that ranking only needs order per query and wastes capacity on calibration across queries. Pairwise (RankNet family) optimizes P(i ≻ j) on within-query pairs — matches the task, but counts all inversions equally. Listwise/LambdaRank weights each pair's gradient by |ΔNDCG| of swapping them — inversions at the top matter more, directly optimizing the position-discounted metric. That ΔNDCG weighting is the entire LambdaRank trick (Phase 05 lab implements it); LambdaMART = those gradients in a GBM, still the tabular-LTR production default.

Q: Why hybrid (BM25 + dense) retrieval instead of dense-only? Complementary failure modes: BM25 is exact on terms — wins on identifiers, rare words, exact phrases; embeds nothing it hasn't seen. Dense wins on paraphrase and semantics; fails on out-of-vocabulary precision (part numbers, names) and domain shift from its training data. RRF fusion (1/(k+rank) sums) combines them without score-scale calibration — rank-based, hence robust to incomparable score distributions (Phase 04 lab measures all three on the same eval set; hybrid wins recall@10 on mixed query sets).

Q: What is position bias and how do you train on clicks despite it? Users click what they see; position causally inflates clicks independent of relevance — naive click-trained rankers learn "whatever the old ranker put first." Mitigations: model the bias (click models — examination hypothesis), inverse-propensity weighting (weight clicks by 1/P(examined at position)), randomization slices to estimate propensities, and interleaving for unbiased ranker comparison. Volunteering this question's existence is itself the senior signal.

Recommendation (Phase 06)

Q: Why BPR rather than rating regression for implicit feedback? Implicit data has no negatives — only observed interactions and unobserved items (which mix "disliked" with "never seen"). BPR sidesteps labeling unobserveds by optimizing a ranking assumption: observed ≻ unobserved per user, via σ(score_i − score_j) on sampled triples. The loss matches the product need (rank items) rather than inventing a regression target. (Phase 06 lab: BPR recall@10 ≈ 4–5× popularity baseline on the synthetic set.)

Q: Cold start — new users and new items? New items: content features (text/image embeddings) into the candidate generators so similarity works pre-interaction; exploration boost bounded by guardrails. New users: popularity-within-context (geo, channel), session-based signals from the first clicks (sequence models shine here), onboarding preference capture. Architecturally: cold-start paths are separate candidate sources in the two-stage design, not patches on the MF model — the ranker blends them.

Q: What feedback loop does a deployed recommender create? It trains on clicks over items it chose to show — exposure bias compounds: popular gets popular, the catalog tail starves, and offline metrics computed on logged data stop reflecting counterfactual quality. Mitigations: exploration traffic (an ε slice, logged with propensities), IPS-weighted offline evaluation, diversity injection, and — the honest one — online experiments as the only unconfounded readout (Phase 11).

Forecasting (Phase 07)

Q: Why MASE over MAPE? MAPE divides by actuals: undefined/exploding near zero (intermittent demand!), and asymmetric (over-forecasting a small actual costs more than under). MASE scales MAE by the in-sample seasonal-naive MAE — defined everywhere, symmetric, scale-free across series (fleet-comparable), and interpretable: <1 beats the naive baseline. The M-competitions standardized it for these reasons.

Q: Why can't you use random K-fold CV on time series, and what replaces it? Folds would train on the future of their test points; autocorrelation lets the model memorize the neighborhood — beautiful CV scores, useless deployment. Rolling-origin replaces it: expanding train window, test on the next horizon, advance, repeat — every fold respects time's arrow and matches the deployment cadence. Corollary (the Phase 07 lab's leak alarm): every feature must be lag ≥ horizon or known-in-advance.

Q: When does ML lose to seasonal-naive, and what do you do? Short histories (features can't be estimated), stable pure seasonality (nothing left to learn), and immediately after regime changes (lags stale). Response: keep the baseline as a floor in the fleet (per-segment method assignment), shrink toward it when backtest MASE ≈ 1, and invest in covariates (promos, holidays) — new information beats more model.

Tabular & General (Phases 03, 11)

Q: Why do GBMs still beat deep learning on most tabular problems? Tabular features are heterogeneous, non-smooth, and often axis-aligned — trees' inductive bias (axis splits, scale invariance, native missing-value handling) matches; NNs' smoothness prior doesn't, and typical tabular n is too small to learn the equivalent from scratch. Empirically (Grinsztajn et al.) the gap persists at n < ~1M. NNs win when modality crosses in (text/image columns) or for embedding-sharing across tasks.

Q: What is training-serving skew and the three mechanisms that prevent it? The same feature computed differently at training vs serving time — the model learned f(x) but receives x′. Mechanisms: (1) single-definition features (one codepath/registry feeding both stores — Phase 03's design), (2) point-in-time training joins (training sees what serving would have seen), (3) continuous parity checks (sample online requests, recompute offline, alert on divergence) plus logging served features for training reuse ("log and wait" — the strongest form).

Q: Your model's offline AUC improved but the A/B shows nothing. Enumerate. (1) Offline eval distribution ≠ live distribution (logged-data bias, the recommender loop); (2) the metric gap — AUC moved where decisions don't change (thresholded actions, top-k cutoffs); (3) the experiment is underpowered for the effect size AUC predicts (do the Phase 11 math before concluding "nothing"); (4) serving-path divergence — skew, latency-induced fallbacks eating the gain; (5) the improvement is real but on a segment too small to move the aggregate. Each has a check; run them in cost order.

MLOps & Platform Questions

The lifecycle-ownership loop. Each answer maps to Phases 01–03, 08–09, 12.

Q: What belongs in CI for an ML repo? Be specific about layers. Four layers (Phase 08): (1) code tests — plain pytest, every commit, seconds; (2) data validation — contracts on schemas/ranges/volumes at every ingestion, every pipeline run (a successful training run on garbage data is the worst outcome — it produces a plausible bad model); (3) training smoke test — tiny fixture dataset in CI, asserts loss decreases, pipeline completes, artifact round-trips serialization, <2 min, no GPU; (4) quality gates on real training — metric vs pinned baseline with significance, golden-prediction diff on a fixed probe set (catches "metric same, behavior different"), latency/size budgets. CD is then registry promotion + progressive rollout + auto-rollback — models ship like code, with statistical gates where code has assertions.

Q: Design a feature store. What problem is it actually solving? Two problems wearing one name: training-serving skew (one feature definition → both online and offline stores; never two codepaths) and point-in-time correctness (training joins must reconstruct what serving would have seen at each historical moment — the as-of join over validity intervals, Phase 03's lab). Architecture: definitions-as-code in a registry; batch + streaming materialization into (a) an offline store (columnar, full history, training frames) and (b) an online store (KV, latest values, single-digit-ms p99); a continuous parity checker sampling both. The deeper answer: "log-and-wait" (log served features, train on the logs) is the strongest skew defense and a legitimate alternative for request-scoped features — name when each wins.

Q: How does content-addressed caching make pipelines incremental — and what are its two preconditions? Cache key = H(code version ‖ params ‖ upstream artifact hashes). A change re-keys exactly its downstream closure — invalidation derives from the key structure, no invalidation logic exists; identical re-computed outputs give early cutoff (downstream stays cached). Preconditions: tasks are pure (declared inputs only — hidden state poisons the key) and deterministic (seeded — else identical keys produce different artifacts and the cache is incoherent). The classic homemade bug: omitting code identity from the key, serving stale artifacts after every refactor. (Phase 08's lab is this, tested.)

Q: Two training runs, identical config, different models. Debug order? (1) Data moved underneath — no snapshot pinning; check lineage first, it's the most common cause. (2) Unseeded randomness — init, shuffling, augmentation; audit the seed fan-out. (3) Environment drift — library versions; compare lock hashes. (4) Hardware nondeterminism (GPU atomics, cuDNN autotune) — real but small and last; people blame it first because it's blameless. The systemic fix: run records carrying data-hash + git SHA + lock-hash + seed (Phase 08 Ch. 9), so this question becomes a diff, not an investigation.

Q: Walk me through a model registry's state machine and why promotions are pointer moves. candidate → staging → production → archived, transitions carrying gates (offline quality gate into staging; shadow/canary verification into production; archive records why). Versions are immutable; "production" is an alias the serving fleet resolves — promotion and rollback are therefore pointer moves: instant, deployment-free, and reversible at 2am without a build. The registry is also where governance attaches (approvals, model cards — Phase 12): one artifact, one audit trail.

Q: Your nightly pipeline takes 6 hours and re-trains everything. The team wants a bigger cluster. Alternative? Measure first: how much of the 6 hours is recomputing unchanged work? Content-addressed caching typically converts nightly full-recompute into incremental runs (only changed partitions' downstream closure re-executes) — often a 5–10× wall-clock cut with zero new hardware. Then: per-task profiling (one pandas anti-pattern stage often dominates — Phase 01), parallelizing independent DAG branches, and only then hardware. The senior pattern: capacity requests follow profiling, not precede it.

Q: How do you roll out a new model when you can't A/B test (e.g., a single global pipeline)? Shadow first — full mirror, compare distributions/divergence/latency offline, weeks if the change is risky. Then time-sliced comparison (switchback if effects are short-lived) or geo-split if units exist at coarser granularity — with honest caveats about confounding vs a true A/B. Gate on the offline harness + golden diffs regardless. And state the real answer: if the decision matters enough, build the experimentation capability — most "can't A/B" situations are "haven't invested in assignment infrastructure."

Q: What observability do you wire up before a model's first production day? The five layers (Phase 12), minimally: system health (latency/error/QPS dashboards + SLO alerts); data quality (contracts on the serving inputs — null-rate and range alarms); drift (prediction-distribution PSI daily vs a pinned reference — the highest value per line of code; per-feature PSI weekly, importance-weighted); model quality (the delayed-label backfill join, scheduled from day one even though it fills in later); business KPI linkage (which dashboard does this model move — agreed with the PM in writing). Plus the retraining policy chosen before launch — "when do we retrain" decided by vibe post-incident is how drift gets laundered.

Q: A junior engineer asks why they can't just deploy from their notebook. Give the senior answer, not the bureaucratic one. Because production failures come from the parts the notebook doesn't have: nobody can re-run it (env unpinned, data un-snapshotted), nothing validates inputs when upstream changes next month, no gate stops a silent metric regression, no lineage answers "what made this prediction" during the incident, rollback means archaeology. Then the constructive half: the paved road makes the right way cheaper than the notebook (template repo, one-command pipeline, auto-wired monitoring) — platform work is making virtue the path of least resistance, not writing policy documents.

GenAI Production Questions

The beyond-the-demo loop. Each answer maps to Phase 10 (+ Phases 04, 09, 11, 12 where they compose).

Q: Users say the assistant "hallucinates." Engineer a response. Instrument before fixing: build the golden set, measure the RAGAS-style triangle — context relevance (was the answer in the retrieved chunks?), groundedness (unsupported-claim rate via sentence-level support scoring), answer relevance. Low context relevance → it's a retrieval bug (hybrid search, reranking, chunking — Phase 04), the most common case. Context fine but support low → generation: grounding instructions, citation requirements, temperature, and the groundedness output-guard with extractive fallback. Then gate the metric in CI so the fix can't regress. Skeleton: measure → localize (retrieval vs generation) → fix the right stage → gate.

Q: Define your hallucination metric precisely enough to gate releases on. Unsupported-claim rate: decompose answers into sentence-level claims; score each for support against the retrieved context (token-overlap baseline → NLI → LLM judge as budget allows); rate = unsupported/total over a versioned, content-hashed golden set. Gate = rate ≤ threshold with the eval harness run on every prompt/retrieval/model change. Name the scorer's failure modes (paraphrase → false alarms; coincidental overlap → false passes) and the calibration anchor: periodic human-labeled samples measuring scorer agreement.

Q: Prompt injection — what's the threat model and what actually defends? Two classes: direct (user instructs the model to misbehave — bounded by the user's own privileges) and indirect (instructions embedded in retrieved documents/emails — the attacker programs your system through your corpus, with the system's privileges; any RAG over user-contributed content has this surface). Defense in depth, honestly ranked: (1) privilege separation — the only layer that holds against novel attacks: least-privilege outputs, allow-listed tools with validated parameters, confirmation on side effects; (2) input/context filtering with pattern detectors; (3) output filtering for leakage; (4) canary regression suites — known attacks run on every change, attack-success-rate tracked like accuracy, because detectors decay as attacks evolve. 100% canary detection ≠ safe; it's a regression floor, not a ceiling.

Q: How do you make LLM outputs safe for downstream systems to consume? The reliability ladder: constrained decoding where you control inference (grammar/schema at the token level — malformed output becomes impossible); otherwise validate-then-repair — schema-validate with aggregated path-specific errors, re-prompt with those errors (one round fixes most failures), bounded at 1–2 repairs, then fallback. Never raw json.loads into business logic. Design detail that matters: error-message quality is now system quality — the errors are the repair prompt.

Q: When do you trust LLM-as-judge evaluation? After validating against human labels on a sample (report agreement); with bias mitigations — position (swap orders), verbosity (length-normalize), self-preference (judge ≠ generator family); judge version pinned and re-validated on change. Use for relative comparisons (prompt A/B) where biases partially cancel; not for absolute quality claims. A judge is a model in production — it gets a golden set too.

Q: Your GenAI feature costs $40k/month in tokens. Cut it without losing quality — and prove you didn't. Levers in ROI order: (1) prompt caching for stable prefixes (system prompt + few-shot header — often 30–50% of tokens); (2) context-assembly discipline — retrieval precision up, chunks down (tokens are bought per request); (3) the cascade — router sends head queries to a small model/cache/extractive path, escalates the tail (enterprise traffic is head-heavy; 2–5× cuts are normal); (4) response caching keyed on (model version, canonicalized query) for repeated questions; (5) output-length budgets. "Prove it": the eval harness as the constant — every lever change passes the same golden + canary gates, and the cascade's router gets its own evaluation (its errors are silent quality loss). Cost per request and quality metrics on one dashboard.

Q: Design the rollout for swapping the underlying LLM (provider upgrade). Treat it as a model deployment, not a config change: full eval-harness run (golden + canaries + false-refusal suite) against the new model offline; prompt adjustments expected (models respond differently to the same template — re-tune, re-gate); shadow on mirrored traffic comparing groundedness/latency/ cost distributions; canary with guardrails (Phase 09's router + Phase 11's significance discipline); model/system card updated with the new version's known limitations. The trap to name: judge-based metrics may shift because the judge relates differently to the new model — keep the judge pinned and human-calibrate the delta.

Q: The PM wants the assistant to also execute account actions (refunds, cancellations). What changes? The risk class changes: from "wrong words" to "wrong actions," and indirect injection now has side effects. Requirements before shipping: allow-listed tool registry with parameter validation (amount caps, account-scope checks), confirmation UX for irreversible actions, per-tool authorization tied to the user's privileges (the model never exceeds its principal), full action audit logs, and the canary suite extended with action-hijack attacks. Plus governance (Phase 12): this moves the system up a risk tier — model card, approval chain, human override path, and an incident playbook. The one-sentence version: tools turn an LLM app into an agent, and agents get platform-grade safety engineering, not prompt engineering.

Behavioral & Product-Judgment Questions — the Senior Bar

At 7–9+ years, behavioral rounds test judgment under ambiguity, leading without authority, and product sense — not whether you can narrate a project. Prepare STAR stories from YOUR history; these frames show what strong answers contain and the traps that sink them. Where you lack a work story, your Phase 13 capstone's engineered incidents are legitimate miniatures.

The story bank to prepare (8 stories cover ~every question)

1. Ambiguity → shipped system: a vague business ask you converted into a scoped ML solution (the JD's "translating ambiguous business problems").
2. The wrong-model decision: a time you chose the simpler model/system against pressure — and the metric that vindicated it.
3. Production incident: an ML failure you owned end to end — detection, diagnosis, fix, and the guard that now exists.
4. Killed project: something you stopped (or recommended stopping) with evidence — negative results handled like a senior.
5. Influence without authority: changing another team's/leadership's direction with analysis, not escalation.
6. Mentorship at scale: turning your practice into a pattern others adopted (the JD: "turn best practices into reusable patterns").
7. Disagree and commit: a decision you opposed, argued properly, lost, and executed faithfully — including what you learned about your own argument.
8. The tradeoff under pressure: quality vs deadline with real stakes — what you cut, what you refused to cut, and the explicit risk register.

For each: 90-second version and 4-minute version; the numbers (impact, scale, time) rehearsed; the failure/learning component honest and specific.

Worked frames

Q: Tell me about translating an ambiguous business problem into an ML system. Strong shape: the ambiguous ask ("reduce churn", "make search better") → the questions you asked to find the decision being made (who acts on the output, with what lever, at what cadence) → the metric negotiation (proxy chosen, its gaming risks named) → the deliberately small first system (baseline! — and what it disproved about everyone's assumptions) → the iteration that followed → quantified outcome. Traps: starting the story at the model; metrics chosen by you alone (senior answers show the negotiation); no mention of what you didn't build.

Q: Describe your worst production ML incident. The grading axes: detection (did your monitoring catch it, or a stakeholder? — if the latter, say so and say what monitoring you built after), diagnosis discipline (the bisection: data vs feature vs model vs serving — Phase 12's playbook), blast-radius honesty (who was affected, how long, quantified), and the systemic fix (the gate/contract/alarm that makes recurrence impossible — a fixed bug without a new guard is half a story). The senior tell: you describe your own mistake plainly, without distributing blame.

Q: When did you push back on shipping a model — and were you right? What it tests: whether your quality bar has teeth and evidence. Strong shape: the specific risk (not vibes — an unsupported slice, a leakage suspicion, an unpowered experiment), the evidence you gathered fast (days, not weeks — the push-back that takes a month is a veto, not a review), the alternative you offered (gate it, scope it, shadow it — senior push-back comes with a path), and the outcome including if you were wrong ("the leak was real but small; we shipped with a monitor and my threshold was too conservative — here's how I recalibrated"). Calibration stories outrank vindication stories.

Q: How have you mentored engineers / raised a team's bar? The senior version is systems, not sessions: the review culture you shaped (what your comments taught — cite Phase-06-style severity discipline if it's authentically yours), the pattern you extracted into a template/library that others now use (the JD's exact phrase), the engineer whose growth you can narrate concretely (what they couldn't do, what you did, what they now own). Trap: stories where you did the work for them; the unit of mentorship is their increased capability, not your heroics.

Q: Tell me about a time you disagreed with a technical decision and lost. What it tests: whether you can be overruled without sandbagging. Strong shape: your position with its evidence; their position steel-manned (if you can't state why reasonable people chose it, you weren't listening); the legitimate process that decided; your faithful execution (including not relitigating at the first wobble); and the retrospective — who turned out right, and what it taught you about your own decision-weights. The phrase "I made sure my disagreement was on record and then made their plan work" is the senior chord.

Q: Why this role / why an enterprise + regional context? (For the SAP-style JD specifically.) What it tests: whether you understand enterprise ML's actual shape — fewer greenfield toys, more: reliability bars, governance (Phase 12 is a selling point — use it), legacy integration, customer diversity, and in a regional build-out (KSA/UAE) the chance to set platform patterns from early days rather than inherit them. Connect your track record to that: systems that lasted, patterns you standardized, ambiguity you structured. Avoid the generic "impact at scale" — name the enterprise-specific work you want.

Product-judgment quick fire (have one-paragraph answers ready)

• "The model is 92% accurate. Ship it?" — Accuracy of what, on which distribution, vs what baseline, with what cost asymmetry between error types, measured how (one split? CIs?)? The question is underdetermined and the senior answer is the checklist that determines it.
• "How would you decide between improving the model vs improving the data?" — Error analysis first: if errors cluster on data problems (label noise, coverage gaps, leakage), data wins — and it usually does at enterprise maturity. Quantify: an afternoon of slice analysis before a quarter of architecture.
• "What ML work would you NOT automate?" — Decision rules with legal/ fairness weight (human accountability), incident judgment calls, metric definitions, anything where the cost of silent failure exceeds the cost of human latency. Automation boundaries are a design decision; having thought about them is the signal.
• "A stakeholder wants daily model updates. Push back or comply?" — Ask what decision degrades with staleness and measure drift velocity (Phase 12); if weekly retrains hold accuracy within bands, show the evidence and the cost delta; if dailies are warranted, the ask becomes an SLO and the pipeline gets engineered for it (Phase 08). Neither reflexive yes nor reflexive no — instrument, then decide.

System Design Walkthroughs — Senior ML Engineer

Five full-depth designs at the altitude a senior loop expects. Each follows the method from interview-prep/01 — requirements → metrics & baseline → data/features → models → serving → evaluation & rollout → operations → failure modes & evolution — and each maps to the track phases (and a Phase 13 capstone) where you built the components.

How to use these

• Read each after its phases — these are consolidations, not introductions.
• Rehearse delivery: 2-minute end-to-end tour first, then deep-dive where steered. Interviewers grade the tour's completeness and the dives' depth; candidates who only have one mode (all-overview or all-rabbit-hole) plateau.
• Every design states its numbers (budgets, scales, thresholds) — recompute them for the interviewer's stated scale rather than reciting these.
• The failure-modes section is the differentiator: volunteer them. The seven recurring seams — skew, leakage, label delay, feedback loops, cold start, experiment hygiene, drift laundering — appear across all five designs with design-specific shapes.

Design 01 — Feature Store & ML Data Platform

"Design the data layer for ML at a company where 30 teams ship models." Phases 01–03 and 08 built every component; capstone 02 assembled them.

1. Requirements

Functional: teams define features once and consume them in training and serving; training sets are point-in-time correct; online reads at single-digit ms; batch + streaming sources; feature discovery/reuse across teams.

Non-functional: online p99 ≤ 10 ms at 50k QPS aggregate; offline frames over 2+ years of history; freshness from daily (batch) to seconds (streaming); correctness > everything (a silently skewed feature is worse than an outage).

The two problems this platform exists to solve (state them up front): training-serving skew and point-in-time correctness. Every architecture decision serves one of them.

2. Architecture

                      feature definitions (code, in a registry)
                                   │ one definition → both paths
            ┌──────────────────────┼──────────────────────┐
   batch sources (warehouse)  streaming sources (Kafka)   │
            │                      │                      │
   batch materialization     stream processor             │
   (orchestrated DAG, P08)   (windows/watermarks, P02)    │
            │                      │                      │
            ▼                      ▼                      ▼
   OFFLINE STORE  ◄── parity checker ──►  ONLINE STORE   metadata/
   (columnar, full history)              (KV, latest)    lineage
            │                                  │
   point-in-time training frames        serving reads
   (as-of joins, P03)                   (feature vectors at p99 ≤ 10ms)

3. Deep dives (where interviews go)

Point-in-time joins (the make-or-break): every feature row carries an effective timestamp; a training frame for (entity, t_label) joins each feature's latest value with effective_ts ≤ t_label − serving_lag. The serving-lag term is the subtle part — training must see what serving would have had, including pipeline delay. Implementation: validity intervals (effective_ts, expiry) + as-of join (the Phase 03 lab); at warehouse scale, ASOF JOIN / window functions over partitioned history.

Skew defenses, ranked: (1) single definition — the registry's feature code generates both materializations; two codepaths is the original sin; (2) continuous parity sampling — online value vs offline recompute for sampled (entity, ts), alert on divergence rate (the Phase 03 detector); (3) the strongest form: log-and-wait — log the feature vector actually served, train on the logs; right answer for request-scoped features (model inputs at decision time), at the cost of waiting for history to accumulate. Offer all three; recommend registry + parity for shared features, log-and-wait for per-request context.

Online store: KV (Redis/Dynamo-class), key = (entity_type, entity_id), value = latest vector per feature group, TTLs matched to freshness contracts. Hot-key mitigation (popular items): client-side caching with short TTL. Offline store: lakehouse tables partitioned by date, append-only, time travel — the data side of reproducibility (Phase 08 Ch. 9).

Streaming features (velocity counters, session aggregates): the Phase 02 machinery — event-time windows, watermarks for lateness, idempotent updates. State the freshness-vs-completeness tradeoff explicitly per feature: a fraud counter takes the watermark early (freshness wins); a billing aggregate waits.

4. The platform-product layer (what makes it senior)

• Registry as the contract: definitions versioned, owned, documented, searchable; consumption tracked (who uses feature X — the deprecation question, Phase 11 PMC-style lifecycle).
• Data contracts at every ingestion (Phase 01): schema/range/volume gates; violations quarantine, never silently drop.
• Lineage (Phase 08): feature → sources, model → features; the taint query ("this upstream table was wrong Tuesday") is an incident-response requirement, not a luxury.
• Cost governance: materialization compute is the platform's bill; usage tracking retires dead features (the most common feature-store pathology: write-only features nobody reads).

5. Failure modes (volunteer these)

6. Evolution

Streaming-first (batch as backfill of the stream); embedding features (vector columns + ANN serving — converges with Design 02's infrastructure); cross-team feature marketplace with quality tiers; push-down of PIT joins into the warehouse engine as data volumes grow.

Design 02 — Recommendation System End-to-End

"Design product recommendations for an e-commerce site: 10M users, 1M items, 5k QPS on the homepage, 100 ms budget." Phases 03–06, 09, 11 built the parts; capstone 01 assembled them.

1. Requirements & metrics

Clarify: surface (homepage feed), feedback (implicit: views/carts/purchases, weighted), 100 ms p99 end-to-end, business metric = conversion-weighted engagement with revenue and diversity guardrails. Cold start matters (20% of traffic is new users; items churn weekly).

Metric stack: offline recall@k / NDCG@k against held-out future interactions (temporal split — random splits leak); online CTR/conversion via A/B; the baseline to beat is popularity-by-segment — and it's embarrassingly strong; say so.

2. Architecture — two-stage (the canonical shape and why)

Scoring 1M items per request with a heavy model at 5k QPS is ~5B scores/sec — impossible. The architecture is the funnel:

request ──► candidate generation (~1M → ~500, cheap, recall-oriented)
              ├─ two-tower / MF (BPR) via ANN index          (P06)
              ├─ popularity-by-segment (cold start floor)
              ├─ item-item neighbors of recent history
              └─ exploration slice (2–5%, propensity-logged)
                         │ union, dedup
                         ▼
            ranking (~500 → ~50, precision-oriented)          (P05)
            GBM/LTR over [user × item × context] features     (P03 store)
                         │
                         ▼
            business layer: diversity (MMR), exclusions,
            merchandising rules, position assignment
                         │
                         ▼
            serving: batched scoring (P09), feature cache,
            response cache for anonymous traffic

3. Deep dives

Candidate generation: two-tower (user tower from history/profile, item tower from content+id) trained with BPR/in-batch negatives (Phase 06); item embeddings refreshed nightly into an ANN index (HNSW: recall@500 ≥ 95% vs exact, ~1–2 ms — quote the tradeoff); user embedding computed at request time from the online store's recent-history features. Multiple sources because no single one covers head, tail, and cold start — the union is the design.

Ranking: LambdaMART-style LTR (Phase 05) with NDCG-weighted pairwise training on impression logs. Features: user (long/short-term aggregates), item (popularity, price band, quality), cross (user-category affinity, co-view counts), context (hour, device, session position) — all from the feature store with point-in-time training frames (Design 01's machinery; mention and move on).

Position bias (they will probe): clicks are biased by position; train with IPS-weighted clicks or click-model debiasing; estimate propensities from a small randomization slice; use interleaving (Phase 05) for cheap ranker iterations — 10×+ sample efficiency vs A/B — reserving A/B for end-to-end ship decisions.

The feedback loop (volunteer it): the system trains on exposures it chose — popularity compounds, the tail starves, offline eval on logged data drifts from counterfactual truth. The exploration slice is the answer and a cost; its propensity logs are what make IPS evaluation possible at all.

Latency budget (do the arithmetic aloud): 100 ms p99 = network 10 + candidate gen 15 (ANN + sources in parallel) + feature fetch 20 (one batched online-store read for 500 candidates) + ranking 25 (batched GBM) + business layer 10 + headroom 20. Each component gets its number; the fan-out tail math (Phase 09 Ch. 2) is why headroom exists.

4. Training & operations

Nightly: two-tower retrain + index rebuild; LTR retrain on trailing impression window — both as orchestrated DAGs (Phase 08) with offline gates (recall/NDCG vs incumbent on temporal holdout) and registry promotion. Online: shadow → canary (engagement + latency + revenue guardrails) → A/B with CUPED on pre-period engagement (Phase 11 — engagement metrics have high pre-period correlation; expect 30–50% variance reduction). Monitoring (Phase 12): prediction-score PSI, per-feature drift importance-weighted, coverage metrics (catalog share recommended — the feedback loop's dashboard), and freshness of embeddings/index as first-class alerts.

5. Failure modes

6. Evolution

Sequence models (SASRec-class) for session-aware candidates; real-time feature streaming (Phase 02) for in-session adaptation; multi-objective ranking (engagement + revenue + diversity scalarization → Pareto conversation with product, Phase 09 model-accuracy style); LLM-augmented item understanding for cold-start content features.

Design 03 — GenAI Assistant Platform

"Design an enterprise RAG assistant over internal knowledge, with safety controls, for 50k employees." Phases 04 and 09–12 built the parts; capstone 03 assembled them.

1. Requirements

Functional: Q&A over internal docs (wikis, policies, tickets) with citations; freshness within hours of doc changes; per-user access control on sources; structured actions later (out of scope v1 — say so and why: the risk class changes, see §6).

Non-functional: TTFT < 1.5 s, streaming; groundedness gated (unsupported- claim rate ≤ threshold on the golden set); injection-tested; cost budget per 1k queries; auditability (who asked, what was retrieved, what was answered).

The framing that wins the room: this is a failure-management design — hallucination measured not anecdotal, injection as an attack surface, evaluation as CI. (Phase 10 Ch. 1.)

2. Architecture

docs ──► ingestion: contracts → chunking → embeddings + BM25 index   (P01, P04)
              │  ACL metadata carried per chunk
              ▼
query ──► input guards (injection scan, policy) ─► retrieval (hybrid BM25+dense
              │                                     + RRF, ACL-filtered)  (P04)
              │                                          │ top-k → rerank
              │                                          ▼
              │                              context assembly (token budget,
              │                              source attribution, freshness)
              │                                          ▼
              │                              generation (provider-abstracted,
              │                              prompt registry, streaming)
              │                                          ▼
              └─► output guards: groundedness · policy · schema (P10)
                                       │
                         respond-with-citations │ extractive fallback │ refuse
                                       │
            logs: full trace (query, chunks, prompt version, answer, guard verdicts)

Release machinery around it: eval harness (golden + canary + false-refusal suites) as the CI gate on every prompt/retrieval/model change (P10 + P08); canary rollout with guardrails (P09); per-release system card (P12).

3. Deep dives

Retrieval (quality bound for everything): hybrid BM25 + dense with RRF (rank-based fusion — no score calibration needed); chunking by structure (headings/sections) with overlap, chunk size traded against context budget; reranking (cross-encoder) on the top-50; ACL filtering inside retrieval (post-generation filtering leaks via the answer — access control must bind before the model sees the text). Measure retrieval recall@k on the golden set separately from end-to-end quality — it localizes every "hallucination" debug (Phase 10's triangle).

Groundedness gating: sentence-level claims scored for support against the retrieved context (overlap baseline → NLI as budget allows); below-threshold answers degrade to the extractive fallback (quote the chunks with citations) — degrade, don't fail. The unsupported-claim rate over the versioned golden set is the release gate and the trended KPI.

Injection defense in depth (the corpus is user-writable — indirect injection is the threat model): least-privilege architecture first (v1 has no tools — the strongest defense is having nothing to hijack), context filtering on retrieved chunks, untrusted-span marking, output leak filtering, and the canary attack suite (including poisoned-document canaries planted in a test corpus) with attack-success-rate tracked per release.

Freshness: doc-change events → re-chunk/re-embed pipeline (streaming, Phase 02 pattern); index versioning with atomic alias flips; staleness as a monitored metric per source.

Cost & latency: prompt caching for the stable system prefix; context-budget discipline (retrieval precision over chunk count); the cascade (head queries → small model/extractive path; tail → big model) with the router itself evaluated; streaming with sentence-buffered output guards. Quote the shape: context tokens dominate cost; caching + assembly discipline routinely cut 30–50%.

4. Evaluation & operations

Three cadences (Phase 10 Ch. 7): rule-based guards per request (ms); the harness per change (golden Q/A + canaries + false-refusal benign suite — over-refusal eats the product silently); human-labeled audits monthly calibrating the scorers. Online: thumbs + citation-click rates as weak labels; A/B for major changes (Phase 11 — with SRM on the router); drift on query-topic distribution (Phase 12) catching "users started asking about things the corpus lacks" — which is a content gap surfaced as an unanswerable-rate metric, the single most actionable GenAI dashboard.

5. Failure modes

6. Evolution

Tool use / actions (the risk-class jump: allow-listed tools, parameter validation, confirmation UX, per-user authorization, action audit — interview- prep/04's last question is the script); multi-turn memory with summarization; fine-tuned domain models where retrieval can't close the gap (with the eval harness as the constant across all of it).

Design 04 — Online Experimentation Platform

"Design the A/B testing system itself — assignment, metrics, analysis — for a company running 200 concurrent experiments." Phase 11 built the statistics; Phase 09 the routing; this design is the platform around them.

1. Requirements

Functional: define experiments (arms, traffic %, audience, metrics); deterministic sticky assignment; concurrent non-interfering experiments; automated analysis (the Phase 11 toolkit, productized); guardrail monitoring with auto-stop; the full audit trail.

Non-functional: assignment adds < 1 ms to any request path; analysis pipelines daily (and near-real-time for guardrails); the platform must be trustworthy — its own error rates measured (A/A tests as a first-class feature); self-serve for PMs with statistical guardrails built in (the platform encodes the expertise so users don't have to have it).

2. Architecture

experiment definitions (config service: arms, %, audience, metrics, dates,
                        pre-registered decision rules)
        │
assignment SDK/service:  bucket = hash(experiment_salt : unit_id) → arm
        │   per-experiment salts (decorrelation!), layers for mutual exclusion
        │   exposure logging AT THE MOMENT OF EXPOSURE (unit, exp, arm, ts)
        ▼
metric pipeline (P02/P08): events ⨝ exposures (event-time!), per-unit
        aggregation, metric definitions from a REGISTRY (one definition per
        metric, owned — or every team invents its own "conversion")
        ▼
analysis engine (P11 toolkit): power/MDE preflight · SRM check (every
        experiment, automatically) · z/t tests with CIs · CUPED (auto, where
        pre-period exists) · multiple-comparison policy by metric role
        ▼
results UI + guardrail monitor (near-real-time; auto-stop on breach)
        ▼
decision log: pre-registered rule + outcome + who decided (the audit trail)

3. Deep dives

Assignment correctness (the foundation; get this wrong and nothing above matters): hash-based (sha-class, not language hash()), salt per experiment — reused salts correlate assignments across experiments, silently confounding both; layers for mutual exclusion (experiments touching the same surface share a layer's traffic split; orthogonal layers stack); sticky across sessions/devices via identity resolution (state the limits honestly: logged-out ↔ logged-in stitching is approximate, and that approximation is a known SRM source).

Exposure semantics: analyze exposed units, not assigned ones — a user bucketed into a treatment they never saw dilutes the effect (triggered analysis). Exposure is logged where the experience diverges, not at assignment. This is the #1 analysis bug in homegrown platforms.

SRM as platform hygiene: chi-square on every experiment, every day, automatically (Phase 11 Ch. 5); detected → experiment flagged untrustworthy, analysis blocked, owner paged with the segment breakdown that localizes the loss (browser? geo? date?). Industry rate ~6–10% of experiments — quote it; it justifies the automation.

Metrics registry: every metric = owned definition (event spec, unit-level aggregation, winsorization policy) computed by one pipeline — the experiment platform and the BI dashboard must agree or trust evaporates. Metric roles declared per experiment (goal/guardrail/diagnostic) drive the multiple-comparison policy (Phase 11 Ch. 8): goal uncorrected, guardrails one-sided, diagnostics labeled exploratory.

The statistics engine: Phase 11's toolkit productized — auto-CUPED (pre-period auto-discovered from the registry; 30–50% variance reduction on returning-user metrics = weeks of runtime saved at platform scale), sequential/ always-valid options for monitored experiments (peeking is a platform problem to solve once, not a user-discipline problem to lecture about), and power preflight blocking launches whose MDE exceeds plausible effects ("this experiment cannot answer your question at your traffic — here's what could").

4. Trust machinery (what distinguishes platform design from script design)

• Continuous A/A tests: a slice of traffic always runs no-op experiments; the platform's realized false-positive rate is a published SLO. A platform that has never validated itself produces numbers with unknown error bars.
• Pre-registration enforced softly: decision rules captured at launch; post-hoc metric switches and segment findings are labeled as exploratory in the UI (forking-paths defense as product design).
• Auto-stop: guardrail breaches (error rates, latency, revenue floor) halt exposure without waiting for a human — the Phase 09 canary logic at experiment scale.
• The decision log: every conclusion linked to its pre-registered rule, data snapshot, and analysis version — experiments end in arguments precisely when this is missing.

5. Failure modes

6. Evolution

Always-valid inference as the default; interleaving as a first-class ranker- comparison mode (Phase 05); heterogeneous-effect exploration (uplift surfacing with exploratory labeling); experiment-aware feature flags unifying release engineering and experimentation (one assignment system, two use cases).

Design 05 — Forecasting Platform

"Design demand forecasting for 50k SKUs × 500 stores, horizon 1–14 days, feeding replenishment." Phase 07 built the methodology; Phases 08, 09, 12 the machinery; capstone 04 assembled them.

1. Requirements

Clarify first (these change the design): consumer = automated replenishment (over/under costs asymmetric → quantile forecasts, not points — the P10/P50/ P90 the inventory policy consumes); 25M series at SKU-store grain; histories from 3 years to 3 weeks; promos/holidays known in advance; nightly cadence with a morning deadline; coherent totals wanted by planners (hierarchy).

Metric: MASE per series (scale-free across 25M series — Phase 07's argument); fleet judged on median AND p90 (the tail is where stockouts live); pinball loss for the quantiles. Baseline: seasonal-naive everywhere, forever — it's the promotion gate's second hurdle, not scaffolding.

2. Architecture

sales/inventory/promo/calendar sources
        │ contracts (P01): the stockout-censoring flag is a FIRST-CLASS column
        ▼
nightly DAG (P08: cached, lineaged):
  validate ─► feature build (lag ≥ horizon!, P07) ─► segment router
        │                                               │
        │                  ┌────────────────────────────┼─────────────┐
        │            smooth/rich history          intermittent     short/new
        │            global GBM per horizon       Croston-class    seasonal-naive
        │            (quantile loss)                               + category prior
        │                  └────────────────────────────┼─────────────┘
        ▼                                               ▼
  rolling-origin backtest gate (P07): challenger vs incumbent vs seasonal-naive,
  identical folds, MASE + pinball — fold table archived as the approval artifact
        ▼
  batch scoring run ─► forecast store (versioned) ─► reconciliation (bottom-up /
        │                                            MinT) ─► consumers
        ▼
  accuracy ledger: forecasts ⨝ actuals as they arrive (event-time, P02)
  per-series trailing MASE control bands ─► retraining FSM (P12)

3. Deep dives

One global model, not 25M models (the scale answer): a single GBM per horizon-bucket trained across all rich-history series, with series identity as features (category, store cluster, price band) — cross-series learning is why ML beats per-series classical methods at retail scale (the M5 lesson); per- series methods remain for segments where the global model's features can't exist (3-week histories). The segment router is itself backtested — assignment is a model decision, gated like one.

The leakage discipline (Phase 07's rule, at platform scale): every feature lag ≥ horizon or known-in-advance (promo flags, calendar); enforced structurally — the feature builder takes horizon as a parameter and the CI suite includes the reached-into-the-future test. One leaked lag produces a fleet of beautiful backtests and a quarter of bad orders.

Stockout censoring (the domain trap — raise it unprompted): sales ≠ demand when shelves were empty; training on censored sales teaches the model that stockouts reduce demand → orders shrink → more stockouts (a feedback loop with a supply chain attached). Mitigation: censoring flags from inventory data, uncensoring estimates (or masking censored days from loss), and monitoring the censored fraction as a data-quality metric.

Quantiles & reconciliation: quantile GBM (pinball loss) per required quantile; calibration monitored (P90 should cover ~90% — a measurable promise, Phase 12's control bands). Hierarchy: bottom-up as the default (simple, coherent); MinT where forecast error covariance justifies it — but reconciliation of quantiles is genuinely hard; say so and scope it (coherent medians + approximate quantile adjustments is the honest v1).

The nightly economics (Phase 08): content-hash caching means unchanged series' pipelines skip — most series don't change most nights; the cold-vs- cached task count is a tracked metric. Scoring is embarrassingly parallel; training is the bottleneck → weekly full retrains + nightly scoring with drift-triggered acceleration (Phase 12 FSM), not nightly retrains by reflex.

4. Operations

Per-series trailing MASE with control bands (25M alert streams is noise — alert on aggregates by segment + the worst-N mover list); residual-drift monitoring distinguishing regime change (→ retrain path) from broken covariates (→ data INCIDENT — the laundering guard, Phase 12); the planted-regime-change fire drill from capstone 04 as a staging test. Forecast versioning: consumers pin to a forecast-run id; revisions are new versions (planners diffing yesterday's plan against a silently mutated forecast is how trust dies).

5. Failure modes

6. Evolution

Probabilistic deep models (DeepAR-class) where the GBM plateaus — gated by the same backtest harness (the harness outlives every model choice; that's the design's actual thesis); intermittent-demand upgrades (Croston → TSB); causal promo-effect models feeding what-if planning; reconciliation done properly (MinT with shrinkage) once consumers demand coherent quantiles.