« All Roles

Senior ML Engineer — Production ML & GenAI Systems Curriculum

Target Roles:

• SAP — Senior ML Engineer, Production ML & Generative AI Systems (the reference JD — jd.md)
• Booking / Spotify / Zalando / Shopify — Senior/Staff ML Engineer, Ranking & Recommendations
• Uber / DoorDash / Careem — Senior ML Engineer, Marketplace ML (forecasting, pricing, matching)
• Databricks / Snowflake / AWS — ML Platform Engineer / Solutions Architect, ML
• Any enterprise AI team — Senior MLE owning data → features → models → serving → experimentation → governance end to end

Duration: 26 weeks core (6 months) — extendable to 12 months with capstone depth Seniority Target: Senior / Staff IC (7–9+ years equivalent depth) Goal: Make you the engineer who can own a production ML system end to end — data pipelines, feature stores, ranking/recsys/forecasting models, serving infrastructure, GenAI applications with guardrails, online experimentation, observability, and governance — and who can lead its technical direction.

Why This Curriculum Exists

The Senior ML Engineer role this curriculum targets is the full-lifecycle role: not the researcher (who hands off a model), not the pure platform engineer (who never owns a metric), but the engineer accountable for the system that moves a business number — from the Kafka topic to the A/B readout.

The JD makes the breadth explicit: data pipelines, feature stores, training workflows, model serving, online experimentation, observability — across deep learning, NLP, ranking, recommendation, forecasting, semantic retrieval, and LLM applications — plus MLOps, distributed systems, and evaluation and governance (offline benchmarking, online experimentation, hallucination analysis, model risk, responsible AI). Most engineers are deep in one or two of these. Senior means: deep in three or four, conversant and dangerous in all of them, and able to design the seams between them — because in production, the failures live at the seams (training-serving skew, eval-online metric divergence, drift nobody monitored).

This track is the seam-first curriculum. Every phase builds a working system component with tests, and every WARMUP teaches the failure modes that distinguish "it works in the notebook" from "it has run for a year."

How this track relates to the other tracks

Shared topics are cross-linked, never duplicated: this track's GenAI phase builds the production layer on the LLM track's RAG foundations; its serving phase builds the platform layer on the LLM track's inference mechanics.

What You Will Build

Every phase ships a tested, runnable artifact. By the end you will have:

1. A typed data-contract library — schema validation with composable rules and property-style tests (the production-Python entrance exam)
2. A mini stream processor — event-time windows, watermarks, late-data handling, idempotent exactly-once-effect aggregation
3. A mini feature store — point-in-time-correct joins (the leakage killer), online/offline parity checking, skew detection
4. A hybrid retrieval engine — BM25 + dense + reciprocal-rank fusion with a labeled-set evaluation harness
5. A learning-to-rank model from scratch — pairwise/LambdaRank-style training with NDCG implemented and tested by hand
6. An implicit-feedback recommender — matrix factorization with BPR, evaluated on recall@k/coverage against a popularity baseline
7. A forecasting backtesting harness — rolling-origin evaluation, seasonal-naive baselines, MASE/sMAPE, and a feature-based ML forecaster that must beat the baseline
8. A mini pipeline orchestrator — DAG execution with content-hash caching, retries, and an artifact ledger (the heart of every MLOps platform)
9. A model-serving simulator — dynamic micro-batching, latency SLO measurement (p50/p99), canary-vs-control comparison logic
10. A GenAI guardrail & evaluation harness — injection canaries, groundedness checks, refusal policy engine, structured-output validation
11. An online experimentation toolkit — power analysis, SRM detection, CUPED variance reduction, and a Thompson-sampling bandit
12. A drift & retraining engine — PSI/KS detectors, retraining policy state machine, and a model-risk report generator
13. Capstone platform projects — composing the above into end-to-end systems

Folder Structure

SeniorMLEngineer/
├── README.md                                   ← you are here
├── jd.md                                       ← target role profile (SAP Senior MLE)
├── phase-01-production-python-engineering/     ← typed, tested, profiled production Python
├── phase-02-data-engineering-streaming/        ← batch vs streaming, Kafka semantics, windows, watermarks
├── phase-03-feature-stores-training-serving-skew/ ← point-in-time correctness, online/offline parity
├── phase-04-semantic-retrieval-search/         ← BM25, dense, hybrid fusion, retrieval evaluation
├── phase-05-learning-to-rank/                  ← LTR objectives, NDCG, position bias
├── phase-06-recommendation-systems/            ← MF/BPR, two-tower, candidate-gen → rank architecture
├── phase-07-forecasting/                       ← baselines, backtesting, hierarchical & intermittent demand
├── phase-08-training-orchestration-mlops/      ← DAGs, caching, experiment tracking, registries, CI/CD for ML
├── phase-09-model-serving-platform/            ← serving patterns, batching, caching, canary, cost
├── phase-10-genai-production-rag-guardrails/   ← production RAG, guardrails, hallucination analysis
├── phase-11-online-experimentation/            ← A/B stats, SRM, CUPED, sequential tests, bandits
├── phase-12-observability-drift-governance/    ← monitoring, drift, retraining strategy, model risk, responsible AI
├── phase-13-capstone/                          ← end-to-end platform capstones
├── interview-prep/                             ← system design, fundamentals, MLOps, GenAI, behavioral
└── system-design/                              ← five full design walkthroughs

Each phase contains:

26-Week Schedule

Prerequisites

• Solid Python (3.10+) — this track sharpens it to expert level; it doesn't teach basics
• SQL fluency and comfort with pandas/numpy
• ML fundamentals: train/val/test discipline, overfitting, gradient descent (the CV track Phase 02 is the refresher if needed)
• Basic Docker and git
• For the GenAI phase: the LLM track Phase 07 WARMUP as background
• No paid cloud account needed: every lab runs locally (numpy/pandas/sklearn/torch), simulating the distributed component where necessary — the concepts are cloud-portable by design

Interview Relevance

This track maps directly to the four interview loops senior MLE candidates face:

• ML system design ("design a recommendation system / fraud detector / forecast pipeline") → phases 03–07 + system-design/
• ML fundamentals + coding → every flagship lab is a past interview question in disguise (implement NDCG; build an A/B significance checker; point-in-time join)
• MLOps/platform depth → phases 08–09, 11–12
• GenAI production judgment → phase 10 + the governance half of phase 12
• Behavioral at senior level → interview-prep/05: product judgment, ambiguity, leading without authority