Lab 01 — Orchestration & Reliability Core

Phase: 13 — Orchestration, Reliability & Observability | Difficulty: ⭐⭐⭐⭐☆ | Time: 6–7 hours

The control plane of a data platform in miniature: a DAG scheduler with content-hash caching (the incrementality behind Dagster/Airflow), SLO/error-budget math, data-quality checks, and a lineage graph for blast-radius reasoning. These are the systems that turn "it ran" into "it's reliable, observable, and reproducible."

What you build

  • topological_order — Kahn's algorithm; reject cycles
  • DagRunner + content_hash — execute tasks in order, caching by content hash so only changed work re-runs (bump a task version or change a seed → that task + downstream re-execute; everything else is reused)
  • error_budget / burn_rate — SLO/SLI reliability math (allowed failures, budget burn)
  • freshness_ok / volume_anomaly / uniqueness_violations / referential_integrity_orphans — the core data-quality dimensions
  • Lineage — downstream / upstream / impact (blast radius)

Key concepts

ConceptWhat to understand
Topological orderdependency-respecting execution; cycles are illegal
Content-hash cachingre-run only what changed → cheap, safe backfills
Error budgetallowed = total×(1−SLO); burn rate > 1 = page-worthy
Data quality dimsfreshness, volume, uniqueness, referential integrity
Lineage / impactthe blast radius of a change or incident

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 18 tests pass — test_version_bump_reexecutes_task_and_downstream and test_second_run_uses_cache certify the incremental-orchestration model.
  • You can explain why content-hash caching makes backfills reproducible and cheap.
  • You can compute an error budget and a burn rate and say when to page.

Extensions

  • Add retries with backoff and a task state machine (pending→running→retrying→ succeeded/failed) to the runner.
  • Add a partition-aware backfill: re-run only the date partitions whose inputs changed.
  • Add multi-window burn-rate alerting (fast 5-min + slow 1-hour windows) — Google SRE's alerting recipe.
  • Emit lineage + run metadata as events (the seed of a lineage browser / data catalog).