Phase 13 — Orchestration, Reliability & Observability
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (30–38 hours) Prerequisites: Phases 04/06 (the jobs being orchestrated), Phase 01 (failure domains)
Why This Phase Exists
This phase makes the platform operable — the force (P00) juniors ignore and principals obsess over. The JD demands orchestration (Airflow/Dagster, Step Functions, backfill & replay frameworks), reliability (SLOs, SLIs, error budgets, incident response, runbooks, capacity & cost), data quality (freshness, volume, uniqueness, referential integrity, anomaly detection, dataset certification), observability (streaming + batch dashboards, lineage, audit), and detecting silent data corruption. Its deliverables include a "streaming observability dashboard," a "data quality framework," and "incident response playbooks." This is where "it works in the notebook" becomes "it has run reliably for a year."
Concepts
- Orchestration: DAG scheduling (topological execution), Airflow (operators, sensors, scheduler, backfill) vs Dagster (software-defined assets, types, content-addressed incrementality) vs Step Functions (serverless state machines); idempotent tasks, retries, SLAs.
- Backfill & replay frameworks: partition-scoped, idempotent, reproducible, resumable, auditable recomputation (P06) + log replay (P02) — the platform's recovery superpower.
- SLOs / SLIs / error budgets: define measurable objectives ("99.9% of critical streams process within 60s"), measure indicators, spend the error budget to gate change velocity, multi-window burn-rate alerting.
- Streaming observability: lag, throughput, watermark delay, checkpoint health/duration, backpressure, DLQ rate, failure rate (P02/P04) — the streaming dashboard.
- Batch observability: job duration, shuffle/spill, skew, small-file counts, cost per run (P06/P07).
- Data quality & correctness: freshness, volume anomaly, uniqueness, referential integrity, duplicate detection, late-data reconciliation, data diffing, audit tables, dataset certification, and detecting silent data corruption (drift detectors aimed at inputs).
- Lineage & auditability: field/dataset-level lineage graph, blast-radius (impact) analysis, audit logging.
- Incident response: runbooks, on-call, the "what's the shared failure domain?" discipline (P01), postmortems, and the runbooks for Kafka/Flink/Spark/EMR/Athena/Cassandra/S3 failures the JD enumerates.
- Capacity & cost: capacity planning (Little's Law, P01), cost reviews, cost attribution.
Labs
Lab 01 — Orchestration & Reliability Core (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a DAG scheduler with content-hash caching, SLO/error-budget math, data-quality checks, and a lineage/impact graph |
| Concepts | Topological execution, incremental caching, error budgets, data quality, lineage |
| How to Test | pytest test_lab.py -v — 18 tests |
| Talking Points | Why does content-hash caching make backfills cheap+reproducible? When does burn rate page you? What's blast radius? |
| Resume bullet | Built an orchestration + reliability core: content-addressed DAG caching, SLO/error-budget tracking, data-quality checks, and lineage/impact analysis |
→ Lab folder: lab-01-orchestration-core/
Extension Project A — Streaming observability dashboard (spec)
Define the metrics + alerts for a Flink pipeline (lag, watermark delay, checkpoint duration, backpressure, DLQ rate, failure rate) and the multi-window burn-rate alert rules — the JD's deliverable #8.
Extension Project B — Incident runbook + postmortem (spec)
Write a runbook for one failure (e.g. "Flink checkpoint duration rising + lag climbing", P04) and a blameless postmortem template — the JD's deliverable #12.
Integrated-Scenario Hooks
- This phase's DAG + caching orchestrates the P06 backfills and P09 compaction jobs.
- Its data-quality checks consume P03's contracts and feed the P14 governance certification.
- Its lineage/impact is the blast-radius reasoning of P01, productized.
Guides in This Phase
Key Takeaways
- Operability is a first-class force; SLOs/error budgets make reliability measurable.
- Content-hash caching = cheap, reproducible backfills (re-run only what changed).
- Data quality is monitored across freshness/volume/uniqueness/integrity — catch corruption before customers do.
- Lineage turns an incident into one root cause via blast-radius reasoning.
- Streaming observability = lag + watermark delay + checkpoint health + backpressure + DLQ rate.
Deliverables Checklist
- Lab 01 implemented; all 18 tests pass
- You can define an SLO/SLI/error budget for a data product
- You can list the streaming + batch + quality metrics to monitor
- You can use lineage to compute the blast radius of a change
- Extension A dashboard spec or Extension B runbook+postmortem