🛸 Hitchhiker's Guide — Phase 13: Orchestration, Reliability & Observability

Read this if: you want the operability toolkit fast — orchestration, SLOs, data quality, lineage, incident response. Skim, read WARMUP, build the core.

0. The 30-second mental model

This is where "it ran" becomes "it's reliable." You orchestrate DAGs (with caching so re-runs are cheap), set SLOs/error budgets (reliability as a number), monitor data quality (catch corruption before execs do), keep lineage (blast radius), and respond to incidents by finding the shared failure domain. One sentence: make the platform observable, reproducible, and reliable — with numbers, not vibes.

1. Orchestrators

Airflow  → operators + sensors; task-centric; huge ecosystem; manual data deps
Dagster  → software-defined ASSETS; data-aware; content-addressed incrementality
Step Fns → AWS serverless state machines; event-driven; no orchestrator to run
primitive under all: TOPOLOGICAL execution (deps first, no cycles)

2. Incrementality + backfills

content-hash caching: re-run only what CHANGED (code version or upstream output)
safe backfill = idempotent (atomic commit P09) + reproducible (event-time P01)
              + resumable (failed partitions only) + auditable (what/when/from)
replay (stream, P02) + backfill (batch, P06): both need event-time + idempotency
acceptance test: re-run → identical result

3. SLO / error budget math

SLI = measured (% within 60s) ; SLO = target (99.9%)
error budget allowed = total × (1 − SLO)   ← permission to fail
burn rate = actual_error_rate / (1 − SLO)  ← > 1 means you'll blow the budget
alert on BURN RATE (fast + slow windows), not every blip
budget healthy → ship; budget burned → freeze & stabilize

4. Streaming dashboard (deliverable #8)

lag · throughput · WATERMARK DELAY · checkpoint health/duration · backpressure · DLQ rate · restarts
they connect: backpressure → checkpoint duration ↑ + lag ↑ (P04 Round-2 chain)

5. Data quality dimensions

freshness (event-time lag vs SLO) · volume (z-sigma anomaly) · uniqueness (dup keys)
referential integrity (orphans) · schema/semantic (P03) · distribution/cardinality drift

6. Silent corruption (the worst failure)

schema-VALID but WRONG (unit changed, join dropped rows, processing-time double-count)
catch with: reconciliation (batch vs stream agree), data DIFFING, audit tables (balances),
            distribution monitors
principle: it won't announce itself → monitor for ABSENCE of invariants

7. Lineage = the incident compass

upstream(node)  → "this looks wrong, what feeds it?"
downstream/impact(node) → blast radius of a change/incident (P01)
at scale: a lineage browser / data catalog

8. Incident response

FIRST question: "what shared failure domain explains ALL these alarms?" (P01)
runbooks per failure class (Kafka/Flink/Spark/EMR/Athena/Cassandra/S3)
mitigate (failover / rollback to good snapshot P09 / DLQ the poison P02) → THEN root-cause
blameless postmortem → incidents decrease in frequency & severity (a JD success metric)

9. Beginner mistakes that mark you

  1. "Reliable" with no SLO/number.
  2. Backfills that aren't idempotent → duplicate data on retry.
  3. Re-running the whole history when only one partition changed (no incrementality).
  4. Alerting on every error instead of burn rate → alert fatigue.
  5. Only monitoring for errors, not for silent corruption (absent invariants).
  6. Triaging 30 alarms as 30 problems instead of finding the shared domain.
  7. No lineage → can't compute blast radius before a change.

10. How this phase pays off later

  • DAG + caching orchestrates P06 backfills, P09 compaction.
  • Data quality consumes P03 contracts, feeds P14 certification.
  • Lineage/impact = P01 blast radius, productized; incident discipline → P15 leadership.

Read WARMUP, build the core, then P14: security, privacy & governance — locking the platform down (IAM, KMS, PII, Lake Formation, Terraform, DR).