Warmup Guide — Orchestration, Reliability & Observability

Zero-to-principal primer for Phase 13: orchestration (Airflow/Dagster/Step Functions), backfills/replay, SLOs/SLIs/error budgets, streaming & batch observability, data quality & silent-corruption detection, lineage, and incident response.

Table of Contents

Chapter 1: Orchestration — DAGs, Airflow, Dagster, Step Functions

An orchestrator runs a DAG of tasks in dependency order, with retries, scheduling, and observability. The three you'll meet:

  • Airflow: the incumbent. Python DAGs of operators (do work) and sensors (wait for a condition); a scheduler; rich backfill. Task-centric (you orchestrate operations), with external data dependencies you wire by hand. Battle-tested, huge ecosystem.
  • Dagster: asset-centric — you declare software-defined assets (the datasets), and Dagster tracks their dependencies, types, and freshness, with content-addressed incrementality (re-materialize only what changed — the lab's caching idea). More data-aware; strong typing and testing story.
  • Step Functions: AWS serverless state machines (JSON/ASL) that orchestrate Lambda/ Glue/EMR/ECS steps with retries and branching — great for serverless, event-driven workflows without running an orchestrator.

The unifying primitive is topological execution (the lab's topological_order): run a task only after its dependencies, never with a cycle. The principal-level choice is by ecosystem and data-awareness (Dagster's asset model vs Airflow's operator model vs Step Functions' serverless), captured in an ADR (P15).

Chapter 2: Incrementality, Backfills, and Replay

  • Content-hash caching (the lab's DagRunner): cache a task's output keyed by a hash of its code version + upstream output hashes, so re-running only re-executes what changed. This is what makes a pipeline cheap to re-run and is the basis of Dagster's incrementality and dbt's state comparison.
  • Backfills (P06): recompute history — partition-scoped (by date), idempotent (a retry doesn't duplicate — table-format commits, P09), reproducible (deterministic, event-time logic — P01), resumable (only failed partitions), auditable (what was rewritten, from what input version). The JD's "reproducible and auditable backfills."
  • Replay (P02): re-read the log from an older offset. Correct only with event-time + idempotency. Backfill (batch) and replay (stream) are the two recomputation tools; both rest on the same correctness requirements.
  • The acceptance test for all three: re-run it and get the identical result.

Chapter 3: SLOs, SLIs, and Error Budgets

The reliability vocabulary (Google SRE), applied to data:

  • SLI (indicator): a measured quantity — e.g. "% of events processed within 60s," "% of accepted events durably persisted," "table freshness."
  • SLO (objective): the target for an SLI — e.g. "99.9% within 60s" (the JD's examples).
  • Error budget: allowed_failures = total × (1 − SLO) (the lab). The budget is permission to fail — if you have budget left, ship faster; if you've burned it, freeze and stabilize. This turns reliability from an argument into a number that gates change velocity.
  • Burn rate (the lab): actual_error_rate / (1 − SLO). Burn rate > 1 means you'll exhaust the budget before the window ends; multi-window burn-rate alerts (fast window for acute, slow window for chronic) are the modern alerting recipe — they page on budget burn, not on every blip.

Every production dataset gets an SLO, an owner, lineage, and a quality policy (a JD success metric). "Reliable" without numbers is a wish.

Chapter 4: Streaming Observability

The metrics that tell you a streaming pipeline (P02/P04) is healthy — and the JD's deliverable #8 dashboard:

  • Consumer lag (P02): unconsumed records; rising = falling behind.
  • Throughput: events/s in and out.
  • Watermark delay (P04): how far behind real time the watermark is; rising = event-time trouble or an idle partition stalling watermarks.
  • Checkpoint health/duration (P04): rising duration = backpressure/alignment stalls.
  • Backpressure (P01/P04): the root signal; the UI shows the bottleneck operator.
  • DLQ rate (P02/P03): a step change = an upstream contract break — your earliest alert.
  • Failure/restart rate: job restarts, rebalances.

These connect: backpressure → checkpoint duration ↑ + lag ↑ (the P04 Round-2 chain). A good dashboard shows them together so the one root cause is obvious.

Chapter 5: Batch Observability

For Spark/EMR jobs (P06/P07): job duration trend, shuffle/spill volume, skew (max vs median task time), small-file counts on output (P08), cost per run (P07), partition freshness, and SLA-miss alerts. The batch analog of streaming's lag is "did the table land by its freshness SLO?"

Chapter 6: Data Quality and Silent Corruption

The JD's most-repeated theme: catch bad data before customers or executives do. The dimensions to monitor per dataset (the lab implements several):

  • Freshness: is the newest data recent enough? (event-time lag vs SLO)
  • Volume: is the row count a statistical anomaly vs history? (a sudden drop = upstream broke; a spike = duplication) — the lab's volume_anomaly.
  • Uniqueness: duplicate primary keys (the lab's uniqueness_violations).
  • Referential integrity: fact rows pointing at missing dimensions (the lab's orphans).
  • Schema/semantic validity (P03): contract conformance.
  • Distribution drift / cardinality drift: the values themselves changing shape — the detector aimed at inputs.

Silent data corruption is the worst failure: schema-valid but wrong data (a unit changed upstream, a join silently dropped rows, a processing-time bug double-counted). It passes basic checks and poisons everything downstream. You catch it with reconciliation (do batch and streaming outputs agree? — the lambda/kappa cross-check), data diffing (does the recomputed output match?), audit tables (counts that must balance), and distribution monitors. The principal builds the detection in, because by definition it won't announce itself.

Chapter 7: Lineage and Auditability

  • Lineage: the dataset/field-level graph of "what was computed from what" (the lab's Lineage). It answers the two questions of every incident: upstream ("this looks wrong — what feeds it?") and downstream/impact ("if I change/break this, what's the blast radius?" — P01). At scale this is a lineage browser (a JD platform component).
  • Auditability: who changed what, when; audit tables and logs that let you reconstruct and prove data history (and satisfy compliance, P14).

Chapter 8: Incident Response

When it breaks (it will), the principal is the escalation point (the JD's words). The discipline:

  • Ask "what's the shared failure domain?" first (P01): 12 jobs down that all read one Hive metastore is one incident. Lineage + failure-domain thinking collapses an alarm storm into a root cause.
  • Runbooks: pre-written diagnosis+mitigation for each failure class (Kafka broker instability, Flink checkpoint failures, Spark OOM, EMR spot loss, Cassandra tombstone storms, S3 throttling, Athena failures — the JD's list). A runbook turns a 2 a.m. panic into a checklist.
  • Mitigate, then fix: restore service (failover, rollback to a good snapshot — P09, DLQ the poison — P02) before root-causing.
  • Blameless postmortems: what happened, why, what we'll change (process + automation), so the same incident can't recur. Incidents decreasing in frequency and severity is a JD success metric.

Lab Walkthrough Guidance

Lab 01 — Orchestration & Reliability Core, suggested order:

  1. topological_order — Kahn; cycle detection.
  2. content_hash + DagRunner.run — caching: first run executes all, second none; bump a version → that task + downstream re-run (the incrementality property).
  3. error_budget + burn_rate — the SLO math.
  4. freshness_ok / volume_anomaly / uniqueness_violations / referential_integrity_orphans — the quality dimensions.
  5. Lineage.downstream/upstream/impact — blast radius.

Success Criteria

You are ready for Phase 14 when you can, from memory:

  1. Contrast Airflow / Dagster / Step Functions and the topological-execution primitive.
  2. Explain content-hash caching and the four properties of a safe backfill.
  3. Define SLI/SLO/error budget and burn-rate alerting.
  4. List the streaming and batch observability metrics and how they connect.
  5. Name the data-quality dimensions and how to detect silent corruption.
  6. Use lineage to reason about upstream cause and downstream blast radius.
  7. Run an incident: shared-failure-domain question, runbook, mitigate-then-fix, postmortem.

Interview Q&A

Q: How do you make backfills safe and cheap? Cheap via incrementality — content-hash caching (or partition-scoped reruns) so you only recompute what actually changed, not the whole history. Safe via four properties: idempotent writes (table-format atomic commits, P09, so a retry can't duplicate), reproducible logic (deterministic, event-time — P01, so a rerun yields identical output), resumable (reprocess only failed partitions), and auditable (record what was rewritten and from which input version). The acceptance test is the replay test: run it twice, diff the output, expect zero. That combination is the JD's "reproducible and auditable backfills."

Q: Define an SLO for a critical stream and the alert. SLI: percentage of accepted events processed and queryable within 60 seconds. SLO: 99.9% over a 28-day window. That implies an error budget of 0.1% — the number of late events I'm allowed. I alert on burn rate, not raw breaches: a fast window (e.g. 5 min) catches acute outages, a slow window (e.g. 1 hour) catches chronic degradation, and I page only when the burn rate says I'll exhaust the budget — so I'm not woken by a single slow event but I am for a real regression. When the budget's healthy we ship features; when it's burned we freeze and stabilize. The budget turns "is it reliable enough?" into a number that gates velocity.

Q: What is silent data corruption and how do you catch it? Data that's schema-valid but wrong — a unit changed upstream, a join silently dropped rows, a processing-time bug double-counted. It's the worst kind because it passes basic validation and quietly poisons everything downstream until an exec notices a number is off. You catch it by building detection in: reconciliation (batch vs streaming outputs must agree), data diffing (recomputed output must match), audit tables (counts that must balance), and distribution/ cardinality monitors on the values themselves — plus the volume and uniqueness checks. The principle: corruption won't announce itself, so reliability means monitoring for the absence of expected invariants, not just the presence of errors.

Q: 30 alerts just fired. What's your first move? Not to triage 30 things — to ask "what shared resource explains all of these at once?" Using lineage and failure-domain thinking (P01), a storm usually collapses to one root cause: 12 jobs failing that all read one metastore is a metastore incident; a whole rack of streaming jobs lagging is a Kafka/broker incident. I find the common upstream, mitigate to restore service (failover, rollback to a good snapshot, DLQ the poison), then root-cause and write a blameless postmortem so it can't recur. The calm comes from the model, not from heroics.

References

  • Google SRE Book & Workbook — SLOs, error budgets, multi-window burn-rate alerting
  • Apache Airflow and Dagster docs; AWS Step Functions
  • Data Quality Fundamentals (Moses, Gavish, Vorwerck) — observability for data
  • Great Expectations / dbt tests / Soda — data-quality frameworks (the lab in production)
  • SeniorMLEngineer Phase 12 WARMUP — drift detection, the input-side of these checks