System Design — Hard-Problem Walkthroughs

The JD's five "Example Hard Problems" as 45-minute whiteboard walkthroughs — the design format, not the build (the build is Phase 16). Each follows the same principal recipe so you can run it under interview pressure.

The recipe (P00): 1. Extract the numbers (five forces) → 2. Draw the lifecycle → 3. Place components + name the dials → 4. Defend the seams (correctness) → 5. State the ADRs (deciding factors) → 6. Observability, cost, failure, scale.

Design 1 — Ingest 5M events/sec → Flink → S3/Iceberg → Athena + Cassandra

  1. Numbers: 5M/sec write; what's the read rate & shape (drives serving split)? p99 SLO
    • percentile? correctness per stream (ad-impression at-least-once vs payment exactly-once)? budget? operators? Peak = ?× average (size to peak).
  2. Lifecycle: produce → Kafka/Kinesis (Protobuf, schema-validated, DLQ) → Flink (event-time) → exactly-once sink → S3 raw + Iceberg curated → Athena/Trino (analytics) + Cassandra (features) → governance → replay → retention.
  3. Components/dials: Kafka vs Kinesis (ops↔control↔lock-in); partition count = max(produce, consume) bound; Flink checkpoint to S3; Iceberg + compaction; Cassandra modeled to the feature lookup.
  4. Seams: event-time (not processing-time) so replay is correct; 2PC/idempotent sink so recovery doesn't duplicate; tee raw to S3 before processing (recoverability).
  5. ADRs: ingestion substrate; engine; table format; serving store — each a dial + deciding factor.
  6. Ops/cost/failure: lag/watermark/checkpoint/DLQ dashboards + SLOs; cost = Athena $/TB (layout) + cluster + storage, attributed per team; multi-region failover with RPO/RTO.

Maps to: P01–P04, P08–P11, P13–P15.

Design 2 — Stateful fraud detection (near-real-time)

  1. Numbers: events/sec; alert latency budget; tolerated false-negative vs false-positive cost; out-of-order distribution (sets watermark delay); state size = keys × windows.
  2. Lifecycle: event stream → Flink (event-time, sliding + session windows, pattern detection) ← broadcast rules stream; enrich from Cassandra (async); RocksDB state; checkpoint to S3; alert sink (exactly-once); replay for validation.
  3. Dials: detection latency↔completeness (watermark delay); state size↔cost (TTL, key design); rule agility (broadcast state + savepoint deploys).
  4. Seams: as-of enrichment (no temporal leakage, P05); savepoint upgrade preserves state + no missed/dup alerts; late events → corrected or side-output.
  5. ADRs: Flink (why, vs Spark-SS); RocksDB backend; rule-versioning approach.
  6. Ops: replay reproduces alerts exactly; alert-latency SLO; state-growth alerting.

Maps to: P04, P05, P11, P12 (JVM/state), P13.

Design 3 — Hive/Tez/Flume → Kafka/Flink/Iceberg/Athena migration

  1. Numbers: dataset count & sizes; current query $; small-file count; acceptable downtime (≈ zero); migration timeline.
  2. Phases (strangler-fig, P15): (0) stand up new alongside, read same S3 via catalog; (1) dual-write; (2) validate parity (data diffing, P13) until 0-diff for a soak; (3) cutover consumers incrementally, old warm; (4) decommission Flume/Hive last.
  3. Dials: speed↔risk; cloud-native↔portable; cost↔freshness.
  4. Seams: schema compatibility through transition (P03); idempotent backfills (P06); lineage preserved (P13); rollback at every step.
  5. ADRs: Iceberg adoption; engine cutover order; deprecation plan.
  6. Outcome metrics: query $ ↓, small files ↓, freshness ↑, incidents ↓; present to leadership in cost/risk/time.

Maps to: P03, P06, P07, P09, P10, P13, P15.

Design 4 — Cassandra real-time feature serving

  1. Numbers: read QPS + p99 SLO; write rate (streaming upserts); feature TTL; partition size over a year; consistency need.
  2. Model: design around the read — partition key (avoid hot + large, bucket time series), clustering keys; RF + LOCAL_QUORUM; TTL + TWCS; idempotent upserts.
  3. Pipelines: Flink idempotent upserts (live) + rate-limited Spark backfill (history) — the backfill must not blow the read SLO.
  4. Dials: serving speed↔analytical flexibility; consistency↔latency; backfill throughput↔ read SLO.
  5. Seams: upserts replay-safe (LWW); tombstones bounded (TWCS, no queue pattern); multi-region LOCAL_QUORUM.
  6. Ops: read-path p99 + tombstone-per-read alerts; repair schedule; partition-size monitor.

Maps to: P01, P04, P06, P11, P14 (multi-region).

Design 5 — Enterprise data governance platform

  1. Numbers: dataset/stream count (thousands); PII fields; access principals; compliance regime (GDPR/etc.).
  2. Components: dataset + schema registry (CI gate, P03); field ownership + PII tags (P03); IAM + column/row masking via Lake Formation (P14); lineage graph + impact (P13); data-quality framework (P13); audit log (P14); dataset certification (owner/SLA/lineage/quality, P00/P13).
  3. Dials: self-service↔governance (answer: self-service with guardrails); flexible schema↔safe contract.
  4. Seams: breaking change blocked in CI (never prod); compliant erasure = delete + snapshot expiry (P09/P14); PII masking enforced at the query engine (P10).
  5. ADRs: registry/compatibility policy; access model (ABAC vs ARNs); retention/erasure flow.
  6. Outcome: every dataset certified; PII auditable; breaking changes impossible to ship.

Maps to: P03, P09, P10, P13, P14.

How to Use

  • Time yourself: 45 minutes per design, whiteboard only.
  • Force the numbers first — if you start drawing boxes before extracting requirements, reset.
  • For every box, be ready for "…and now make it cheaper / faster / exactly-once" — name the dial it turns (P00/P15).
  • After each, write the ADRs you'd file. The interview is a live ADR session.