Warmup Guide — Scala & Functional Data Engineering

Zero-to-principal primer for Phase 12: Scala for platform code, the Cats abstractions (Functor→Monad, Validated, Traverse, Resource), effect systems (Cats Effect/ZIO), FS2 and Akka, JVM tuning, and how to build an SDK other engineers can't misuse.

Table of Contents

Chapter 1: Why Scala/JVM for Data Infrastructure

The data world runs on the JVM: Spark, Flink, Kafka, Kafka Streams are written in Scala/ Java, and their performance-critical extension points (UDFs, custom sinks, state functions, serializers) are JVM code. Scala specifically because it combines:

  • Immutability + ADTs for safe modeling of events/contracts.
  • A powerful type system (generics, typeclasses, higher-kinded types) for building reusable abstractions that prevent bugs rather than catch them.
  • Functional + OO so you can be pragmatic.

The JD wants "more than basic Scala syntax" — it wants you to build platform libraries. That requires the abstractions in this phase. (Java is the interop floor; you'll read it and sometimes write it, but the leverage is in Scala's type system.)

Chapter 2: Type-Level Modeling — Make Illegal States Unrepresentable

The principal's favorite phrase. The idea: encode constraints in types so violating them doesn't compile — moving whole bug classes from runtime to compile time:

  • Newtypes / opaque types: OrderId(String) and UserId(String) are different types, so you can't pass a user id where an order id is expected. No more "I swapped two string args" bugs.
  • Smart constructors: a value's constructor is private; the only way to make one is a factory that validates and returns Validated/Either — so an Amount literally cannot hold a negative number anywhere in the program.
  • ADTs (sealed traits): model your events and errors as a closed set of cases; pattern matches are exhaustive (the compiler warns if you miss a case) — so adding a new event type forces every handler to address it.

This is the deepest form of the JD's "prevent entire classes of bugs": the bug can't be written. The lab's typed OrderId/Amount (in reference.scala) show it.

Chapter 3: The Typeclass Ladder — Functor, Applicative, Monad

Cats gives a vocabulary of abstractions, ordered by power. You must know the ladder because which one you reach for changes behavior:

  • Functor — map: transform the value inside a context (F[A] => (A=>B) => F[B]). "I have an Option[Int], make it an Option[String]."
  • Applicative — mapN/ap: combine independent computations (F[A], F[B] => F[(A,B)]). Crucially, applicatives can accumulate (Ch. 4). "Validate three fields independently and collect all errors."
  • Monad — flatMap: sequence dependent computations where each step depends on the previous (F[A] => (A => F[B]) => F[B]). Monads short-circuit (fail-fast). "Load config, then use it to fetch, then …"
  • Traverse — traverse/sequence: turn List[F[A]] into F[List[A]] (e.g. validate a whole list, collecting results). The workhorse for "apply an effectful function over a collection."

The interview-grade distinction: applicative = independent + can accumulate; monad = dependent + short-circuits. Choosing the wrong one (a monad for validation) is why you get fail-fast error reporting when you wanted all errors (P03).

Chapter 4: Validated, Either, and Error Modeling

The concrete payoff of Ch. 3:

  • Either[E, A] (and its monad) — fail-fast: the first Left stops the chain. Right for dependent steps (step 2 is meaningless if step 1 failed). EitherT[F, E, A] stacks it on an effect F (e.g. EitherT[IO, Error, A]) for effectful fail-fast pipelines.
  • Validated[E, A] (applicative) — accumulating: ValidatedNel collects a non-empty list of errors via mapN. Right for independent validations (a data contract: report every violation at once — the lab + P03).
  • Error ADTs: model errors as a sealed trait (NotFound | Downstream(msg) | ...) not strings/exceptions, so handling is exhaustive and typed.
  • OptionT[F, A] similarly stacks Option on an effect.

Rule of thumb: typed errors over exceptions; Validated for parallel validation; Either/ EitherT for sequential business logic.

Chapter 5: Resource Safety and Bracket

A leaked connection/file/cursor is a production incident. bracket (acquire → use → release, release guaranteed even on failure or cancellation) is the primitive; Resource is its composable form:

  • Resource.make(acquire)(release) packages the lifecycle; .use(f) runs f with the resource and always releases it.
  • Resource composes monadically (for { a <- r1; b <- r2 } yield (a,b)), releasing in LIFO order (open Kafka, open Cassandra, … close Cassandra, close Kafka) — even if use throws or the fiber is cancelled. The lab tests exactly this LIFO-on-exception property.
  • This is compile-time-enforced discipline: if you have a Resource, the only way to get its value is .use, which guarantees release. You can't forget to close it.

This is correctness most languages get via try/finally discipline (easily forgotten); Cats makes it a type you can't misuse.

Chapter 6: Effect Systems — Cats Effect and ZIO

The modern way to write correct concurrent, effectful code:

  • Referential transparency: an IO[A] (Cats Effect) or ZIO[R,E,A] (ZIO) is a value describing an effect, not the effect itself. Building it does nothing; running it (unsafeRunSync/the runtime) performs it — and running it twice performs it twice. This is the lab's IO. Why it matters: effects become values you can pass around, retry, combine, and test without performing them.
  • ZIO's ZIO[R, E, A]: three type params — R (environment/dependencies it needs), E (typed error channel), A (success). Encodes dependency injection and typed errors in the type. Cats Effect's IO[A] keeps it simpler (errors via Throwable, deps via constructor/Resource).
  • Structured concurrency: fibers (lightweight green threads) with parMapN/race/ start; cancellation is safe and propagates; Ref (atomic state), Deferred (promise), Queue, Semaphore for coordination. You get massive concurrency without thread-pool juggling or callback hell.
  • Error handling: handleErrorWith, retry/recover; the lab's retry combinator is a miniature (Cats Effect's real one does exponential backoff + jitter — see reference.scala).

The principal value: effect systems make concurrency composable, cancelable, resource-safe, and testable — the opposite of ad-hoc threads and futures.

Chapter 7: FS2 and Akka Streams

Two ways to build streaming inside a service (vs cluster engines, P05):

  • FS2 (Functional Streams for Scala): pure, lazy, back-pressured streams on Cats Effect. Stream[IO, A] — compose with map/evalMap/parEvalMap(n) (bounded concurrency = backpressure), and resource safety is built in (Stream.resource releases even mid- stream). The functional choice for connectors, ingestion sidecars, CDC consumers.
  • Akka Streams: a Reactive-Streams implementation on the Akka actor runtime — Source → Flow → Sink graphs with backpressure via demand signaling. Plus actors (isolated state, message-passing, supervision hierarchies) for stateful concurrent components. Widespread, battle-tested; a different (effects-as-actors) philosophy than FS2's pure effects. (Note Akka's licensing change pushed some orgs to Pekko, the Apache fork.)

Both give the backpressure of P01 Ch. 8 as a first-class, typed construct.

Chapter 8: JVM Performance and SDK Design

  • JVM tuning (the JD's "JVM memory, GC, thread pools, serialization, profiling"):
    • GC: G1 (default, balanced) vs ZGC/Shenandoah (low-pause, large heaps — good for big Flink/Spark state); watch GC pause time and frequency (a top cause of latency spikes and Flink checkpoint stalls, P04).
    • Heap & off-heap: size the heap, but large state belongs off-heap/RocksDB (P04) to avoid GC; tune spark.memory.fraction (P06).
    • Serialization: Java serialization is slow/large; Kryo (with registered classes) is the Spark standard; Flink has its own serializers. Serialization is a real shuffle/ checkpoint bottleneck.
    • Thread pools & blocking: never block the compute pool with I/O — Cats Effect/ZIO give you a separate blocking pool; mixing them is a classic deadlock/starvation bug.
  • SDK design (Round 4): a platform library is judged by whether people can't misuse it. Type-safe APIs (Ch. 2), error modeling (Ch. 4), resource safety (Ch. 5), sensible defaults, binary compatibility (don't break the ABI across minor versions — MiMa checks this), semantic versioning, and ergonomics (good error messages, discoverable API). The JD's deliverable #2 — a Flink framework with templates + observability baked in — is this.

Lab Walkthrough Guidance

Lab 01 — Functional Data SDK (Python companion), suggested order:

  1. Validated.map/map2 — map2 accumulates on the invalid path (applicative).
  2. Result.flat_map — fail-fast (the second step doesn't run after an error).
  3. bracket then Resource.make/flat_map — release on exception, then LIFO composition.
  4. IO.map/flat_map/run — lazy: building does nothing; the lazy-until-run + runs-each-time tests are the point.
  5. retry — a new lazy IO with bounded attempts.
  6. Read reference.scala to see each in real Cats/Cats-Effect/FS2.

Success Criteria

You are ready for Phase 13 when you can, from memory:

  1. Explain "make illegal states unrepresentable" with newtypes, smart constructors, ADTs.
  2. Place Functor/Applicative/Monad/Traverse on the ladder and say what each does.
  3. Explain why Validated accumulates and Either short-circuits, and when to use each.
  4. Explain Resource/bracket and the LIFO-release-on-failure guarantee.
  5. Explain referential transparency and why effects-as-values are composable/testable.
  6. Contrast Cats Effect IO and ZIO ZIO[R,E,A]; contrast FS2 and Akka Streams.
  7. Name the JVM tuning levers (GC, off-heap, Kryo, blocking pools) and SDK design principles.

Interview Q&A

Q (Round 4): Design a Scala SDK so product teams define event schemas, validate contracts, build stream processors, and write to the lakehouse — safely. The spine is make misuse uncompilable. Events are typed ADTs with newtype ids and smart constructors (illegal states unrepresentable, Ch. 2). Decoding returns ValidatedNel so a bad payload yields all contract violations at once (Ch. 4), routed to a DLQ (P02/P03). Processing is expressed as effects (IO/ZIO) so it's lazy, composable, cancelable, and testable without side effects (Ch. 6); resources (Kafka, Cassandra, the Iceberg writer) are Resources so they cannot leak (Ch. 5). The lakehouse write is an exactly-once two-phase-commit sink (P04) exposed as a single safe API. Cross-cutting: typed errors (not exceptions), a retry combinator with backoff, observability baked into the templates, and binary compatibility + semver so upgrading the SDK never breaks consumers. The win is a library where the happy path is the only path that compiles.

Q: Why Validated instead of Either for validation? Because validation steps are independent and you want all the errors, while Either is a monad that short-circuits at the first Left. Validated is an applicative: mapN runs each validation and accumulates failures into a NonEmptyList, so a user submitting a bad record gets "name required, age must be ≥ 0, currency invalid" in one pass instead of fixing them one at a time. The flip side: use Either/EitherT for dependent business logic where step 2 is meaningless if step 1 failed — there short-circuiting is correct. Same shape (E and A), opposite combine semantics; pick by independence.

Q: What does an effect system buy you over Futures/threads? Referential transparency and structured concurrency. An IO/ZIO is a value describing an effect, so nothing runs until the edge of the program — which makes effectful code composable, retryable, and testable without performing side effects, and eliminates the "Future starts running the moment you create it" footgun. On top, fibers give cheap structured concurrency with safe cancellation that propagates, Resource guarantees cleanup even under cancellation, and typed error channels (especially ZIO's E) make failure explicit. Net: you write highly concurrent, resource-safe code that you can reason about and test, instead of juggling thread pools and callbacks.

Q: A Flink/Spark job has latency spikes and checkpoint stalls. JVM angle? Prime suspect is GC pauses: large on-heap state with G1 can produce long stop-the-world pauses that stall the operator and delay checkpoint barriers (P04). Fixes: move big state off-heap/RocksDB so the heap stays small, switch to a low-pause collector (ZGC/Shenandoah) for large heaps, and size the heap to avoid constant pressure. Also check serialization (use Kryo with registered classes, or Flink's serializers — Java serialization is a hidden tax on every shuffle/checkpoint) and blocking calls on the compute pool (offload I/O to a blocking pool). These JVM levers are exactly the "debug at the byte and GC level" the role expects.

References

  • Functional Programming in Scala (Chiusano & Bjarnason) — the red book; the ladder & laws
  • Cats and Cats Effect docs
  • ZIO docs — ZIO[R,E,A], fibers, layers
  • FS2 and Akka/Pekko Streams
  • Practical FP in Scala (Gabriel Volpe) — building a real app with these tools
  • JVM: Optimizing Java (Evans, Gough, Newland); ZGC/G1 docs for GC tuning