🛸 Hitchhiker's Guide — Phase 12: Scala & Functional Data Engineering

Read this if: you want the Scala/Cats/ZIO/FS2 toolkit fast and to pass Round 4 ("design a type-safe Scala SDK"). Skim, read WARMUP, build the SDK.

0. The 30-second mental model

This is a Scala/JVM role, and the leverage is building platform libraries other engineers can't misuse. You do that with: types that make illegal states uncompilable, Cats abstractions (Validated/Either/Resource), and effect systems (Cats Effect/ZIO/FS2) that make effects values you can compose, retry, and test. One sentence: make the happy path the only path that compiles.

1. Type-level modeling

newtype: OrderId(String) ≠ UserId(String) → can't swap args (won't compile)
smart constructor: Amount.of(cents): Validated → an Amount can NEVER be negative
ADT (sealed trait): closed set of cases → exhaustive pattern match (compiler enforces)
→ whole bug classes moved from runtime to COMPILE time

2. The typeclass ladder

Functor   map      F[A] => (A=>B) => F[B]            transform inside a context
Applicative mapN   F[A], F[B] => F[(A,B)]            INDEPENDENT combine → can ACCUMULATE
Monad     flatMap  F[A] => (A=>F[B]) => F[B]         DEPENDENT sequence → SHORT-CIRCUITS
Traverse  traverse List[F[A]] => F[List[A]]          effectful map over a collection

Applicative = independent + accumulate. Monad = dependent + fail-fast. Pick by independence.

3. Error modeling

Validated (applicative) → collect ALL errors (data contracts, P03)  [ValidatedNel + mapN]
Either / EitherT (monad) → fail-fast (dependent business logic)
errors as a sealed-trait ADT, NOT strings/exceptions → typed, exhaustive

4. Resource safety

bracket(acquire)(use)(release): release ALWAYS runs (even on failure/cancel)
Resource composes monadically → releases LIFO (open A,B → close B,A)
if you HAVE a Resource, the only way to its value is .use → you CAN'T forget to close

5. Effect systems (the big idea)

IO[A] / ZIO[R,E,A] = a VALUE describing an effect; building it does NOTHING; run() performs it
referential transparency → effects are values: pass around, retry, combine, TEST without running
ZIO[R,E,A]: R=deps, E=typed error, A=value. Cats Effect IO[A]: simpler (Throwable errors)
structured concurrency: fibers, parMapN, race, cancellation-safe; Ref/Deferred/Queue/Semaphore

6. FS2 vs Akka Streams (in-service streaming)

FS2     → pure, lazy, back-pressured streams on Cats Effect; parEvalMap(n) = bounded concurrency;
          resource-safe by construction
Akka    → actor model (isolated state, supervision) + Reactive-Streams graphs (Source→Flow→Sink)
          (note: Akka relicensed → Pekko is the Apache fork)
both give P01's backpressure as a typed first-class construct

7. JVM tuning levers

GC: G1 (default) vs ZGC/Shenandoah (low-pause, big heaps) ← GC pauses stall Flink checkpoints (P04)
big state → off-heap/RocksDB (keep heap small) ; spark.memory.fraction (P06)
serialization: Kryo (registered classes) ≫ Java serialization ; a hidden shuffle/checkpoint tax
never block the compute pool with I/O → use a separate blocking pool

8. SDK design (what Round 4 grades)

type-safe API + typed errors + Resource safety + sensible defaults
binary compatibility (MiMa) + semver → upgrading never breaks consumers
ergonomics: good errors, discoverable API, templates with observability baked in
goal: people CANNOT misuse it (the JD's "prevent entire classes of bugs")

9. Beginner mistakes that mark you

  1. Either/monad for validation → fail-fast when you wanted all errors (use Validated).
  2. Stringly-typed everything → swapped-arg bugs; no newtypes/ADTs.
  3. Eager side effects / Futures-that-run-on-creation instead of lazy IO.
  4. try/finally resource handling that someone eventually forgets (use Resource).
  5. Blocking I/O on the compute thread pool → starvation/deadlock.
  6. Breaking binary compatibility in a "minor" SDK release → downstream breakage.
  7. Java serialization in Spark → slow shuffles (register Kryo).

10. How this phase pays off later

  • Validated = P03 contract validator (real).
  • Resource/IO/FS2 = P02 ingestion sidecar, P05 CDC connector.
  • Type-safe SDK = the "internal developer platform" of P13/P15 + JD deliverable #2.
  • JVM/GC tuning = P04 checkpoint stalls, P06 spill.

Read WARMUP, build the SDK, read reference.scala, then P13: orchestration, reliability, and observability — running all of this in production with SLOs.