Phase 12 — Scala & Functional Data Engineering

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 3 weeks (40–50 hours) Prerequisites: Phase 03 (Validated/contracts), Phase 01 (the concurrency this models)

Why This Phase Exists

The JD is emphatic that this is a Scala/JVM role: "build production-grade Scala services and libraries … Cats abstractions such as Functor, Applicative, Monad, Traverse, Validated, EitherT, OptionT, Resource … effect systems such as Cats Effect or ZIO … Akka or Akka Streams … JVM memory, GC, thread pools … build internal SDKs." And: "This role expects more than basic Scala syntax. You should be able to build reusable frameworks and platform libraries that prevent entire classes of bugs." Round 4 is exactly "design a type-safe Scala SDK."

This phase teaches the abstractions that let you build a platform library other engineers cannot misuse — type safety that makes illegal states uncompilable, resource safety that makes leaks impossible, and effect systems that make concurrency composable and testable.

Concepts

  • Scala for data infra: case classes, ADTs (sealed traits), pattern matching, immutability, type parameters, given/implicits (typeclasses); why Spark/Flink/Kafka-Streams internals are Scala/JVM.
  • Type-level modeling: make illegal states unrepresentable — newtypes/opaque types (an OrderId is not a String), smart constructors returning Validated, ADTs for exhaustive error/event modeling. The bug-class-eliminating discipline.
  • Cats typeclasses: Functor (map), Applicative (independent combine — Validated accumulates), Monad (dependent sequence — fail-fast), Traverse (List[F[A]] → F[List[A]]), Validated, Either/EitherT, OptionT, Semigroup/Monoid, Resource.
  • Effect systems (Cats Effect IO, ZIO ZIO[R,E,A]): referential transparency (describe vs run), composition, fibers/structured concurrency, cancellation, Ref/Deferred, resource safety, error channels — the modern way to write correct concurrent code.
  • FS2: pure, back-pressured streaming on Cats Effect; resource-safe streams; parEvalMap for bounded concurrency — the in-service pipeline tool (P05).
  • Akka / Akka Streams: the actor model (state isolation, supervision) and Reactive-Streams back-pressured graphs — the older but widespread alternative.
  • JVM performance: heap/GC (G1/ZGC), thread pools, blocking vs async boundaries, serialization (Kryo/Java), off-heap, profiling — what "JVM tuning" means in practice.
  • SDK/library design: type-safe APIs, error modeling, retry semantics, binary compatibility, versioning, developer ergonomics — building the platform's internal libraries.

Labs

Lab 01 — Functional Data SDK (flagship, implemented)

FieldValue
GoalBuild the core FP abstractions: Validated (accumulating), Result/Either (fail-fast), Resource/bracket (LIFO release on failure), IO (lazy effect), retry — plus authentic Scala/Cats reference source
ConceptsApplicative vs monad, resource safety, referential transparency, effect combinators
How to Testpytest test_lab.py -v — 19 tests (Python companion); reference.scala for the real Scala
Talking PointsWhy is Validated applicative and Either monadic? Why does IO do nothing until run? How does Resource guarantee release?
Resume bulletBuilt a functional data SDK (accumulating validation, resource-safe brackets, a lazy effect type, retry combinators) with an authentic Cats/Cats-Effect/FS2 reference implementation

→ Lab folder: lab-01-functional-sdk/

Lab 02 — Functional Data SDK in real Scala (implemented, sbt test)

FieldValue
GoalThe real Scala version of Lab 01: an idiomatic Cats + Cats Effect SDK — typed domain (value classes + smart constructors), ValidatedNel accumulating contracts (mapN), Resource LIFO-safe lifecycles, IO effects + retry — verified with ScalaTest
ConceptsMake-illegal-states-unrepresentable, applicative validation, resource safety, effects-as-values — in production Scala
How to Testsbt test — 8 specs across ContractSpec/ResourceSpec/EffectsSpec (real Scala, runs here)
Resume bulletBuilt a type-safe Scala data SDK on Cats/Cats-Effect (accumulating validation, resource-safe lifecycles, effectful retry) with ScalaTest coverage

→ Lab folder: lab-02-scala-sdk/ (the reference.scala from Lab 01 is the extended Cats/FS2 tour)

Extension Project A — Real Scala SDK (spec)

Port reference.scala into an sbt project; add munit-cats-effect tests; publish a tiny event-contract library (schema + Validated decode) and consume it from a second module — exercise binary compatibility (don't break the API).

Extension Project B — JVM tuning memo (spec)

Profile a (synthetic) Spark/Flink JVM: GC pauses, heap pressure, serialization hotspots; recommend GC (G1 vs ZGC), heap sizing, off-heap, and Kryo registration — the JD's "JVM performance tuning."

Integrated-Scenario Hooks

  • This phase's Validated is P03's contract validator, for real.
  • Its Resource/IO/FS2 are how you build P02's ingestion sidecar and P05's CDC connector.
  • Its type-safe SDK design is the "internal developer platform" of P13/P15 and the JD's deliverable #2 (a Flink stream-processing framework with templates).

Guides in This Phase

Key Takeaways

  • This is a Scala/JVM role; you build platform libraries, not just pipelines.
  • Make illegal states unrepresentable: types eliminate whole bug classes at compile time.
  • Validated accumulates (applicative); Either/Monad fail-fast (dependent) — use each correctly.
  • Resource/bracket guarantee release even on failure; IO/ZIO make effects lazy and composable.
  • A great SDK prevents misuse by construction; that's the JD's bar ("prevent entire classes of bugs").

Deliverables Checklist

  • Lab 01 implemented; all 19 tests pass; you've read reference.scala
  • You can explain applicative vs monad and when each is right
  • You can explain resource safety and referential transparency
  • You can sketch a type-safe event-contract SDK API
  • Extension A real Scala SDK or Extension B JVM tuning memo