Lab 02 — Functional Data SDK in real Scala (Cats + Cats Effect)

Phase: 12 — Scala & Functional Data Engineering | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

Lab 01's Python companion taught the semantics. This is the real thing — an idiomatic Scala SDK built on Cats and Cats Effect, the exact stack Spark/Flink/Kafka-Streams platform teams use, with ScalaTest specs that compile and pass via sbt test. This is what Round 4 ("design a type-safe Scala SDK") actually wants to see.

What it demonstrates

FileConceptCats/CE feature
Domain.scalamake illegal states unrepresentablevalue classes + private smart constructor + ValidatedNel
Contract.scalaaccumulating data-contract validationmapN (applicative) collects all errors
Resources.scalaresource safetyResource.make → guaranteed LIFO release on failure
Effects.scalaeffects-as-values + retryIO, handleErrorWith, exponential backoff

The specs prove the principal-grade properties: validation accumulates (not fail-fast), resources release in LIFO order even when use throws, retry recovers after transient failures, and IO is lazy (building ≠ running).

Run

sbt test

First run downloads Scala 2.13, Cats, Cats Effect, and ScalaTest (a minute or two), then compiles and runs all specs. Expected: 8 tests, all green, across ContractSpec, ResourceSpec, EffectsSpec.

sbt compile      # just typecheck
sbt console      # a REPL with the SDK + Cats on the classpath

Layout

build.sbt                         # Cats + Cats Effect + ScalaTest
project/build.properties          # sbt version
src/main/scala/datasdk/           # the SDK (Domain, Contract, Resources, Effects)
src/test/scala/datasdk/           # ScalaTest specs (the proof)

See also ../lab-01-functional-sdk/reference.scala for an extended Cats/FS2 reference (FS2 streaming, EitherT, an IOApp) — read it for breadth.

Success criteria

  • sbt test → all 8 specs pass.
  • You can explain why Order.validate uses mapN (accumulate) and not a for-comprehension (flatMap, fail-fast), and when each is correct.
  • You can explain why ResourceSpec sees release cassandra before release kafka even though use raised — and why that guarantee is impossible to forget with Resource.
  • You can explain why building ref.update(_ + 1) changes nothing until the IO is run.

Extensions

  • Add an FS2 Stream that reads a Resource, processes with parEvalMap(n) (bounded backpressure), and prove resource safety across the stream.
  • Add a EventContract[A] typeclass (a schema + a ValidatedNel decoder) and derive instances — the seed of the platform library the JD asks for.
  • Port Effects.retry to add jitter, and write a deterministic test using Cats Effect's TestControl (virtual time) instead of real IO.sleep.
  • Wire a fake "lakehouse writer" as a Resource[IO, Writer] with an idempotent commit — the exactly-once sink (P04) as a typed, leak-proof API.