Warmup Guide — Production Python Engineering

Zero-to-expert primer for Phase 01: the Python engineering practices that make ML systems survive production — typing as design, error discipline, honest concurrency, testing that catches regressions, and profiling that finds the truth.

Table of Contents

• Chapter 1: What "Production-Grade" Means for ML Code
• Chapter 2: Typing as API Design
• Chapter 3: Dataclasses, Immutability, and Value Objects
• Chapter 4: Error Design — Collect, Don't Raise
• Chapter 5: Generators and Pipeline Composition
• Chapter 6: Concurrency Without Folklore
• Chapter 7: Testing Discipline
• Chapter 8: Profiling and the Measure-First Ethic
• Chapter 9: Packaging and Reproducibility
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What "Production-Grade" Means for ML Code

ML code has a specific pathology: it runs even when it's wrong. A web service with a type error crashes; an ML pipeline with a silent unit mismatch trains a worse model and tells no one. So production-grade ML code is defined by how loudly it fails:

1. Boundaries are guarded: every ingestion point validates its inputs against an explicit contract (the flagship lab) — because upstream will change a column's meaning without telling you.
2. Interfaces are typed: the type signature is the documentation that can't go stale, and mypy/pyright turn a class of runtime surprises into edit-time squiggles.
3. Behavior is pinned by tests: not "does it run" but "does it still compute the same NDCG on the golden file."
4. Performance claims are measured: the profiler, not intuition, names the bottleneck.

The rest of this guide is those four habits, mechanized. Everything later in the track (stream processors, feature stores, experiment analyzers) is written in this dialect.

Chapter 2: Typing as API Design

From zero: Python type hints are annotations (def f(x: int) -> str) checked by external tools (mypy, pyright), not the runtime. Their value compounds with codebase size and team size — exactly the senior-engineer regime.

The constructs that carry ML platform code:

• Protocol — structural typing: define what an object must do, not what it must inherit:

class Validator(Protocol):
    def validate(self, value: object) -> list[str]: ...

Anything with a matching validate method satisfies it — no inheritance, no registration. This is how you accept "anything model-like" (.predict()) or "anything rule-like" without coupling callers to your class hierarchy. Prefer Protocol over ABC at boundaries you don't control; ABCs still fit when you own the hierarchy and want shared behavior.

• Generics: class Result(Generic[T]) / def first(xs: Sequence[T]) -> T — containers and pipelines that preserve their element types end to end. An untyped pipeline returns Any and infects everything downstream; a generic one keeps IDE/checker leverage.
• Discriminating unions: int | None forces the None-check; Literal["mean", "sum"] makes string options checkable; assert_never() in the else branch makes enum matches exhaustive — adding a variant then fails the type check at every unhandled site (the same make-illegal-states-unrepresentable discipline the PMC track teaches for state machines).
• The pragmatic stance: type the public surface 100%, the internals as helpful; Any is a quarantine zone you shrink over time; never lie to the checker (cast/# type: ignore with a reason comment only).

Chapter 3: Dataclasses, Immutability, and Value Objects

@dataclass(frozen=True, slots=True)
class Field:
    name: str
    dtype: type
    required: bool = True

• @dataclass generates __init__/__repr__/__eq__ — value semantics for free.
• frozen=True: instances are immutable — hashable, safely shareable across threads, usable as dict keys, and immune to the spooky-mutation-at-a-distance bugs that plague config objects passed through five layers. Schemas, configs, and events should be frozen by default; accumulators and builders stay mutable.
• slots=True: no per-instance __dict__ — smaller, faster attribute access, and typo-proof (obj.requierd = ... raises instead of silently creating an attribute — a real bug class).
• The value-object pattern: parse/validate raw input once at the boundary into a frozen, typed object; everything downstream trusts it ("parse, don't validate" — the same principle, third track running). The flagship lab is this pattern as a library.

Chapter 4: Error Design — Collect, Don't Raise

The exception model serves control flow; data validation needs a different shape. If a record has five problems, raising on the first reports one and hides four — and in a batch of a million records, fail-on-first turns data triage into whack-a-mole.

The pattern (the lab's core):

@dataclass(frozen=True)
class ValidationIssue:
    path: str          # "user.age" — where
    rule: str          # "range(0, 150)" — what
    message: str       # human-readable why

@dataclass(frozen=True)
class ValidationReport:
    issues: tuple[ValidationIssue, ...]
    @property
    def ok(self) -> bool: return not self.issues

Rules: validators return issues (empty = pass), never raise for data problems; exceptions are reserved for programming errors (schema misconfigured, contract violated by the caller); paths make issues actionable at scale (group by path+rule and you have a data-quality dashboard — which is exactly what Phase 02's pipeline does with this library's output). Error messages are UX: include the offending value (truncated), the expectation, and the path — the person reading it is debugging at 2am with no context.

Chapter 5: Generators and Pipeline Composition

Data work is stream-shaped; generators are Python's native streams:

• A generator function (yield) produces lazily — constant memory over any input size, and composable: clean(parse(read(path))) builds a pipeline that pulls one record at a time through all stages.
• The traps that bite in production: single consumption (a consumed generator is silently empty — the "second epoch is empty" bug; materialize or re-create), late binding in closures (loop-variable capture), and exhaustion hiding errors (a generator that raises mid-stream has already yielded results — partial output without a clear failure marker; wrap stages with error channels when completeness matters).
• itertools literacy (islice, chain, groupby — sorted input!, tee — memory caveat) replaces a hundred hand-rolled loops.
• This chapter is why Phase 02's stream processor will feel natural: windows and watermarks are generator pipelines with timestamps.

Chapter 6: Concurrency Without Folklore

The truths, stated plainly:

• The GIL: one thread executes Python bytecode at a time per process. Threads therefore do not speed up CPU-bound pure-Python code — at all. They do speed up I/O-bound work (the GIL releases during blocking I/O) and C-extension work that releases it (numpy ops, many model runtimes).
• The decision table:

• asyncio in one paragraph: cooperative single-threaded concurrency — await yields control to the event loop; anything that blocks without awaiting freezes everything (the classic bug: a synchronous model call inside an async handler — Phase 09 and the CV track's serving phase both hit this). It buys massive I/O concurrency, not parallelism.
• Process pools' fine print: arguments and results are pickled (cost and pickleability constraints); prefer chunked work over per-item tasks; on Linux fork vs spawn semantics differ (copied state surprises).
• The senior reflex: name the bound (CPU? I/O? memory bandwidth?) before choosing the tool — the same diagnosis-first ethic as every performance chapter in this curriculum.

Chapter 7: Testing Discipline

The testing stack for ML systems code, in order of leverage:

1. Example tests (pytest, parametrize): the bread and butter; table-driven cases keep them dense and readable.
2. Property-style tests: instead of hand-picked cases, assert invariants over generated inputs — round-trips (decode(encode(x)) == x), idempotence (normalize(normalize(x)) == normalize(x)), oracle equivalence (optimized vs naive implementation on random inputs — the differential-testing pattern used throughout this curriculum's labs). The hypothesis library industrializes this; a seeded loop over random inputs gets you 80% with zero dependencies (the lab does the latter).
3. Golden tests: freeze a known-good output artifact (a validation report, a metric value); fail on any diff. Cheap regression armor for numeric code — with the discipline that golden updates are reviewed, not reflexive.
4. Fixtures for shared setup; test doubles only at true boundaries (network, clock, randomness — inject the clock and the RNG; code that calls time.time()/random() inline is untestable by construction, a lesson Phases 02 and 11 enforce structurally).
5. The ML-specific layer — data validation as tests, model-quality gates, golden-prediction files — arrives in Phase 08; it stands on this chapter.

Chapter 8: Profiling and the Measure-First Ethic

• cProfile (python -m cProfile -s cumtime script.py): deterministic, whole-program, function granularity. Read cumtime for "where does time go," tottime for "who is doing the work."
• py-spy (py-spy top --pid / record for flamegraphs): sampling, attachable to running production processes, near-zero overhead — the production instrument.
• line_profiler for the one hot function cProfile names.
• The pandas/numpy reality: most data-code slowness is one of four shapes — row-wise apply where a vectorized op exists, repeated concat in a loop (quadratic), object dtypes where native ones fit, or accidental copies of large frames. Extension B is a guided hunt for exactly these.
• The ethic (this curriculum's most repeated sentence): profile before optimizing; re-measure after. An optimization without a before/after number is a rumor.

Chapter 9: Packaging and Reproducibility

The minimum that makes your work installable and re-runnable:

• pyproject.toml + src layout (src/contracts/): the import-path hygiene that prevents "works from the repo root only."
• Pinned environments: a lock file (pip-tools/uv/poetry — pick one) for applications; looser ranges for libraries. "It worked last month" bugs are unpinned-dependency bugs ~half the time (the CV track's Pillow-resize incident is the canonical ML example).
• Determinism hooks: seed everything seedable (random, numpy, torch) through one function; surface the seed in logs/configs. Reproducibility is a feature you build, not a property you hope for — and it's a prerequisite for Phase 11's experimentation discipline.

Lab Walkthrough Guidance

Lab 01 — Typed Data-Contract Library, suggested order:

1. Value objects first: Field, ValidationIssue, ValidationReport (frozen, slotted). Get report.ok and issue aggregation right before any rules exist.
2. Built-in rules as small classes satisfying the Rule protocol — type check, range, regex, enum, nullability. Each returns a list of issues (Ch. 4).
3. Schema.validate(record) — walk fields, run rules, aggregate with paths; unknown-key and missing-key policies explicit.
4. Coercion ("42" → 42 where the schema says int, with issues on failure) — note how coercion-then-validation ordering is a design decision; document yours.
5. Composition: schema.extend(...), schema.subset(...) — frozen objects make these return new schemas naturally.
6. The property-style loop: generate random valid records from the schema, assert they validate clean; mutate one field at a time, assert exactly one issue at the right path. This test finds rule bugs nothing else does.

Success Criteria

You are ready for Phase 02 when you can, from memory:

1. Argue Protocol vs ABC and write a Protocol from scratch.
2. Justify frozen+slots dataclasses for configs/schemas with two concrete bug classes they prevent.
3. State the collect-don't-raise pattern and where exceptions still belong.
4. Recite the concurrency decision table and explain the GIL's exact consequence.
5. Name the four pandas slowness shapes and the profiler that finds each.
6. Write a property-style test (round-trip or oracle) without reference.

Interview Q&A

Q: Your feature pipeline silently produced garbage for three days. What engineering practices would have caught it in three minutes? Boundary contracts (schema validation at ingestion — unknown column, dtype drift, and range violations all fire on record one), golden tests on the pipeline's output statistics, and typed interfaces so the upstream rename couldn't have silently mapped to Any. The meta-answer: ML code runs-when-wrong, so the discipline is making wrongness loud — validation, types, goldens, in that order of leverage.

Q: Why doesn't threading speed up your pure-Python feature transform, and what are your actual options? The GIL serializes Python bytecode across threads — CPU-bound pure Python gains nothing. Options: vectorize into numpy/pandas (the usual 10–100× answer — the GIL releases inside C), ProcessPoolExecutor with chunked work (true parallelism, pickling costs), or move the hot loop to a compiled layer (numba/Cython). And measure first — if the transform is actually I/O-bound on reads, threads do help and the question was a trap.

Q: How do you test code that depends on time, randomness, and a database? Inject all three: a clock object (or freezegun-style patching) instead of time.time(), a seeded random.Random/numpy.random.Generator passed in, and a repository interface with an in-memory fake at the boundary. The design rule that makes testing possible is dependency injection of nondeterminism — code that reaches for global time/randomness is untestable by construction, which is why the track's later labs (stream windows, bandits) all take clocks and RNGs as parameters.

References

• mypy docs and typing module docs — Protocol, generics, Literal
• Fluent Python (Ramalho, 2nd ed.) — ch. 5 (dataclasses), 17–18 (iterators/context), 19–21 (concurrency)
• pytest docs — fixtures, parametrize; Hypothesis docs — when you graduate from hand-rolled property loops
• Alexis King, Parse, don't validate — the boundary-pattern essay
• py-spy and cProfile docs
• pydantic and pandera — the industrial versions of the flagship lab; read their docs after building yours
• Brett Slatkin, Effective Python (2nd ed.) — items on generators, concurrency, and testing