Lab 01 — Experiment Tracking Store

Phase 26 · Lab 01 · Phase README · Warmup

The problem

You are sweeping a model over a dozen hyper-parameter configurations. Three weeks later someone asks: "which run got 0.94 accuracy, what learning rate was it, and can you reproduce it?" If the answer lives in a scrolled-past terminal or a filename like model_final_v2_REAL.pkl, you do not have MLOps — you have folklore. An experiment tracking store is the fix: every training execution becomes a queryable run with its params, its metrics (including the full step-by-step learning curve), its output artifacts, and lineage back to the code and data that produced it.

This is the data model and selection logic underneath MLflow, Weights & Biases, Neptune, and Comet. The UI, the database, and the network hop are theirs; the part that matters — runs grouped under experiments, metric step-history, structured search, best-run selection, and provable reproducibility — is what you build here, in pure stdlib.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• start_run hands out run-1, run-2, ... in order; the same script twice yields identical run records (ids, params, metric history, artifact hashes, tags).
• log_metric in a loop builds a metric_history of (step, value) pairs with auto- incrementing steps; the run's "current" metric value is the last one logged.
• log_artifact returns a deterministic content hash; identical content always hashes the same.
• Runs are grouped by experiment; list_runs(exp) returns only that experiment's runs.
• search_runs filters on metrics/params/tags/status with > >= < <= = != ==, ANDs multiple conditions, excludes runs missing a referenced metric, and can order_by a metric.
• best_run selects correctly for mode="max" and mode="min", excludes runs without the metric, and breaks ties by run id.
• All 28 tests pass under both lab and solution.

How this maps to the real stack

• MLflow Tracking is the closest analogue: mlflow.start_run(), log_param, log_metric (which really does keep a per-step history you can plot), log_artifact, and mlflow.search_runs(filter_string="metrics.accuracy > 0.9"). Our Condition is that filter string already parsed into (kind, key, op, value); real MLflow compiles the string to the same shape before evaluating it against its backing store (a SQL database or the local mlruns/ directory).
• Weights & Biases models the same run/metric/artifact triple with wandb.log({...}) streaming metrics by step, wandb.config for params, and wandb.Artifact for versioned outputs — its artifacts are content-addressed by a hash exactly like hash_artifact, which is how W&B deduplicates identical files across runs.
• Run ids in production are UUIDs, not counters — but every serious tracker treats a run as immutable once finished and content-addresses artifacts precisely so a rerun is diffable. Our sequential ids make that determinism assertable in a test; the lesson (reproducibility is designed in, not discovered) is identical.
• Lineage is the tags in start_run(tags={"git_sha": ..., "data_version": ...}). Real trackers auto-capture the git commit, the entrypoint, the environment, and (with a data-versioning tool like DVC or LakeFS) the dataset hash — so a run points back at the exact code and data. That triple — code + data + config — is the "three axes of change" the Warmup opens with, and it is what makes a model reproducible.

Limits of the miniature. No database, no concurrency, no UI, no distributed writers — a production tracking server handles thousands of concurrent runs writing metrics, which needs a real datastore and careful write batching. Our "artifact storage" is a hash of an in-memory string, not bytes in S3/GCS. Metric history here is append-in-call-order; real trackers also record a wall-clock timestamp per point (we deliberately omit it — wall clock is non-deterministic, per the track's rules).

Extensions (your own machine)

• Add delete_run / restore_run and a lifecycle_stage (active/deleted) field, matching MLflow's soft-delete.
• Add a parent_run_id so a hyper-parameter sweep is one parent run with N child runs (MLflow's nested-run model), and make search_runs able to scope to children of a parent.
• Persist the store to a JSON file and reload it, then prove a reloaded store still round-trips a best_run query — the first step toward a real backing store.
• Wire a real mlflow server in an Extensions script (guarded by an import check) and mirror the same three runs into it, to see your miniature's API next to the real one.

Interview / resume signal

"Built an MLflow-style experiment tracking store: runs grouped under experiments, each carrying params, metric step-history (learning curves, not scalars), and content-hashed artifacts, with a structured search_runs filter, a compare view, and deterministic best-run selection by any metric in min or max mode — reproducibility asserted via sequential run ids and content-addressed artifacts, which is the actual mechanism W&B and MLflow use to make a rerun diffable."