Brother Talk — MLOps

Off the record. The stuff I'd tell you over a beer, not in a design review.

The 2 a.m. truth this whole phase is built around: one day a model in production will be making expensive, confident, wrong decisions, and a room of smart people will not be able to tell you which version is live or what trained it. I've watched it happen at companies you'd recognize. The service was green the whole time — 200s, good latency, happy dashboards — while the model quietly torched money because the data had drifted and nobody was watching the predictions. The reason this phase exists isn't that tracking is intellectually deep (it isn't). It's that the boring plumbing — which model, what made it, is it still good, how do I revert — is the thing that's on fire when it matters, and almost nobody builds it until after the fire.

ML doesn't fail like software, and that messes with people's instincts. A software bug throws; you get a stack trace and a pager. A stale model throws nothing. It degrades. Your brain, trained on software, keeps waiting for an exception that never comes while accuracy bleeds out. Internalize this early: in ML, green is not the same as fine. The most dangerous state is a perfectly healthy service serving a rotting model. If you take one emotional lesson from this phase, make it paranoia about silent failure.

Resume-driven tool sprawl is the disease of this field, and it's seductive. There is enormous social pull toward "we use a feature store and Kubeflow and three trackers." It sounds senior in a hallway. It is, almost always, a team drowning in tooling they can't operate, shipped before they had a second model in prod. I've never once regretted starting with just tracking + a registry and adding the rest when pain demanded it. I have absolutely regretted the opposite. The actual senior move — the one that's unglamorous in the demo and heroic on the on-call rotation — is building the smallest platform that closes the loop, and knowing which famous tool to skip.

Tracking and a registry beat any fancy platform, and it's not close. If a startup handed me one week to make their ML "operable," I would not touch Kubeflow or a feature store. I'd make every training run logged with its params, metrics, git sha, and data version; I'd put the winners in a registry with stages and lineage; and I'd wire one drift alert. That's it. That's 80% of the value of a million-dollar platform, built in a week, and it's exactly what lab-01 + lab-02 teach. The fancy stuff is for when you've outgrown the boring stuff — which most teams never do.

"Tests are evals" is the idea that will make you look senior in any ML-release conversation. In software, CI means the tests pass. In ML, "the training script ran without error" tells you nothing — the model can run fine and be worse. The grown-up move is an eval gate: the candidate doesn't promote unless it provably clears the incumbent's bar and doesn't quietly regress some slice that matters. The number of teams that ship model changes with no eval gate, on vibes, is genuinely frightening. Be the person who builds the gate. It's the cheapest insurance you'll ever write.

Own the loop. That's the job. The classic dysfunction is the data scientist who trains a model in a notebook and throws it "over the wall" to engineers, and nobody owns what happens after deploy. The Senior AI Engineer role — the one that pays and the one that's defensible — is the person who owns the whole loop: data in, model out, monitored, gated, rollback-able, retrained. Not the best modeler. The person who makes models operable. That's rarer and worth more.

Nobody is coming to do the boring part for you. Here's the uncomfortable bit: the modeling work is fun, and everybody wants to do it. The tracking, the registry, the drift alert, the rollback runbook — that's the work people quietly hope someone else does, and so it doesn't get done, and so the model that took three months to train sits in prod with no safety net. The engineer who picks up the boring part becomes indispensable shockingly fast, because they're the only one who can answer the questions that matter when things break. Volunteering for the unglamorous plumbing is one of the highest-leverage career moves in this entire field. I'm not being noble about it — it's self-interest dressed as diligence.

Delayed labels will humble you. The thing that catches even good teams: you usually don't find out if a prediction was right for days or weeks (was that transaction actually fraud? did that churn-flagged user actually leave?). So by the time your accuracy metric drops, you've been wrong for a month and the damage is done. This is why input-drift monitoring (PSI/KS on the features) is worth building even though it feels indirect — it's the only signal you get in real time. Watching the inputs while you wait for the labels is the difference between catching rot in a day and catching it in a quarter.

What's actually worth caring about: the loop, the reproducibility triple (code+data+model), the registry+lineage, and one honest drift signal. What's not worth losing sleep over: picking the "perfect" orchestrator, the exact PSI cutoff to three decimals, or whether you use stages or aliases. The tools churn every two years; the concepts in this phase — versioning, lineage, gating, monitoring the loop — are permanent. Learn the concepts cold and you'll re-skin them onto whatever tool is fashionable when you arrive.

The career framing. Every other phase makes you a better modeler. This one makes you someone a company can trust with production. The first gets you hired as an IC; the second gets you trusted with the system, the budget, and the on-call decisions — and that's the actual ladder. The people who own MLOps are the ones who can say, calmly, at 2 a.m., "it's v6, trained on the March snapshot, PSI spiked on merchant_country, rolling back to v5 now." Be that person. Now go make pytest green.