Brother Talk — Custom Models in Production

Off the record. The stuff I'd tell you over a beer, not in a design review.

Here's the career-shaped truth nobody says out loud: the person who fine-tunes the model and the person who ships it are usually not the same person, and only one of them gets promoted. I've watched this play out over and over. Someone spends three weeks on a beautiful QLoRA run, posts a notebook with a lovely eval table, says "it's ready," and then somebody else spends the next three weeks turning that .safetensors file into a service with a readiness probe, an autoscaler, and a rollback plan. Guess who gets asked to lead the next thing. The gap between "I trained a model" and "I run models" is maybe two weeks of learning and it is worth an entire level. That gap is this phase.

Deployment feels beneath you until the day it isn't. The modeling work is genuinely fun — you're doing math, watching loss curves, reading papers. Packaging feels like plumbing. So people skip it, and then their model sits in a bucket for six months while everyone agrees it's "basically done." I have never once regretted learning the deployment layer. I have repeatedly regretted assuming someone else would.

The thing that will actually hurt you is that a bad model deploy doesn't throw. This is the single most important instinct to rewire. You come from software, where wrong means an exception, a stack trace, a pager. Here? You point a LoRA at the wrong base checkpoint and nothing happens. Shapes match. No error. The model is still fluent, still fast, still returning 200s — and quietly worse. You find out from an eval you happened to run, or from a customer, eleven days later. Every "why is this validation so paranoid?" in Lab 01 and Lab 02 is a response to that one fact: when there's no runtime signal, correctness has to move to build time. Pin everything. Bind the digest. Make the config parser hostile. It feels excessive right up until the first time it saves you.

Nobody will tell you that most of your GPU bill is idle time. Everybody argues about GPU sticker prices — H100 versus A100, this cloud versus that one — and almost nobody computes their utilization. I've seen teams spend a month negotiating a 15% discount on instances they were using 20% of the time. Fixing utilization was worth 4×, sitting right there, for free. When someone shows you a $/1M number, the first question is always "at what utilization?" Nine times out of ten they haven't measured it, and that's the whole conversation.

The honest answer to "should we self-host?" is usually "not yet," and being the person who says that is worth more than being the person who builds it. Against a cheap hosted small model you will probably lose on token price forever — someone else runs it at higher utilization than you ever will and passes a bit of it on. Say so. Then make the argument that's actually true: latency, data residency, control over a model that otherwise changes under you on a Tuesday, and a capability nobody sells. Those are real, they're often decisive, and they don't need a fake break-even chart propping them up. The engineer who kills their own project's premise and replaces it with a better one gets trusted with bigger things. The one who ships a business case built on 100% utilization gets a very awkward quarterly review.

Multi-LoRA is the closest thing to a free lunch in this whole field, and most teams don't know it exists. One base model, forty 16-megabyte adapters, one GPU. It's a 40× cost collapse for a few percent of latency, and the reason people don't do it is simply that "one deployment per model" is the obvious architecture and nobody stopped to check the ratio. If you're at a B2B company promising "an AI trained on your data," walk in with this and you will look like you invented electricity. Just — please — resolve the adapter from the authenticated session and not from a request field, because the other version of that story is a data-leak incident with your name on it.

Fine-tuning is over-prescribed and it's a status thing. "We fine-tuned a model" sounds more impressive in a standup than "we improved the retriever." So teams fine-tune knowledge problems, and get a model that is more confident about facts it still doesn't have, which is strictly worse than where they started. Meanwhile the highest-ROI customization in most RAG systems — a fine-tuned embedding model, two hours on one GPU, 400 MB, improves every query forever and halves the generator's prompt — sits undone because it isn't glamorous. Do the unglamorous one. Then, if you still need the LLM fine-tune, do that too, and you'll have a much better baseline to beat.

The dataset is the job. I know you know this and I know you'll still under-invest, because hyperparameters feel like engineering and data cleaning feels like chores. But 500 clean, deduplicated, entity-split examples beat 100k scraped ones, and the two bugs that quietly wreck fine-tune projects are both dataset bugs: near-duplicates that inflate your eval into a memorization score, and a random row-level split that leaks the same customer's phrasing into both sides. If your eval number looks amazing on the first try, be suspicious before you be proud.

What's actually worth caring about: the readiness contract, the finally that releases the slot, pinning, layer order, one base shared across tenants, up-fast/down-slow, and knowing your $/1M at your real utilization. Seven things. What's not worth losing sleep over: which vendor, the exact predict_concurrency to the integer, whether you picked Truss or Cog or BentoML. The platforms churn every two years and they're all the same five steps with different YAML. The concepts are permanent; learn them cold and you can re-skin them onto whatever is fashionable when you get there.

Build one thing end to end and put it on the internet. Not five half-projects — one. Fine-tune something small on data you care about, merge it, package it, deploy it with scale-to-zero so it costs you nothing at rest, put an eval gate in CI that can fail the build, and write a one-page memo about the trade you made. That artifact — the URL, the before/after table with your numbers, the CI gate, and especially the honest note about the run that didn't work — will carry an entire interview loop. I've hired people off exactly that. The 40 GPUs you didn't rent is a better story than the 40 you did.

The career framing. Every other phase makes you someone who understands models. This one makes you someone a company can put in front of production. That's the difference between being consulted and being trusted with the budget, the on-call rotation, and the decision. The people who own this layer are the ones who can say, at 2 a.m., without raising their voice: "it's deployment 47, adapter support-bot at 1.10.0 against base 8c22764a, cache hit rate dropped when we onboarded tenant 41, rolling the environment back to 46 now." Be that person. Now go make pytest green.