Brother Talk — PEFT: LoRA, QLoRA & Distillation

Off the record. The stuff I'd tell you over a beer, not in a design review.

This is the phase that turns "fine-tuning is for big labs" into "I'll have it by Friday." A few years ago, adapting a real model meant a cluster, a budget, and a prayer. LoRA quietly deleted that barrier — you can now fine-tune a serious model on one GPU in your IDE. When you internalize that, you stop being the person who asks for a custom model and become the person who makes one over a weekend. That shift is most of what "senior" means here.

The B-is-zero thing is a litmus test, and people fail it constantly. I have watched strong engineers explain LoRA fluently and then stall on "why is B initialized to zero?" The answer is one sentence — so the delta is zero at step 0 and the model starts as the pretrained model — but it reveals whether you actually understand that fine-tuning is continued training from a known-good point, not training from scratch. Learn that sentence cold. It's free points, and it signals you get the deeper idea.

QLoRA is the cheat code, and "the base is frozen" is the whole secret. Everyone hears "4-bit fine-tuning" and panics — surely 4 bits destroys the model? No. The base never trains, so its quantization error is fixed at load time and never compounds, and the little 16-bit adapter learns to work around it. Once that clicks, you'll confidently quantize bases that more nervous people insist on keeping in fp16, and you'll fit models on hardware they swear is too small. That confidence, backed by the frozen-base argument, wins arguments.

Distillation is the boring decision that saves the company money. A distilled small model is unglamorous in a demo and heroic on the P&L — because (the P00 lesson that never stops mattering) you pay inference cost on every request forever. The pull toward "just use the 70B, it's smarter" is real and emotional. The senior move is to distill the smallest model that clears the bar and serve that for the life of the product. Train yourself to feel good about shipping small; the spreadsheet will love you even when the demo crowd doesn't.

Don't fall in love with the hyperparameters. Juniors burn days sweeping r and α. The dirty truth: r=8, α=16, q-and-v is a fine default for most things, and the thing that actually decides whether your fine-tune works is the data. A few thousand clean, consistent instruction/response pairs beat a million scraped ones every time. If your fine-tune is bad, look at your dataset before you touch the rank. I promise that's where the bug is.

Fine-tuning is not always the answer, and saying so is a senior move. Here's the thing nobody says out loud in the LoRA hype: half the time the right call is a better prompt or RAG (P11), not a fine-tune. Fine-tuning teaches behavior, style, format, or latency — things you can't prompt your way to. It does not reliably teach new facts (that's RAG's job). When someone says "let's fine-tune it to know our docs," the move is to gently ask whether they actually want retrieval. Knowing when not to reach for the shiny tool is what separates an engineer from a hammer looking for nails.

Merge before you ship, and remember unmerged exists for a reason. For a single deployed task: merge_and_unload() and forget the adapter was ever there — zero latency, one matmul. But the unmerged form is a superpower people forget: one frozen base in GPU memory, dozens of tiny adapters swapped in per request. That's how you serve twenty specialized behaviors without twenty copies of a 13B model. Knowing when to merge vs keep separate is a real architecture decision, not a default.

What's actually worth caring about: the low-rank idea, why B is zero, why a frozen base makes 4-bit safe, and the distillation loss with its T and T². What's not worth losing sleep over: the exact NF4 quantile values, whether α should be r or 2r, the third decimal of your eval. Estimates and clean defaults ship; false precision just slows you down.

The career framing. This phase is where you become the person who can take any frontier model and make it yours — adapt it for pennies, shrink it for scale, fit it on the hardware you actually have. Combine it with the P00 cost math and you can walk into any "how do we customize this model?" meeting and own it: here's LoRA for the cheap tasks, QLoRA if it won't fit, distillation if we're serving it forever — with the param-count and memory numbers to back each. That's not "the person who uses models." That's the person who engineers them. Now go make pytest green.