Warmup Guide — PEFT: LoRA, QLoRA & Knowledge Distillation

Zero-to-senior primer for Phase 05. We start from the brutal arithmetic of a full fine-tune (recall Phase 00: weights + gradients + optimizer state ≈ 16 bytes/param) and build, from first principles, the three techniques that made adaptation cheap: LoRA (the low-rank update ΔW=(α/r)·A·B, the zero-initialized B, and the mergeable adapter), QLoRA (a 4-bit NF4 frozen base, double quantization, paged optimizers — a 65B fine-tune on one GPU), the other PEFT methods (adapters, prefix/prompt tuning, IA³) and when each, and knowledge distillation (training a small student on a big teacher's softened logits). Every section gives you the mechanism under the hood, not the framework call.

Table of Contents

• Chapter 1: Why PEFT — The Cost of a Full Fine-Tune
• Chapter 2: The LoRA Hypothesis — Updates Are Low-Rank
• Chapter 3: The LoRA Mechanism — A, B, r, α, and Merging
• Chapter 4: Why B Is Zero-Initialized (The Soul of LoRA)
• Chapter 5: Which Layers to Adapt, and Choosing r and α
• Chapter 6: QLoRA — 4-Bit NF4, Double Quantization, Paged Optimizers
• Chapter 7: The Rest of the PEFT Zoo — Adapters, Prefix/Prompt, IA³
• Chapter 8: Knowledge Distillation — Softened Logits & Dark Knowledge
• Chapter 9: The Practical Recipe — SFT → Data → Eval
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why PEFT — The Cost of a Full Fine-Tune

From zero. "Fine-tuning" means continuing to train a pretrained model on your data so its weights shift toward your task. The obvious way is to update every weight — full fine-tuning. The problem is not the idea; it is the memory bill, and that bill is arithmetic you already did in Phase 00.

The memory of training (recall P00, Chapter 3). To train a parameter you hold, per parameter, much more than the parameter itself:

…plus activations (which scale with batch × sequence length). So a 7B full fine-tune is 7e9 × 16 = 112 GB before activations — already over one 80 GB GPU — and a 65B model is ~1 TB, hopelessly out of reach without a cluster.

The insight that unlocks PEFT. Almost all of those 16 bytes are spent on the optimizer state and gradients of weights you barely change. What if you froze the pretrained weights entirely (no gradient, no optimizer state, no master copy — 0 extra bytes for them) and only trained a small set of new parameters? Then the expensive ~14 bytes/param of training overhead applies only to that small set. That is the entire premise of Parameter-Efficient Fine-Tuning (PEFT): keep the giant frozen, learn something tiny.

Why this matters in production. PEFT is not a research curiosity — it is how teams ship. One frozen base + a few-MB adapter per task means you can serve dozens of specialized behaviors from one set of weights, version adapters like code, and fine-tune on a single GPU. And recall the asymmetry from P00: training cost is paid once, but inference cost is paid on every request forever — so a technique like LoRA that adds zero inference cost (after merging) is doubly attractive.

Common misconception. "PEFT trades quality for cheapness." On most adaptation tasks, a well-tuned LoRA matches full fine-tuning to within noise — because (next chapter) the update you needed was low-rank all along. You are not approximating a full fine-tune; you are exploiting its structure.

Chapter 2: The LoRA Hypothesis — Updates Are Low-Rank

The claim. When you fine-tune a pretrained model, the change to each weight matrix — ΔW = W_finetuned − W_pretrained — has a low intrinsic rank. Empirically, ΔW can be approximated well by a matrix of rank r far smaller than the matrix's dimensions. Hu et al. (2021) measured this directly and found r as small as 1–8 often suffices.

What "rank" means here, from zero. The rank of a matrix is the number of independent directions it actually uses. A d×k matrix can have rank up to min(d,k), but a low-rank matrix — say rank r — can be written as the product of a d×r and an r×k matrix:

$$ \underbrace{\Delta W}{d\times k} = \underbrace{A}{d\times r};\underbrace{B}_{r\times k}, \qquad r \ll \min(d,k). $$

The full d×k matrix has d·k numbers; the factored form has only r·(d+k). For a 4096×4096 matrix at r=8: 16.8M vs 65.5K — a 256× reduction.

Why fine-tuning updates are low-rank (the intuition). Pretraining already taught the model general language. Your fine-tune is a comparatively small, directional nudge — "prefer this style," "answer in this format," "know this domain." A nudge does not need to rewrite all min(d,k) directions of a weight matrix; it needs to push a handful. The data confirms the intuition: the useful signal in ΔW concentrates in a few singular directions, so a rank-r factorization captures it.

   full fine-tune learns ΔW directly (d·k params, all of it trainable)
        ┌──────────────┐
   W +  │     ΔW        │          rank up to min(d,k)
        └──────────────┘

   LoRA learns ΔW = A·B  (only r·(d+k) params)
        ┌──┐
   W +  │A │ · [  B  ]              rank exactly r ≪ min(d,k)
        │  │   (r×k)
        └──┘
        (d×r)

Common misconception. "Low-rank means low-quality." Rank limits the number of directions the update can move, not the magnitude of the move (that is what α controls — Chapter 3). For an adaptation that genuinely needs few directions, a rank-r update is not lossy; it is exactly the right capacity, and it also regularizes.

Chapter 3: The LoRA Mechanism — A, B, r, α, and Merging

The setup. Take any frozen linear layer with weight W of shape (d×k) that maps an input x (a row vector of width d) to x·W (width k). LoRA leaves W frozen and adds a parallel low-rank path:

$$ h = x W + \frac{\alpha}{r}, x A B, \qquad A\in\mathbb{R}^{d\times r},; B\in\mathbb{R}^{r\times k}. $$

(Orientation note — this lab fixes W as (d×k), inputs as (n×d). The LoRA paper writes W as (k×d) acting on column vectors; the algebra is identical, just transposed. Pick one and be consistent — the lab is consistent.)

The roles of the two new matrices.

• A (d×r) projects the input down into an r-dimensional bottleneck.
• B (r×k) projects that bottleneck up to the output width.
• Their product A·B is (d×k) — the same shape as W — but constrained to rank ≤ r.

The role of r (rank = capacity). r is the width of the bottleneck — how many independent directions the update may use. Bigger r = more capacity (and more params, r·(d+k)). Typical values: 4, 8, 16, 64. Past a point, more r stops helping because the true update's rank is the ceiling.

The role of α (scaling = magnitude). The update is scaled by α/r before it is added. Why divide by r? So the effective magnitude of the update is decoupled from the rank: if you raise r to get more capacity, the 1/r keeps the output scale roughly constant, so you do not have to re-tune the learning rate. In practice people fix α (often α = 2r or α = r) and tune r. Doubling α doubles the delta linearly — the lab tests this exactly.

$$ \Delta W = \frac{\alpha}{r} A B \quad\Rightarrow\quad \Delta W \big|{2\alpha} = 2,\Delta W \big|{\alpha}. $$

The cheap forward path. Notice the lab computes (x·A)·B, not x·(A·B). Both are equal, but x·A is (n×r) — a skinny intermediate — so the adapter's extra compute is O(n·d·r + n·r·k) instead of materializing a (d×k) delta. During training this skinny path is what makes LoRA cheap to run.

Merging — the zero-latency trick. Because the adapter is linear and parallel, you can fold it into the base once training is done:

$$ W_{\text{merged}} = W + \frac{\alpha}{r} A B. $$

Now x · W_merged is a single matmul — identical to the LoRA forward, with zero extra inference cost. This is a defining advantage over adapters (Chapter 7), which insert new layers you cannot fold away. (You can also keep adapters unmerged to hot-swap many tasks against one base — you trade a little latency for flexibility.)

Common misconception. "LoRA is slower at inference." Only if you leave it unmerged. Merged, it is exactly as fast as the original model — that is the whole point.

Chapter 4: Why B Is Zero-Initialized (The Soul of LoRA)

This is the single detail interviewers love and beginners miss, so it gets its own chapter.

The mechanism. At the start of training, LoRA initializes:

• A with small random Gaussian values, and
• B with exactly zeros.

Therefore the initial update is

$$ \Delta W \big|_{t=0} = \frac{\alpha}{r}, A \cdot \mathbf{0} = \mathbf{0}. $$

The product of anything with the zero matrix is the zero matrix. So at step 0:

$$ h = x W + \frac{\alpha}{r} x A B = x W + 0 = x W. $$

The adapted model is bit-for-bit the pretrained model at initialization. The lab's soul test asserts exactly this: delta() is all zeros and forward(x) == x · W_base with no floating-point drift.

Why this is the right design, not a convenience. Fine-tuning is continued training from a known-good model. If your adapter started by injecting random noise into W (which a random B would do), step 0 would corrupt a model that took millions of dollars to pretrain — you would spend the first part of training just climbing back to where you started, and you might never recover. Zero-init means training begins at the pretrained optimum and can only ever add skill, gradient step by gradient step.

Why not zero-init both A and B? If both were zero, the gradient of the loss with respect to both would also be zero (the product is zero and so are its partial derivatives in this symmetric configuration), and the adapter would be stuck — it could never start learning. You need exactly one side non-zero to break the symmetry and let gradients flow: random A, zero B. (PEFT does exactly this.)

  step 0:   A = random,  B = 0   →   ΔW = (α/r)·A·0 = 0   →   model == pretrained
  step 1+:  gradients flow into B (A non-zero breaks symmetry) → ΔW grows from 0

Common misconception. "It doesn't matter which one is zero." It must be exactly one. Zero A + random B also gives ΔW=0 at start, but PEFT conventionally zeros B; the non-negotiable part is that one is zero (so you start at the pretrained model) and the other is non-zero (so learning can begin).

Chapter 5: Which Layers to Adapt, and Choosing r and α

Which weight matrices? A transformer block has attention projections (W_q, W_k, W_v, W_o) and a two-layer MLP (W_up/W_gate, W_down). You do not have to LoRA all of them. The original paper found that adapting just the attention query and value projections (q, v) captures most of the benefit at minimal cost. Modern practice (and QLoRA) often adapts all linear layers including the MLP — the MLP holds ~2/3 of the parameters (P00, Chapter 2), so for harder adaptations it carries real signal. The tradeoff:

Choosing r. Start at r=8 or 16. Increase if the model underfits your task (loss plateaus high); rarely helps past r=64. The r·(d+k) cost is linear in r, so doubling r doubles the trainable params and optimizer memory.

Choosing α. A common heuristic is α = 2r (so the scaling α/r = 2) or α = r (scaling 1). Because α and the learning rate both scale the effective update, you can fix one and tune the other; do not tune both at once or you will chase your tail.

LoRA dropout. A small dropout (e.g. 0.05) on the LoRA path regularizes; it is a standard knob in LoraConfig.

Common misconception. "More targets and higher r is always better." It costs more memory and can overfit a small dataset. The senior move is to start small (q,v, r=8), measure on a held-out eval, and only grow capacity if the eval says you are underfitting.

Chapter 6: QLoRA — 4-Bit NF4, Double Quantization, Paged Optimizers

The problem QLoRA solves. LoRA shrinks the trainable memory, but you still have to hold the frozen base in GPU memory to compute the forward pass. For a 65B model that is 130 GB in fp16 — two or more big GPUs just to hold the thing you are not even training. QLoRA (Dettmers et al., 2023) attacks that base memory with three tricks, and the result is the headline: fine-tune a 65B model on a single 48 GB GPU.

Trick 1 — NF4 4-bit quantization (blockwise, with an absmax scale). The frozen base is stored in 4 bits per weight. Naively, 4 bits gives only 16 distinct values, so you must choose them and scale them well:

• Blockwise scaling. Split each weight tensor into small contiguous blocks (e.g. 64). Each block gets its own scale = the block's maximum absolute value (absmax). Every weight is divided by its block's scale to land in [-1, 1], then snapped to a 16-level grid; the index (4 bits) is stored. One outlier only blows up its block's scale, not the whole tensor — far better than a single tensor-wide scale.
• NF4 (NormalFloat4). The 16 grid levels are not uniform — they are the quantiles of a unit normal distribution, because pretrained weights are roughly normally distributed, so quantile-spaced levels are information-theoretically optimal for them.

The lab builds a simplified version: a per-block absmax scale and a uniform symmetric grid. The key property to internalize is the error bound: reconstruction error per weight is at most scale × (grid step)/2, so a finer grid (more levels) → strictly smaller error. The lab tests this monotonicity directly.

$$ \hat w = \text{grid}[,\text{code},]\times \text{scale}{\text{block}},\qquad |w-\hat w|\le \tfrac{\text{scale}}{2}\cdot\frac{2}{n{\text{levels}}-1}. $$

Trick 2 — Double quantization. Each block needs its own scale (a 32-bit float). For a 65B model with 64-weight blocks that is ~1B / 64 extra floats — not free. So QLoRA quantizes the scales themselves (8-bit, with their own meta-scale), saving ~0.4 bits per parameter. Quantizing the quantization constants is exactly as recursive as it sounds.

Trick 3 — Paged optimizers. Even with a frozen 4-bit base, optimizer state for the LoRA adapters can spike during gradient checkpointing. QLoRA uses NVIDIA unified memory to page optimizer state to CPU RAM when the GPU is tight, like OS virtual memory, so a transient spike does not OOM the run. (This previews the OS-style memory management you'll see in PagedAttention, P09.)

Why 4-bit is safe here but scary elsewhere. The base is frozen. Its quantization error is fixed at load time and never compounds through training — only the high-precision (16-bit) LoRA adapters learn, and they can compensate for the base's quantization error. That decoupling of storage precision (4-bit, frozen) from training precision (16-bit, the adapter) is the conceptual heart of QLoRA.

Common misconception. "QLoRA trains in 4-bit." No — it stores the base in 4-bit and dequantizes on the fly (per block) for each matmul; the adapters and all gradients are in 16-bit. You never compute a gradient through a 4-bit weight.

Chapter 7: The Rest of the PEFT Zoo — Adapters, Prefix/Prompt, IA³

LoRA is the default, but a senior knows the alternatives and when each wins.

Adapter layers (Houlsby et al., 2019). Insert a small bottleneck MLP (down-project → nonlinearity → up-project) inside each transformer block and train only it. It works, but it adds new sequential layers, so it has an inference-latency cost you cannot merge away — the very thing LoRA fixed. This is the historical ancestor of PEFT.

Prefix / prompt tuning (Li & Liang; Lester et al.). Don't touch any weights — instead prepend a handful of trainable "virtual token" vectors to the input (prompt tuning) or to every layer's attention keys/values (prefix tuning). You train only those vectors (kilobytes). Extremely cheap and fully swappable, but generally lower-quality than LoRA on hard tasks and it eats context-window budget. Shines for tiny per-task customizations on a shared frozen base.

IA³ (Liu et al., 2022). Even leaner: learn a few per-feature scaling vectors that rescale the keys, values, and MLP activations element-wise (activation ⊙ learned_vector). Tiny parameter count, and like LoRA the scalings can be merged into the surrounding weights for zero inference cost. A good fit when you want the absolute minimum trainable params.

Common misconception. "They're interchangeable." They differ on mergeability (LoRA, IA³ yes; adapters, prefix no), on parameter budget (IA³ < prompt < LoRA < adapters, roughly), and on quality ceiling (LoRA generally highest). The choice is an engineering decision, not a coin flip.

Chapter 8: Knowledge Distillation — Softened Logits & Dark Knowledge

The other lever. LoRA/QLoRA adapt a frozen model cheaply but leave its size — and thus its forever-inference-cost — unchanged. Distillation makes a permanently smaller model. You train a small student to mimic a large teacher, and the student often matches the teacher's task quality at a fraction of the inference cost.

Why not just train the student on the labels? Hard labels are a single bit of information per example ("the answer is class 3"). The teacher's full output distribution carries far more: it says class 3 is most likely, but class 7 is plausible and class 2 is absurd. Those relative probabilities of the wrong answers — Hinton's "dark knowledge" — encode how the teacher generalizes, and the student learns enormously from them.

The temperature trick. A confident teacher's softmax is nearly one-hot ([0.99, 0.005, …]), which hides the dark knowledge. Dividing the logits by a temperature T > 1 before the softmax softens the distribution, raising the small probabilities so the student can see and learn the structure:

$$ \text{softmax}_T(z)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}. $$

As T → ∞ it approaches uniform (maximum entropy); at T = 1 it is the ordinary softmax. The lab tests that higher T increases the entropy of the distribution — the precise statement of "softens."

The loss. You blend a soft term (match the teacher's softened distribution) with a hard term (still get the true label right):

$$ \mathcal{L} = T^2\cdot D_{\mathrm{KL}}!\big(\text{softmax}_T(z^{teacher}),\big|,\text{softmax}_T(z^{student})\big) ;+; \alpha\cdot \mathrm{CE}\big(\text{softmax}(z^{student}),,y\big). $$

• The KL term pulls the student's softened distribution toward the teacher's.
• The T² factor. Dividing logits by T shrinks the softmax gradients by ~1/T². Multiplying the soft loss by T² rescales those gradients back up so the soft and hard terms contribute on a comparable scale regardless of T. (Drop the T² and the soft term silently vanishes as you raise T.)
• The hard CE term keeps the student anchored to the ground truth at T = 1.

The vanishing-KL invariant. If the student's logits ever equal the teacher's, the two softened distributions are identical, so D_KL = 0 and the loss collapses to exactly α · CE. The lab tests this — it is the cleanest check that your KL is stable and your blend is correct.

Numerical stability (the part interviewers probe). Both the softmax and the KL must be computed carefully. softmax_T subtracts the max logit before exponentiating (exp(z - max z)) so large logits never overflow. kl_divergence works in log-space and skips p_i = 0 terms (0·log 0 = 0 by convention) while raising on the genuinely undefined q_i = 0, p_i > 0 (infinite divergence). These are not optional polish — they are the difference between a loss that trains and one that returns nan.

Three flavors of distillation.

• Response (logit) distillation — match the output distribution (what we built; Hinton).
• Feature distillation — also match intermediate hidden states (e.g. an MSE between a student and teacher layer; FitNets, TinyBERT). Helps when the logit signal alone is weak.
• Relation distillation — match relationships between examples (e.g. pairwise distances in feature space), so the student preserves the teacher's geometry.

Common misconception. "Distillation just copies the teacher's answers." It copies the teacher's function — the full conditional distribution — which is why a distilled model generalizes better than one trained on the same hard labels alone.

Chapter 9: The Practical Recipe — SFT → Data → Eval

Where PEFT fits in the lifecycle. The usual adaptation pipeline is Supervised Fine-Tuning (SFT): take instruction/response pairs and train the model to produce the response. LoRA/QLoRA is the how of SFT (which params you train); it is also the first stage before alignment (DPO/RLHF, P07), which itself is commonly done with LoRA on top.

The recipe, in order.

1. Data first. A few thousand high-quality instruction/response pairs beat a million noisy ones. Curate, dedupe, and format consistently — bad data is the #1 cause of a failed fine-tune, not the wrong r.
2. Pick the lever. Adapt-and-serve → LoRA (or QLoRA if the base won't fit). Want a permanently smaller model → distillation. Memory-bound training → QLoRA.
3. Configure conservatively. q,v or all-linear targets, r=8–16, α=2r, small LR (e.g. 1e-4 to 2e-4 for LoRA — higher than full FT because you train few params), 1–3 epochs. Over-training a LoRA on a small set overfits fast.
4. Merge or keep. Merge for a single deployed model with zero overhead; keep separate for an adapter zoo (many tasks, one base).
5. Eval honestly. Hold out a real test set and check for regressions on general capability (a fine-tune can make the model worse at everything else — "catastrophic forgetting"). Eval is the deliverable, not the loss curve.

Common misconception. "Fine-tuning is the fix for everything." Often RAG (P11) or a better prompt is cheaper and good enough. Fine-tune when you need a behavior or style or latency that prompting can't give — and prove it beat the cheaper option on the eval.

Lab Walkthrough Guidance

The lab (lab-01-lora-qlora-distill) turns these chapters into code. Suggested order (matches the file):

1. Matrix helpers (matmul, mat_add, transpose) — the foundation. The only trick is raising ValueError on shape mismatch; everything else builds on these.
2. LoRALinear — Chapters 3–4. __init__ must Gaussian-init A and zero-init B (the soul). delta = (α/r)·(A·B); merge = base + delta; forward uses the cheap (x·A)·B path. The tests test_lora_delta_is_zero_at_init and test_lora_forward_equals_base_at_init are the soul of the lab — get those green first.
3. Param counts (lora_param_count, full_param_count) — Chapter 2. r·(d+k) vs d·k; validate positivity.
4. Blockwise quantization (quantize_blockwise, dequantize_blockwise) — Chapter 6. Per-block absmax scale, snap to the symmetric grid, store the index. The monotonicity test (test_quant_finer_grid_has_smaller_error) is the lesson; the bound test proves it is principled.
5. Distillation (softmax_T, kl_divergence, cross_entropy, distillation_loss) — Chapter 8. Max-subtract the softmax, log-space the KL, blend T²·KL + α·CE. The "student==teacher ⇒ loss == α·CE" test is the cleanest invariant.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can derive ΔW=(α/r)·A·B, state the orientation (A:d×r, B:r×k), and explain why (x·A)·B is the cheap path.
• You can explain in one sentence why B is zero-initialized and why exactly one of A/B (not both) must be zero.
• You can compute the LoRA trainable fraction for (d=4096, k=4096, r=8) ≈ 0.39% and use it to justify "fine-tune on one GPU."
• You can state QLoRA's three tricks and explain why a frozen base makes 4-bit safe.
• You can write the distillation loss from memory, explain T and T², and explain why distillation_loss(t, t, …) reduces to α·CE.

Interview Q&A

• "Why is LoRA's B zero-initialized?" — So ΔW = (α/r)·A·0 = 0 at step 0; the adapted model is identical to the pretrained model and training starts from the known-good optimum, only ever adding skill. Exactly one of A/B is zero (to start at the pretrained model) and the other non-zero (to break symmetry so gradients flow).
• "What do r and α control?" — r is the rank/capacity (how many directions the update may use); α/r is the magnitude scaling, decoupled from r so you don't re-tune the LR when you change rank. Doubling α doubles the delta linearly.
• "How does LoRA give zero inference latency?" — The adapter is linear and parallel, so you merge() it: W_merged = W + (α/r)AB. The served model is one matmul again — no extra cost. (Unmerged, you keep it for hot-swapping adapters at a small latency cost.)
• "How does QLoRA fit a 65B fine-tune on one GPU?" — Store the frozen base in 4-bit NF4 (blockwise absmax scale, normal-quantile grid), double-quantize the scales, page the optimizer to CPU, and train only the small 16-bit LoRA adapters. The base never trains, so its 4-bit error is fixed and never compounds.
• "Does QLoRA train in 4-bit?" — No. It stores the base in 4-bit and dequantizes per block on the fly for each matmul; all gradients and adapters are 16-bit.
• "Why blockwise absmax instead of one tensor-wide scale?" — One outlier weight would inflate a global scale and crush the precision of every other weight; a per-block scale confines the damage to its block.
• "Walk me through the distillation loss." — T²·KL(softmax_T(teacher) ‖ softmax_T(student)) + α·CE(student, target). The soft term transfers dark knowledge (relative wrong-answer probabilities), T softens to expose it, T² rescales the soft gradient, and the hard CE anchors the student to the truth.
• "What is dark knowledge?" — The information in a teacher's non-target probabilities — that some wrong answers are plausible and others absurd — which encodes how it generalizes and is invisible in hard labels.
• "LoRA vs adapters vs prefix tuning?" — LoRA/IA³ are mergeable (zero inference cost); adapters add unmergeable layers (latency); prefix/prompt tuning trains virtual tokens (ultra-cheap, swappable, lower quality ceiling, eats context). LoRA is the default.
• "When distill vs LoRA?" — LoRA adapts a frozen model cheaply but keeps its size and forever-inference-cost; distillation shrinks N permanently. One huge-scale task → distill; many cheap swappable tasks → LoRA zoo; can't fit the base → QLoRA.
• "Why must softmax/KL be numerically stable?" — Max-subtraction stops large logits overflowing exp; log-space KL plus the 0·log0=0 convention avoids nan/inf. Without them the loss returns nan and the run silently dies.
• "What's the #1 cause of a failed fine-tune?" — Data quality, not hyperparameters. A few thousand clean instruction/response pairs beat a million noisy ones; eval honestly for task gain and general-capability regressions (catastrophic forgetting).

References

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021) — the low-rank hypothesis, the A·B factorization, r/α, and the q,v finding.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023) — NF4, double quantization, paged optimizers; the 65B-on-one-GPU result.
• Dettmers et al. / bitsandbytes docs — the 8-bit and 4-bit (NF4) blockwise quantization implementation behind QLoRA.
• Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (2015) — softened logits, the T² factor, "dark knowledge."
• Sanh et al., DistilBERT (2019) — a canonical response-distillation recipe at scale.
• Houlsby et al., Parameter-Efficient Transfer Learning for NLP (2019) — adapter layers.
• Li & Liang, Prefix-Tuning (2021); Lester et al., The Power of Scale for Parameter- Efficient Prompt Tuning (2021).
• Liu et al., IA³ / (Few-Shot PEFT) — "Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning" (2022).
• Hugging Face PEFT library docs — LoraConfig, get_peft_model, print_trainable_parameters, merge_and_unload; the production home of all of the above.