« Phase 30 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 30 Warmup — Hugging Face: Transformers, Pipelines, the Hub & PEFT

Who this is for: someone who has built agent infrastructure from scratch (Phases 00–17) and framework internals (Phases 18–29) and now needs fluency in the ecosystem every open-model conversation runs through: Hugging Face. By the end you will be able to explain, from first principles, how pipeline() composes a tokenizer, a model, and task post-processing; how the Auto* classes load any architecture from a config file; how BPE/WordPiece/Unigram tokenizers actually work; what every generate() knob does to the logits; how the Hub versions and gates model artifacts and why safetensors exists; how datasets handles corpora bigger than RAM; how Trainer and accelerate run fine-tuning; the LoRA/QLoRA math and economics; the bitsandbytes/GPTQ/AWQ quantization tradeoff; and how TGI serves all of it with continuous batching. One boundary: transformer internals (attention math, backprop, building GPT from scratch) belong to the sibling Senior AI Engineer track — this phase is the practitioner's ecosystem, workflow, and APIs. No GPU, no network — the labs are pure mechanism.

Table of Contents

  1. What Hugging Face actually is, and why it won
  2. The pipeline abstraction
  3. AutoConfig, AutoTokenizer, AutoModel: loading by convention
  4. from_pretrained under the hood: revisions, cache, safetensors, trust_remote_code
  5. Tokenization algorithms: BPE, WordPiece, Unigram
  6. Tokenizer mechanics: special tokens, padding, truncation, attention masks, offsets
  7. Text generation: decoding strategies and the KV cache
  8. The Hub: models, datasets, Spaces, cards, gating, versioning
  9. The datasets library: Arrow, memory-mapping, streaming
  10. Fine-tuning: Trainer, TrainingArguments, accelerate
  11. PEFT: LoRA, QLoRA, adapters, prompt tuning
  12. Quantization: bitsandbytes, GPTQ, AWQ
  13. Serving: TGI, Inference Endpoints, continuous batching
  14. Licensing, gated access, and the model supply chain
  15. Common misconceptions
  16. Lab walkthrough
  17. Success criteria
  18. Interview Q&A
  19. References

1. What Hugging Face actually is, and why it won

Hugging Face is not one library — it is an ecosystem of composable libraries around a central artifact registry. The pieces you must be able to name and place:

  • transformers — the model library: load, run, and train transformer models behind a uniform API (AutoModel, pipeline, generate, Trainer). It supports PyTorch first-class (TensorFlow/JAX support exists but has been progressively de-emphasized).
  • tokenizers — the fast, Rust-backed tokenization library underneath transformers' "fast" tokenizers: trains and runs BPE/WordPiece/Unigram at production speed and provides offset mappings.
  • datasets — Arrow-backed dataset loading and processing: memory-mapped (bigger than RAM), streamable, with cached map/filter transforms.
  • huggingface_hub — the Python client for the Hub itself: download/upload files, manage repos, revisions, and access tokens. from_pretrained calls into it.
  • accelerate — the device-and-distribution layer: one code path that runs on CPU, one GPU, multiple GPUs, or multiple nodes; also "big model inference" (device_map="auto").
  • peft — parameter-efficient fine-tuning: LoRA and friends as wrappers around any transformers model.
  • The Hub (hub.huggingface.co is the site; the thing is a git-backed artifact store) — models, datasets, and Spaces (hosted demo apps), each a versioned repo with a card.
  • Text Generation Inference (TGI) — the production LLM serving server, and Inference Endpoints, the managed deployment product around Hub models.

Why did this stack become the default? Three compounding reasons. First, a packaging convention: a model on the Hub is a directory with config.json (architecture + hyperparameters), weight files (safetensors), and tokenizer files — enough for one line of code to reconstruct the whole thing. Standardized packaging did for models what container images did for services. Second, network effects: when BERT landed in 2018, the (then) pytorch-pretrained-bert library was how most people could actually use it; every subsequent architecture shipped a transformers port (often day-one, by the model's own authors), so the library became the reference implementation registry of the field — over a million public model repos now sit behind the same three-line loading idiom. Third, the pieces compose without requiring each other: you can use datasets with your own training loop, peft over raw PyTorch, or the Hub as a pure artifact store. Like every good platform in this track, it is a set of separable primitives, not a monolith.

The engineering consequence: "we use Hugging Face" does not mean "we depend on a vendor" the way "we use OpenAI" does. Weights are local files; the license question (§14) is between you and the model's publisher; the Hub is replaceable with any file store (and mirrorable on-prem). What you actually adopt is the convention — and conventions are cheap to keep and expensive to leave only when they stop being industry-wide, which this one shows no sign of.

2. The pipeline abstraction

pipeline() is the front door of transformers, and it is worth understanding precisely because it looks like magic and is not:

from transformers import pipeline
clf = pipeline("sentiment-analysis")             # or: pipeline("text-generation", model="gpt2")
clf("I love this")                               # [{'label': 'POSITIVE', 'score': 0.9998}]

Every pipeline is the same three-stage assembly line:

  1. preprocess — the tokenizer turns raw input into model inputs (input_ids, attention_mask, task extras like question+context pairing for QA), applying padding and truncation policy.
  2. _forward — the model runs, producing raw outputs (logits, hidden states), wrapped in device placement and torch.no_grad().
  3. postprocess — the task-specific step, and the one that actually differs per task:
    • text-classification: softmax the logits, map through model.config.id2label, return {label, score} (with top_k/return_all_scores variants);
    • token-classification (NER): argmax per token, then aggregate subtoken labels into entity spans using the offset mapping (aggregation_strategy="simple"/"first"/"average"/ "max");
    • text-generation: call model.generate() and decode;
    • question-answering: find the argmax start/end logits over context tokens and slice the answer span out of the original string;
    • fill-mask: softmax the vocab logits at the [MASK] position, return top-k tokens;
    • zero-shot-classification: run an NLI model over "premise = your text, hypothesis = 'This example is {label}'" per candidate label and rank entailment scores — a "new task" created entirely in post-processing.

pipeline(task) is a factory: it resolves the task string (aliases included — "sentiment-analysis" is text-classification, "ner" is token-classification), selects the Pipeline subclass, loads a default model for the task if you didn't name one (fine for demos; never in production — pin your model and revision), and returns a callable that also accepts lists (batching) and generators/datasets (streaming iteration).

Two production notes. Batching in pipelines is not automatically faster — for generation tasks especially, batch_size interacts with padding waste and memory; HF's own docs tell you to benchmark. The default-model trap: pipeline("sentiment-analysis") with no model downloads whatever default the task currently maps to — an unpinned dependency on someone else's choice. Lab 01 builds this entire mechanism — the three stages, three tasks' post-processors, the factory with aliases, batching semantics, truncation — in ~200 lines, so none of it stays magic.

3. AutoConfig, AutoTokenizer, AutoModel: loading by convention

Below pipeline() sits the layer you actually use in serious code — the Auto* classes:

from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3",
                                             torch_dtype="auto", device_map="auto")

The mechanism: every Hub model repo carries a config.json whose model_type field (e.g. "llama", "mistral", "bert") keys a registry inside transformers. AutoConfig reads that field and instantiates the right config class; AutoModel* uses the config to instantiate the right architecture class and then loads weights into it; AutoTokenizer does the same via tokenizer_config.json. You write zero architecture-specific code — swap the repo id and the same three lines load a different model family. That dispatch-on-config trick is the whole "Auto" idea, and it's why libraries and serving stacks can accept "any HF model id" as configuration.

The task heads matter: AutoModel gives you the bare transformer (hidden states out — useful for embeddings); AutoModelForCausalLM adds the language-model head (generate() lives here); AutoModelForSequenceClassification adds a classification head (randomly initialized — with a warning — if the checkpoint doesn't include one, which is exactly what you want before fine-tuning); likewise ForTokenClassification, ForQuestionAnswering, ForSeq2SeqLM (T5/BART encoder-decoder). Loading the same checkpoint under different heads is the standard fine-tuning setup move, and the "some weights were newly initialized" warning is a signal to read, not noise: it tells you which parameters have no pretrained values.

The counterpart methods complete the loop: save_pretrained(dir) writes config + weights + tokenizer files to a directory from_pretrained can read back; push_to_hub(repo_id) does the same into a Hub repo. Model, tokenizer, config, and even GenerationConfig all support these three verbs — the ecosystem's uniform artifact contract.

4. from_pretrained under the hood: revisions, cache, safetensors, trust_remote_code

from_pretrained("org/name") looks atomic; a principal engineer should know its four moving parts.

Resolution and revisions. The id resolves against the Hub as a git repo; revision= accepts a branch (default "main"), a tag, or a commit sha. Unpinned, you get whatever main points at today — meaning a model author's force-push or update can change your production behavior without any change on your side. The discipline is the same as container digests: pin a revision (ideally a full commit sha) in anything that ships. Lab 03's hub miniature makes pinning mechanical: tag v1.0, push more commits, prove the pin still resolves to the old artifact.

The cache. Downloads land in ~/.cache/huggingface/hub (override with HF_HOME), organized per repo with deduplicated blobs and per-commit snapshot directories, so two revisions sharing files don't store them twice, and repeated loads hit disk, not network. HF_HUB_OFFLINE=1 makes the cache the only source — the standard trick for airgapped or reproducible CI.

safetensors vs pickle. PyTorch's classic .bin checkpoint is a pickle, and unpickling executes arbitrary code by design — a malicious checkpoint is a remote-code-execution payload. safetensors is the fix: a format that is pure tensor data plus a JSON header — nothing executable, and as a bonus it memory-maps for fast, zero-copy, lazy loading (which is also why sharded multi-file checkpoints load efficiently). transformers prefers safetensors when a repo has them; treat "safetensors only" as a supply-chain policy, not a preference. The Hub runs pickle-scanning on uploads, but scanning is a mitigation, not a guarantee.

trust_remote_code. Some repos ship custom modeling code — architectures not (yet) in transformers — and loading them requires executing Python from the repo. That is what trust_remote_code=True opts into, and it should be treated exactly like curl | bash: fine for a repo you've reviewed and pinned to a specific revision, indefensible as a default. Lab 03 models the whole gate: safe formats load freely; anything else is refused unless the flag is explicit.

5. Tokenization algorithms: BPE, WordPiece, Unigram

A tokenizer is a trained artifact, not a design choice made by hand: its vocabulary is statistics extracted from a corpus, and it ships with the model because the model's embedding matrix is indexed by exactly that vocabulary. Same weights + different tokenizer = gibberish. Three algorithm families cover essentially every modern model:

Byte-Pair Encoding (BPE) — GPT-2 and most decoder-only LLM families. Training: start with a base alphabet (characters, or — in byte-level BPE — the 256 bytes, which eliminates unknown tokens entirely since any string is bytes); count every adjacent symbol pair across the corpus; merge the most frequent pair into a new symbol; repeat until the vocabulary budget is reached. The learned artifact is the ordered merge list. Encoding replays those merges, in training order, on the new text — frequent words end up as single tokens, rare words split into frequent pieces. Lab 02 makes you implement exactly this (with a deterministic lexicographic tie-break, so tests can assert the merge table).

WordPiece — BERT and its family. Same merge-loop shape as BPE, but the pair to merge is chosen by likelihood gain, not raw frequency: merge the pair maximizing \( \frac{count(ab)}{count(a),count(b)} \) — "merge what co-occurs more than its parts predict." Continuation pieces are marked with the ## prefix (playing → play, ##ing), and encoding is greedy longest-match against the vocabulary rather than a merge replay.

Unigram — the SentencePiece-trained tokenizer used by T5 and others. Philosophically inverted: start from a large candidate vocabulary and iteratively prune the pieces whose removal least hurts corpus likelihood under a unigram language model; encoding picks the segmentation with the highest product of piece probabilities (Viterbi). SentencePiece itself is a library, not an algorithm — it can train Unigram or BPE, and its distinguishing habit is treating input as a raw character stream with spaces encoded as ▁, making tokenization fully reversible and language-agnostic (no whitespace pre-splitting assumption — crucial for languages that don't use spaces). Llama-family tokenizers are SentencePiece-flavored BPE.

Why you care in production: token count is cost, latency, and context budget. The same 100 words can be 130 tokens on one tokenizer and 210 on another (multilingual and code-heavy text diverges hardest). And fertility (tokens per word) on your domain's text is a real model-selection metric: a model whose tokenizer fragments your domain vocabulary burns context and quality.

6. Tokenizer mechanics: special tokens, padding, truncation, attention masks, offsets

Around the algorithm sits the mechanical surface you touch daily — and where the classic silent bugs live.

Special tokens. Reserved vocabulary entries with structural meaning: [CLS]/[SEP] (BERT's sequence start/separator), BOS/EOS (sequence boundaries for causal LMs), PAD (filler), UNK (unknown), MASK (masked-LM training). Two production gotchas: GPT-2-family models have no pad token (the standard fix is tokenizer.pad_token = tokenizer.eos_token, and forgetting it crashes batched generation); and chat models overlay a chat template — special role-formatting (tokenizer.apply_chat_template) that turns a messages list into exactly the token pattern the model was instruction-tuned on. Hand-formatting chat prompts instead of using the template is one of the most common "the model got dumber" bugs in the wild.

Padding and the attention mask. Batching requires rectangular tensors; shorter sequences are padded to the longest (or to max_length). The attention mask — 1 for real tokens, 0 for pads — is how the model knows to ignore filler. Drop it and pad embeddings leak into attention, degrading outputs silently. A subtler one: decoder-only models must be left-padded for batched generation — generation continues from the sequence's end, and if the end is padding (right-padded), the model continues from garbage. padding_side="left" is the fix, and it's the kind of detail that separates "ran the quickstart" from "debugged this at 2 a.m."

Truncation cuts to max_length (with strategies for pair inputs — truncate the longer, or one side only). Silent truncation of long RAG contexts is a classic quality bug: the evidence you retrieved fell off the end and nobody noticed. Log your token counts.

Offset mapping. Fast (Rust) tokenizers return, per token, its (start, end) character span in the original string — the bridge between token space and text space that NER span extraction and QA answer-slicing depend on. Labs 01 and 02 both implement offsets, because "map the model's token-level decision back to the user's actual text" is a recurring production need (highlighting, redaction, citation).

7. Text generation: decoding strategies and the KV cache

model.generate() is a loop: each step runs the model for next-token logits, transforms those logits, picks a token, appends it, and checks stopping. Every decoding knob is one transform in that loop, and GenerationConfig (a real, savable object shipping with models — which is why the same model can feel different across versions: its default generation settings are an artifact too) is the bag of knobs. Lab 02 implements the loop and every transform below.

Greedy (do_sample=False): take the argmax each step. Deterministic, and prone to repetition loops on open-ended text; fine for constrained extraction.

Beam search (num_beams=k): keep the k highest-cumulative-log-prob hypotheses, expand each, keep the best k, with a length_penalty to counter short-sequence bias. Strong for closed-form tasks (translation, summarization) where "highest-probability sequence" aligns with quality; weak for open-ended generation, where it converges on bland high-probability text.

Sampling (do_sample=True) draws from the (transformed) distribution:

  • Temperature divides logits by \(T\) before softmax: \(T < 1\) sharpens the distribution toward the top tokens, \(T > 1\) flattens it toward uniform, \(T \to 0\) degenerates to greedy. It reshapes probabilities, never the ranking.
  • Top-k keeps only the k highest-probability tokens, renormalized. Static: k candidates whether the model is certain or lost.
  • Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability reaches p — adaptive: a confident distribution yields few candidates, a flat one yields many. This is why top-p generally supersedes top-k as the default sampling filter.
  • Repetition penalty (CTRL-style, what HF implements): for every token already in the sequence, divide its logit by the penalty if positive, multiply if negative — both push seen tokens down. Related: no_repeat_ngram_size hard-bans repeating any n-gram, a blunter tool.

Stopping: eos_token_id emitted, max_new_tokens exhausted, or a custom StoppingCriteria (stop-strings, budget). In HF's implementation, each transform above is literally a LogitsProcessor object applied in a fixed sequence — the same composition Lab 02's loop makes explicit.

The KV cache is the performance half of generation. Attention at step \(n\) needs keys and values for all previous tokens; recomputing them every step makes decoding \(O(n^2)\) per token. Caching past keys/values (use_cache=True, on by default) makes each step incremental — compute K/V only for the newest token. Consequences a senior engineer should recite: the cache is why token generation is memory-bandwidth-bound, why long contexts eat VRAM linearly (per layer, per head), and why serving-side innovations (§13's paged/continuous batching) are mostly KV-cache management innovations. Prefill (the whole prompt in parallel) vs decode (one token at a time against the cache) is also why time-to-first-token and inter-token latency are different metrics with different bottlenecks.

8. The Hub: models, datasets, Spaces, cards, gating, versioning

The Hub is three artifact types on one storage substrate:

  • Model repos — weights (safetensors, often sharded), config.json, tokenizer files, a README.md model card. Over a million public repos, from frontier-lab releases to one-off fine-tunes.
  • Dataset repos — data files (commonly Parquet) plus a dataset card; load_dataset("org/name") pulls them (§9).
  • Spaces — hosted demo apps (Gradio/Streamlit/static/Docker) attached to the same repo mechanics; the standard way a model release ships a try-it-now page.

Every repo is a git repository (large files via an LFS-style mechanism): commits, branches, tags, diffs, PR-like "discussions" with proposed changes. This is the fact that everything else hangs off: revision= (§4) is git resolution; push_to_hub is a commit; reproducibility is sha-pinning. Lab 03's miniature keeps exactly this shape — full-snapshot commits, tags, main as a floating head — with deterministic counter shas.

Model cards are the repo's README.md with YAML front-matter: license, language, tags, base model, datasets used, eval results. The prose half documents intended use, training data, limitations, and biases. Treat cards as an engineering input: the license field gates adoption (§14), the base-model field tells you what an adapter needs, and eval claims tell you what to verify. The huggingface_hub client (HfApi, hf_hub_download, snapshot_download, create_repo, upload_folder) is the programmatic surface over all of it.

Gated models add an access-request step: you accept terms (Llama's community license is the canonical example) before downloads are authorized for your access token. Mechanically: authenticate (huggingface-cli login or HF_TOKEN), accept once per repo, and your CI needs a token with the grant — a real deployment-pipeline consideration, not a browser formality.

9. The datasets library: Arrow, memory-mapping, streaming

datasets answers "how do I feed corpora bigger than RAM through preprocessing and training without writing a data engine." Three mechanisms:

Arrow + memory-mapping. A loaded dataset is stored as Apache Arrow files on disk and memory-mapped — the OS pages data in on access, so a 500 GB dataset "loads" instantly and iterates at disk speed with near-zero Python-heap footprint. Arrow's columnar layout also means zero-copy handoff to tensors for many types.

map with fingerprinted caching. dataset.map(fn, batched=True, num_proc=8) applies a transform (tokenization being the canonical one) with batching and multiprocessing — and writes the result to a cache keyed by a fingerprint of the input data + the function. Re-run the same script: instant cache hit. Change the tokenizer or the function: the fingerprint changes and it recomputes. This idempotent-preprocessing property is what makes "tokenize the whole corpus up front" a sane default. filter, shuffle, train_test_split, select and friends follow the same lazy/cached discipline, and with_format("torch") yields tensors directly.

Streaming. load_dataset(..., streaming=True) returns an IterableDataset that yields examples straight from remote files, downloading nothing up front — how you sample or train on web-scale corpora without provisioning their footprint. The tradeoff is honest: no random access, no global shuffle (a shuffle-buffer approximation instead), no fingerprinted caching — it is a pipe, not a table.

The composition that shows up in every fine-tuning script: load_dataset → map the tokenizer over everything (batched=True) → hand to Trainer, whose data collator handles per-batch padding (§6) — static preprocessing cached once, dynamic padding done per batch.

10. Fine-tuning: Trainer, TrainingArguments, accelerate

Trainer is the batteries-included supervised training loop of transformers. You provide a model, a TrainingArguments, datasets, and optionally a collator/metrics function; it owns the loop, checkpointing, evaluation, logging, mixed precision, and distribution:

from transformers import TrainingArguments, Trainer
args = TrainingArguments(
    output_dir="out", per_device_train_batch_size=8, gradient_accumulation_steps=4,
    learning_rate=2e-5, num_train_epochs=3, bf16=True,
    eval_strategy="steps", eval_steps=500, save_strategy="steps", save_steps=500,
    load_best_model_at_end=True, push_to_hub=True,
)
trainer = Trainer(model=model, args=args, train_dataset=train, eval_dataset=dev,
                  data_collator=collator, compute_metrics=metrics_fn)
trainer.train()

The TrainingArguments fields worth being able to explain, not just set: gradient_accumulation_steps (simulate a large batch on small memory — effective batch = per-device × accumulation × world size), bf16/fp16 (mixed precision — bf16 preferred on hardware that has it, for fp32-like dynamic range), warmup + weight_decay (the standard AdamW recipe), eval_strategy/save_strategy + load_best_model_at_end (checkpoint discipline), and push_to_hub (checkpoints as Hub commits — versioned training artifacts for free). Data collators matter conceptually: DataCollatorWithPadding does dynamic per-batch padding (pad to the batch max, not a global max — §6's efficiency point), and DataCollatorForLanguageModeling builds causal-LM labels. For chat-style LLM fine-tuning, the TRL library's SFTTrainer layers prompt-formatting and packing conveniences over this same machinery.

accelerate is the layer that makes one training script run anywhere. Wrap the loop's objects (accelerator.prepare(model, optimizer, dataloader)) and launch with accelerate launch script.py; the same code runs single-GPU, multi-GPU (DDP), multi-node, or under FSDP/DeepSpeed sharding (for when the optimizer states and gradients — the memory hogs of full fine-tuning — must be sharded across devices). Two facts to keep straight: Trainer is itself built on accelerate internally (configure FSDP/DeepSpeed through TrainingArguments and it flows down), and accelerate also powers big-model inference — device_map="auto" shards weights across GPUs/CPU/disk at load time so a model that doesn't fit one device still loads.

The economics that motivate the next section: full fine-tuning with AdamW keeps ~4 bytes of optimizer state per parameter on top of weights and gradients — for 7B parameters, on the order of 100 GB of training state. That figure is why PEFT exists.

11. PEFT: LoRA, QLoRA, adapters, prompt tuning

LoRA (Low-Rank Adaptation) rests on one empirical claim: fine-tuning weight updates have low intrinsic rank. So freeze the pretrained weight \(W \in \mathbb{R}^{d \times k}\) and learn a factored update:

$$ W' = W + \frac{\alpha}{r} A B, \qquad A \in \mathbb{R}^{d \times r},\ B \in \mathbb{R}^{r \times k},\ r \ll \min(d, k) $$

Only \(A\) and \(B\) train. For a 4096×4096 attention projection at \(r = 8\), that is 65 K trainable parameters instead of 16.8 M — well under 1% model-wide, which collapses optimizer state, lets you fit fine-tuning where it otherwise wouldn't, and produces megabyte adapter files instead of full checkpoints. One factor is zero-initialized so training starts exactly at the base model's behavior. In peft:

from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                    target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(base_model, config)
model.print_trainable_parameters()   # e.g. "trainable params: 4.2M || all params: 7B || 0.06%"

The knobs: r (rank — capacity of the update; 8–64 typical), lora_alpha (scaling numerator; the effective multiplier is \(\alpha / r\)), target_modules (which named layers get adapters — attention projections q_proj/v_proj are the classic minimal set per the LoRA paper; adding k_proj/o_proj and the MLP projections is the "more capacity" dial). Deployment is a real choice with a real tradeoff, and Lab 03 proves both sides: merge (merge_and_unload() folds \(\frac{\alpha}{r}AB\) into \(W\) — mathematically identical outputs, zero added latency, but the adapter is no longer separable) versus swap (keep adapters separate; one shared base serves many tasks/tenants by switching adapters — at a small extra-matmul cost per forward).

QLoRA = the same adapters over a 4-bit quantized base. Its three tricks: NF4 (NormalFloat4 — a 4-bit datatype whose quantization levels are optimal for normally-distributed weights), double quantization (quantize the quantization constants themselves, saving ~0.4 bits/parameter), and paged optimizers (spill optimizer state to CPU on memory spikes). Compute still happens in bf16 (weights dequantize per-operation); gradients flow through the frozen 4-bit base into the full-precision A/B. Net effect: fine-tune a 65B model on a single 48 GB GPU — the paper's headline, and the reason "we fine-tuned our own model" stopped requiring a cluster.

The wider PEFT family, one line each: prompt tuning learns soft embedding vectors prepended to the input (the model is untouched); prefix tuning learns per-layer key/value prefixes; p-tuning learns prompts via a small encoder; IA³ learns multiplicative rescaling vectors. All share the shape freeze the model, train a small bolt-on — LoRA dominates in practice because it matches full fine-tuning quality most reliably and merges away at serving time.

12. Quantization: bitsandbytes, GPTQ, AWQ

Quantization stores weights in fewer bits — fp16 → int8/int4 — cutting memory (a 7B model: ~14 GB fp16, ~3.5–4 GB at 4-bit) and usually improving speed on memory-bandwidth-bound decode (§7). Three names, three strategies, and knowing which is which is the interview signal:

  • bitsandbytes — load-time, no calibration. load_in_8bit implements LLM.int8() (vector-wise int8 with an fp16 side-path for emergent outlier features — the trick that made int8 work on large transformers); load_in_4bit is QLoRA's NF4. Zero preparation (a BitsAndBytesConfig on from_pretrained), moderate quality cost, and the default for fitting a model on your GPU right now and for QLoRA training.
  • GPTQ — post-training, calibrated, one-shot. Using a few hundred calibration samples, quantize layer by layer, choosing quantized values that minimize output error with a Hessian-based (approximate second-order) compensation: when a weight is rounded, remaining weights adjust to absorb the error. Produces a quantized artifact (many pre-quantized GPTQ repos on the Hub) with strong 4-bit quality and fast inference kernels.
  • AWQ — activation-aware, calibrated, reconstruction-free. Its observation: a small fraction of weight channels matter disproportionately, and activation magnitudes (not weight magnitudes) reveal them. AWQ scales those salient channels up before quantization (and inverts the scale in the adjacent op), protecting what matters without mixed precision. Comparable quality to GPTQ, often better latency, cheaper calibration; also shipped as pre-quantized Hub artifacts.

The tradeoff summary a senior engineer gives: bitsandbytes when you want zero-prep loading or QLoRA training; GPTQ/AWQ when you're producing a serving artifact and can spend calibration effort; measure quality on your evals, because degradation is task-dependent — 4-bit is usually a modest hit on general language but can visibly hurt math/code/long-tail knowledge. (Adjacent, worth naming: GGUF is llama.cpp's quantized format for CPU/edge serving — a different runtime family, same idea.)

13. Serving: TGI, Inference Endpoints, continuous batching

Text Generation Inference (TGI) is Hugging Face's production LLM server — the thing you actually run behind an API. docker run ghcr.io/huggingface/text-generation-inference --model-id mistralai/Mistral-7B-Instruct-v0.3 gives you an HTTP server with token streaming (SSE), an OpenAI-compatible chat-completions endpoint (the Messages API), tensor parallelism for multi-GPU sharding, and quantized loading (bitsandbytes/GPTQ/AWQ, §12). The two mechanisms that make it (and its peer vLLM) categorically better than naive serving:

Continuous batching. Naive static batching runs a batch of requests to completion; because generation lengths vary wildly, finished sequences leave GPU slots idle until the longest straggler ends, and new requests wait outside. Continuous (in-flight) batching schedules at iteration granularity: every decode step, completed sequences exit the batch and queued requests join it immediately. GPU utilization stops depending on length variance, throughput multiplies under real traffic, and p99 latency stops being hostage to whoever asked for 2,000 tokens. This is the single most important LLM-serving idea of the last few years.

Paged KV-cache management. §7 established that the KV cache dominates serving memory. PagedAttention (introduced by vLLM, adopted across the ecosystem) manages the cache in fixed-size blocks — virtual-memory style — instead of contiguous per-request allocations, eliminating the fragmentation that otherwise caps batch size, and enabling prefix sharing. Together with continuous batching, this is why "requests per GPU" on a modern server can be an order of magnitude above a hand-rolled generate() loop behind FastAPI.

Inference Endpoints is the managed product: pick a Hub repo (public, private, or gated), pick cloud/region/instance, get an autoscaling (including scale-to-zero) HTTPS endpoint running TGI (or custom handlers) with your access controls. The build-vs-buy line is the same one this track draws everywhere: Endpoints when the team shouldn't own GPU infrastructure yet; self-hosted TGI/vLLM when cost, data locality, or custom serving logic says you should. And the composition with the rest of this track: TGI is what sits behind the provider adapters of Phase 24-style gateways when the model is one you serve yourself.

14. Licensing, gated access, and the model supply chain

"Open model" is not one legal category, and a principal engineer is expected to know the difference before the lawyers do:

  • Permissive open source — Apache-2.0 (Mistral's early releases, Qwen's newer ones, most BERT-era models), MIT. Commercial use, modification, redistribution: yes.
  • Community licenses — Meta's Llama license is the canonical example: broadly permissive but not OSI-open (acceptable-use policy, redistribution conditions, and a scale threshold clause requiring very large companies to request a separate license). Fine for most; a real legal review item for some.
  • RAIL-style licenses (Responsible AI Licenses, e.g. BigScience's OpenRAIL) — permissive with use-based behavioral restrictions baked into the license.
  • Non-commercial / research-only — evaluation allowed, production forbidden. Check before the prototype ships.

Mechanics that follow: the model card's license field is the first artifact of any adoption review; gated repos (§8) enforce terms-acceptance before download, meaning production pipelines need authenticated tokens with the grant; derivatives inherit obligations (a LoRA adapter of Llama is a Llama derivative — your adapter repo on the Hub should say so, and Lab 03's ModelCard carries license metadata for exactly this reason); and datasets have licenses too — training on data you had no right to train on is a different, equally real risk.

Fold in §4's load-safety story and you get the full supply-chain checklist this phase wants you to internalize: license reviewed → gated terms accepted → revision pinned → safetensors only → no trust_remote_code without code review → adapter/base lineage documented. Six lines; each one has a production incident behind it somewhere.

15. Common misconceptions

  • "Hugging Face is a model." It's an ecosystem (libraries + a registry). The models belong to their publishers; HF is the packaging convention and distribution rails.
  • "pipeline() is for production." It's a superb default and demo surface; production code usually drops to AutoModel/AutoTokenizer for control over batching, devices, revisions, and error handling — or to TGI, which replaces the loop entirely.
  • "The tokenizer is an implementation detail." It's a trained artifact versioned with the model. Mismatching them, skipping the chat template, or right-padding batched generation are all silent-quality bugs, not crashes.
  • "Temperature changes what the model knows." It rescales the existing next-token distribution. It cannot add knowledge; at 0 it just becomes argmax.
  • "generate() is one forward pass." It's a loop — one forward per token, against a KV cache. That's why output length drives latency and why serving optimizations target the cache.
  • "LoRA is approximate fine-tuning." LoRA is a constrained parameterization of the update (low-rank), not a lossy compression of a full fine-tune. And a merged LoRA is exactly equivalent to the unmerged adapter's outputs — Lab 03 proves the identity numerically.
  • "QLoRA quantizes your fine-tune." QLoRA quantizes the frozen base during training; the adapters stay high-precision. The artifact you ship is a normal LoRA adapter.
  • "safetensors is just faster." It's safer (no code execution — pickle is an RCE surface) and faster (zero-copy mmap). The security property is the reason it exists.
  • "trust_remote_code=True is a compatibility flag." It executes arbitrary repo-provided Python. Treat it as a code-review event, pinned to a revision, or not at all.
  • "Open model means do whatever you want." Llama's community license, RAIL restrictions, and research-only releases are all "open weights" with real obligations. Read the card.
  • "Streaming datasets are always better." Streaming trades random access, global shuffling, and cachability for zero-footprint iteration. Map-style memory-mapped datasets are the right default until data size forces the trade.

16. Lab walkthrough

Build the three miniatures in order; each isolates one layer of the ecosystem and injects the model as a pure function, so everything is deterministic and offline.

  1. Lab 01 — The pipeline() Abstraction. Implement softmax/argmax, the Pipeline base (preprocess → forward → postprocess, batch-vs-single contract), three task post-processors (text-classification with id2label + score ranking; token-classification with span aggregation over offset mappings; text-generation as a greedy decode loop), and the pipeline() factory with real task aliases and unknown-task errors. 31 tests.
  2. Lab 02 — BPE Tokenization & generate(). Train a BPE tokenizer (most-frequent-pair merges, deterministic tie-break), implement encode_plus/decode with special tokens, padding, truncation, attention masks, and offset mappings — then the generate() loop: repetition penalty, temperature, top-k, top-p, seeded sampling, EOS/budget stopping, all honoring a GenerationConfig. 30 tests.
  3. Lab 03 — PEFT/LoRA & the Hub. Implement the list-of-lists linear algebra, LoRALinear (frozen base, zero-init adapters, the \(\alpha/r\) scaling, merge with a proven output identity, save/load/swap), and the Hub miniature (full-snapshot commits, tags, revision pinning, the safetensors-vs-pickle gate behind trust_remote_code), glued by push_adapter/load_adapter. 31 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

17. Success criteria

  • You can name the major libraries in the HF ecosystem and each one's single job.
  • You can describe pipeline()'s three stages and what changes per task — and when to drop below it.
  • You can explain how Auto* classes dispatch on config.json and what the task heads are.
  • You can state the safetensors-vs-pickle difference as a security property and say when trust_remote_code is justifiable.
  • You can run BPE training by hand on a toy corpus and contrast BPE/WordPiece/Unigram in one sentence each.
  • You can explain attention masks, left-padding for generation, and chat templates as concrete bug classes.
  • You can walk through generate()'s transform order and explain the KV cache's role in latency and memory.
  • You can write the LoRA update formula from memory, explain r/alpha/target_modules, and state the merge-vs-swap tradeoff; you can say what QLoRA quantizes and what it doesn't.
  • You can place bitsandbytes, GPTQ, and AWQ on the "when would I use it" map.
  • You can explain continuous batching and paged KV-cache management and why they multiply serving throughput.
  • All three labs pass under both lab and solution (92 tests total).

18. Interview Q&A

Q: What actually happens inside pipeline("sentiment-analysis")("I love this")? A: Three stages. Preprocess: the tokenizer converts the string to input_ids + attention mask, with padding/truncation policy. Forward: the model runs under no_grad, producing one logit per label. Postprocess — the task-specific part: softmax the logits, map the argmax through model.config.id2label, return {label, score}. The pipeline() call itself is a factory that resolved the alias ("sentiment-analysis" is text-classification), picked the subclass, and loaded a default model if none was named — which is also why unpinned pipeline defaults don't belong in production.

Q: Why does the same text tokenize to different lengths on different models? A: The tokenizer is a trained artifact — its vocabulary comes from merge statistics (BPE), likelihood gains (WordPiece), or pruned unigram likelihood (Unigram/SentencePiece) over that model's training corpus. Different corpora and budgets yield different segmentations, so token counts — and therefore cost, latency, and context usage — differ per model for identical text. Fertility on your own domain text is a legitimate model-selection metric.

Q: What bugs does the attention mask prevent, and why left-padding for generation? A: The mask marks real tokens vs padding; without it the model attends to pad embeddings and quality degrades silently. Decoder-only generation continues from the last position, so right-padded batches would continue from padding — batched generation therefore left-pads, keeping every sequence's real end adjacent to where generation starts.

Q: Walk me through what temperature, top_k, and top_p each do. A: All are transforms on the next-token distribution inside the decode loop. Temperature divides the logits — below 1 sharpens toward the top tokens, above 1 flattens, 0 degenerates to greedy argmax; it never reorders tokens. Top-k truncates to a fixed k candidates and renormalizes — static regardless of model confidence. Top-p keeps the smallest set of tokens reaching cumulative probability p — adaptive: few candidates when the model is confident, many when it's uncertain, which is why it's generally preferred. They compose in a fixed order (HF applies them as a LogitsProcessor chain) before sampling.

Q: What is the KV cache and why does it matter for serving? A: Attention at each decode step needs keys/values for all prior tokens; caching them makes each step compute K/V only for the new token instead of recomputing the whole prefix — turning per-token cost from quadratic to incremental. It's also the dominant memory consumer at serving time, growing linearly with context per request — which is why modern servers' big wins (paged KV blocks, continuous batching) are cache-management wins, and why decode is memory-bandwidth-bound while prefill is compute-bound.

Q: Explain LoRA to a skeptical infra engineer. A: Fine-tuning updates have low intrinsic rank, so instead of updating W we freeze it and learn W' = W + (α/r)·A·B where A and B are thin rank-r matrices — typically under 1% of parameters. That collapses optimizer memory, makes the artifact megabytes instead of gigabytes, and gives a deployment choice: merge A·B into W for zero-overhead serving (outputs are mathematically identical — it's addition of linear maps), or keep adapters separate and swap them per task over one shared base. QLoRA is the same thing over a 4-bit NF4-quantized frozen base, which is what moved fine-tuning onto single GPUs.

Q: What's the difference between bitsandbytes, GPTQ, and AWQ? A: Strategy. bitsandbytes quantizes at load time with no calibration — LLM.int8's outlier handling for 8-bit, NF4 for 4-bit; it's the zero-prep option and the QLoRA training substrate. GPTQ is post-training and calibrated: layer-by-layer, it picks quantized weights minimizing output error with Hessian-guided compensation, producing a quantized artifact with strong 4-bit quality. AWQ is also calibrated but activation-aware and reconstruction-free: it identifies the small set of salient weight channels via activation magnitudes and protects them by scaling. Rule of thumb: bitsandbytes to fit or train now, GPTQ/AWQ to produce a serving artifact — and always re-run your own evals, because 4-bit degradation is task-dependent.

Q: Why does safetensors exist? A: PyTorch's .bin checkpoints are pickle, and unpickling executes arbitrary code — a model file is an RCE payload waiting for torch.load. safetensors is pure tensor data plus a JSON header: nothing executable, plus zero-copy memory-mapped loading as a performance bonus. It converts "loading a stranger's model" from code execution into data parsing, which is why "safetensors only" is a supply-chain policy. trust_remote_code=True is the separate, explicit opt-in for repos that ship custom architecture code — treat it like running an untrusted install script: review it and pin the revision, or don't.

Q: How do you make a Hugging Face deployment reproducible? A: Pin everything the Hub lets float: revision= a commit sha (not main) for model and tokenizer, snapshot the GenerationConfig (it ships with the model and changes behavior), record the adapter's revision if PEFT is involved, and cache artifacts (HF_HUB_OFFLINE in CI) so builds don't depend on live network. The Hub is git underneath, so this is exactly container-digest discipline applied to models.

Q: What is continuous batching and why did it become the serving standard? A: Static batching runs a fixed batch to completion, so variable generation lengths leave GPU slots idle behind the longest sequence while new requests queue. Continuous batching reschedules every decode iteration: finished sequences leave, waiting requests join mid-flight. Combined with paged KV-cache allocation (block-based, virtual-memory style, killing fragmentation), it takes GPU utilization from length-variance-hostage to consistently high — the order-of-magnitude throughput difference between TGI/vLLM and a hand-rolled generate() loop behind FastAPI.

Q: Product wants to fine-tune and self-host an open model. Sketch the workflow and its gotchas. A: Shortlist on the Hub with licenses reviewed (community licenses and gated terms are real obligations) and revisions pinned. Prepare data with datasets (memory-mapped, map tokenization with the model's own tokenizer and chat template). Fine-tune with QLoRA — 4-bit base, LoRA on attention projections, Trainer/TRL on accelerate — producing a small adapter; eval against the base before celebrating. Publish the adapter to a private repo with a card naming the base and license lineage; tag it. Serve by merging the adapter and quantizing (AWQ/GPTQ) onto TGI with continuous batching, or keep adapters separate if one base serves many tasks. Gotchas: tokenizer/chat-template mismatch, unpinned revisions, trust_remote_code, left-padding, and skipping the quantized-model eval.

Q: Where does Hugging Face sit relative to Bedrock, and when do you pick which? A: Different layers of the same decision. Bedrock (Phase 24) is a managed serving platform — you rent invocation of models someone else operates. Hugging Face is the ecosystem for owning the model layer: pick open weights, adapt with PEFT, quantize, serve on TGI/Endpoints. Pick managed APIs while product-market fit is the bottleneck; pick the HF path when unit economics at scale, data locality, domain adaptation, or model control dominate — and note the two compose: Bedrock's Custom Model Import ingests HF-format safetensors, and an Endpoints or TGI deployment slots behind the same gateway abstractions this track builds everywhere else.

19. References

  • Hugging Face — Transformers documentation (pipelines, Auto classes, generation, Trainer). https://huggingface.co/docs/transformers
  • Hugging Face — Tokenizers documentation (BPE/WordPiece/Unigram, fast tokenizers, offsets). https://huggingface.co/docs/tokenizers
  • Hugging Face — the NLP/LLM Course, chapter 6 (training tokenizers, algorithm comparisons). https://huggingface.co/learn
  • Hugging Face — Hub documentation (repos, revisions, model cards, gated models, safetensors). https://huggingface.co/docs/hub
  • Hugging Face — huggingface_hub client documentation (hf_hub_download, HfApi, push_to_hub). https://huggingface.co/docs/huggingface_hub
  • Hugging Face — Datasets documentation (Arrow backing, memory-mapping, map/filter, streaming). https://huggingface.co/docs/datasets
  • Hugging Face — Accelerate documentation (prepare/launch, FSDP/DeepSpeed, big model inference). https://huggingface.co/docs/accelerate
  • Hugging Face — PEFT documentation (LoraConfig, get_peft_model, merging adapters). https://huggingface.co/docs/peft
  • Hugging Face — Text Generation Inference documentation (continuous batching, Messages API, quantized serving). https://huggingface.co/docs/text-generation-inference
  • Hugging Face — Inference Endpoints documentation. https://huggingface.co/docs/inference-endpoints
  • Hugging Face — safetensors documentation. https://huggingface.co/docs/safetensors
  • Sennrich, Haddow & Birch — Neural Machine Translation of Rare Words with Subword Units (the BPE-for-NMT paper). https://arxiv.org/abs/1508.07909
  • Kudo — Subword Regularization (the Unigram LM tokenizer) and Kudo & Richardson — SentencePiece. https://arxiv.org/abs/1804.10959 and https://arxiv.org/abs/1808.06226
  • Holtzman et al. — The Curious Case of Neural Text Degeneration (nucleus/top-p sampling). https://arxiv.org/abs/1904.09751
  • Keskar et al. — CTRL: A Conditional Transformer Language Model (the repetition penalty). https://arxiv.org/abs/1909.05858
  • Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685
  • Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs (NF4, double quantization, paged optimizers). https://arxiv.org/abs/2305.14314
  • Dettmers et al. — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (bitsandbytes 8-bit). https://arxiv.org/abs/2208.07339
  • Frantar et al. — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
  • Lin et al. — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
  • Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM; the paged KV-cache idea TGI also adopted). https://arxiv.org/abs/2309.06180
  • Mitchell et al. — Model Cards for Model Reporting. https://arxiv.org/abs/1810.03993