Warmup Guide — Classical NLP & Embeddings

Zero-to-expert primer for Phase 02: how meaning becomes geometry — from counting words to word2vec's learned vectors — the conceptual foundation every embedding system (including modern RAG) still stands on.

Table of Contents

Chapter 1: The Distributional Hypothesis

From zero: computers need numbers; words are symbols. The naive encoding — one-hot vectors (vocabulary-sized, single 1) — makes every pair of words equidistant: cat is as far from kitten as from carburetor. No similarity structure, vocabulary-sized dimensionality, and no generalization (what the model learns about cat says nothing about kitten).

The escape is a 1950s linguistic idea (Firth: "you shall know a word by the company it keeps"): words appearing in similar contexts have similar meanings. This turns semantics into a statistics problem — measure each word's context distribution, and words with similar distributions get similar representations. Every embedding method in this phase (and every contextual model after it) is an implementation of this single hypothesis; the methods differ only in how they compress context statistics into vectors.

Chapter 2: Count-Based Representations — BoW and TF-IDF

The pre-neural baselines — still in production everywhere (search engines, spam filters), and the honest baseline your lab compares against:

  • Bag of Words: document → vector of word counts. Order discarded ("dog bites man" = "man bites dog"), but for topical tasks counts carry most of the signal.
  • TF-IDF fixes BoW's flaw — common words (the) dominate counts while carrying no discriminative information:

$$\text{tfidf}(t, d) = \text{tf}(t, d) \cdot \log\frac{N}{\text{df}(t)}$$

Term frequency × inverse document frequency: a term scores high if frequent in this document and rare across documents. The log dampens; smoothing (+1) avoids division by zero. The IDF weighting idea — downweight the ubiquitous — recurs everywhere from BM25 (RAG's sparse-retrieval workhorse, Phase 07) to attention entropy analysis.

  • Limits that motivate the rest of the phase: vectors are vocabulary-sized and sparse; synonyms remain orthogonal (car ⊥ automobile — exact-match only); word order and polysemy invisible.

Chapter 3: Vector Similarity — Why Cosine

$$\text{cosine}(u, v) = \frac{u \cdot v}{|u|,|v|} \in [-1, 1]$$

— the angle, ignoring magnitude. Why magnitude is ignored deliberately: in count spaces, document length inflates magnitude without changing topic; in learned embedding spaces, frequency-related magnitude artifacts similarly pollute. Direction carries the semantics.

Three operational facts you'll reuse for the rest of the curriculum:

  1. On L2-normalized vectors, cosine, dot product, and Euclidean distance produce identical rankings ($|u-v|^2 = 2 - 2u\cdot v$) — which is why vector databases normalize once and use dot product (cheapest) internally.
  2. Exact top-k search is O(N·d) per query — fine to ~1M vectors, then approximate indexes (HNSW, IVF — Phase 07's territory) trade recall for speed.
  3. In high dimensions, random vectors concentrate near orthogonality — cosine similarities of unrelated items cluster around 0 with small variance, so small absolute differences (0.31 vs 0.35) can be meaningful. Calibrate thresholds empirically per embedding model; never copy a threshold across models.

Chapter 4: word2vec — Learning Vectors by Prediction

The 2013 leap: don't count contexts, predict them — and harvest the network's weights as the representation.

Skip-gram (the variant your lab implements): for each word, predict the words in a window around it. Architecture is almost embarrassingly simple — two matrices: $W_{in} \in \mathbb{R}^{V \times d}$ (center-word vectors) and $W_{out} \in \mathbb{R}^{V \times d}$ (context-word vectors). Score of context $o$ given center $c$: $u_o^\top v_c$, softmaxed over the vocabulary:

$$P(o \mid c) = \frac{\exp(u_o^\top v_c)}{\sum_{w \in V} \exp(u_w^\top v_c)}$$

Training maximizes this over all (center, context) pairs in the corpus. The product: $W_{in}$'s rows — vectors where similarity-of-context became geometric proximity, because words with the same neighbors received the same gradient pushes.

CBOW is the mirror (predict center from averaged context) — faster, slightly worse on rare words. The deeper unification (Levy & Goldberg): skip-gram with negative sampling implicitly factorizes the PMI (pointwise mutual information) co-occurrence matrix — the neural method and the count-based methods converge on the same statistics, which is the distributional hypothesis showing through.

Chapter 5: Negative Sampling — The Trick That Made It Tractable

The softmax's denominator sums over the whole vocabulary — at V = 100K, every training step costs a 100K-way normalization. Skip-Gram with Negative Sampling (SGNS) replaces the multiclass problem with binary discrimination: for each true (center, context) pair, sample $k$ random "negative" words and train the model to score real pairs high and fakes low:

$$\mathcal{L} = -\log \sigma(u_o^\top v_c) - \sum_{i=1}^{k} \log \sigma(-u_{n_i}^\top v_c)$$

Cost per step: $k+1$ dot products instead of $V$. Details that matter (and that the lab implements):

  • Negatives are drawn from the unigram distribution raised to the 3/4 power — flattening it so rare words get sampled enough; the 0.75 is empirical and load-bearing.
  • Frequent-word subsampling: drop training occurrences of very frequent words with probability $1 - \sqrt{t/f(w)}$ — the appears millions of times and teaches nothing new after the first thousand.
  • $k$ = 5–20 for small corpora, 2–5 at scale.

This "turn an intractable softmax into sampled binary classification" move is a recurring pattern — you'll meet its cousins in contrastive learning (CLIP, Phase 06 of the CV track) and InfoNCE.

Chapter 6: What Embedding Spaces Encode (and Don't)

  • The famous linear structure: king - man + woman ≈ queen — relations as directions (gender, tense, capital-of). Real but overstated: it works for frequent, clean relations and degrades elsewhere; evaluate with analogy suites, don't assume.
  • Similarity vs relatedness: embeddings conflate them — coffee is close to cup (related) and to tea (similar). Tasks that need one but not the other (retrieval vs substitution) need different training objectives; this distinction returns with force in RAG retrieval quality (Phase 07).
  • Antonyms are close: hot/cold share contexts almost perfectly. Distributional methods cannot see the polarity flip — a permanent limitation downstream tasks must handle.
  • Bias is faithfully learned: occupation–gender associations and worse, present in the corpus statistics, become geometry. Debiasing-by-projection exists and is partial (Gonen & Goldberg's "lipstick on a pig"); the production answer is evaluation and task-level mitigation, not pretending the vectors are neutral.
  • One vector per word: bank (river/finance) gets a frequency-weighted average of its senses — the polysemy failure that motivates Chapter 7.

Chapter 7: From Static to Contextual — Why This Still Matters

Static embeddings assign one vector per type; contextual models (ELMo → BERT → every LLM layer) produce a vector per occurrence, computed from the sentence — solving polysemy and word order at the cost of running a model per text. The modern landscape you're heading toward:

  • Sentence/document embedding models (the RAG workhorses, Phase 07) are contextual encoders pooled to one vector and trained contrastively — conceptually: word2vec's objective, upgraded with transformers and curated positives/negatives.
  • The mechanics transfer wholesale: cosine similarity, normalization, the similarity-vs-relatedness distinction, threshold calibration, nearest-neighbor search — everything in Chapters 3 and 6 is daily working knowledge for an inference engineer running a vector database.
  • And inside every LLM, the embedding table (Phase 01's contract) is a static embedding layer — token vectors whose geometry you'll inspect when debugging glitch tokens and quantization damage to lm_head/embeddings (model-accuracy track).

Lab Walkthrough Guidance

Lab 01 — word2vec from Scratch:

  1. Build the data pipeline first: tokenize (Phase 01's tools), build vocab with min frequency, generate (center, context) pairs with a window; verify pair counts by hand on a toy sentence.
  2. Implement SGNS loss exactly as Chapter 5's equation — two embedding matrices, $k$ negatives from the unigram^0.75 table (build the sampling table once; verify its marginal distribution empirically).
  3. Add frequent-word subsampling; measure its effect on training speed and on rare-word neighbor quality (it should help both — understand why before believing it).
  4. Train on text8 or a Wikipedia slice; sanity-check during training with a fixed probe set (king, paris, coffee nearest neighbors every N steps — watching neighbors sharpen is the lab's payoff moment).
  5. Evaluate: nearest neighbors, a handful of analogies, and a 2-D projection (PCA or t-SNE — with the standard caveat that t-SNE distorts global structure; don't over-read the picture).

Success Criteria

You are ready for Phase 03 when you can, from memory:

  1. State the distributional hypothesis and show how both TF-IDF and word2vec implement it.
  2. Write the TF-IDF formula and explain each factor's job.
  3. Explain why cosine over raw distance, the normalization-makes-them-equivalent fact, and the high-dimensional concentration caveat.
  4. Write the SGNS loss and justify the 3/4 power and subsampling.
  5. Name four things embedding spaces get wrong (antonyms, polysemy, relatedness conflation, bias) with the mechanism behind each.
  6. Trace the lineage: one-hot → counts → static learned → contextual, with the failure that drove each transition.

Interview Q&A

Q: Why did word2vec beat count-based methods if they capture the same statistics? Levy & Goldberg showed SGNS implicitly factorizes the PMI matrix — so the information is the same; the win was practical: dense low-dimensional vectors directly usable downstream, scalable online training over arbitrarily large corpora (no V×V matrix to materialize), and hyperparameters (subsampling, negative distribution, dynamic windows) that act as effective regularizers. With matched preprocessing, tuned SVD-over-PMI is competitive — knowing that is the difference between folklore and understanding.

Q: Your RAG system retrieves topically-related but unhelpful passages. Connect to this phase. Similarity-vs-relatedness conflation (Ch. 6): the embedding model scores coffee prices near coffee brewing because they share context distributions, but the task needed answer-bearing similarity. Fixes are objective-level — embedding models trained on (query, answer) pairs rather than symmetric similarity, hybrid BM25+dense scoring (exact terms carry intent), and rerankers (Phase 07). The diagnosis vocabulary comes from static-embedding days; the failure never went away.

Q: Why do hot and cold embed nearby, and when does it bite? They're distributionally near-identical ("the soup is ", " weather"). It bites in any polarity-sensitive downstream use: sentiment lexicon induction, contradiction detection, retrieval where negation flips the answer. Contextual models mitigate but don't eliminate (negation remains a known weak spot through modern LLMs); the mitigation is task-specific supervision, not better unsupervised geometry.

References

  • Mikolov et al., Efficient Estimation of Word Representations in Vector Space (2013) — arXiv:1301.3781
  • Mikolov et al., Distributed Representations of Words and Phrases (2013) — arXiv:1310.4546 — negative sampling, subsampling
  • Levy & Goldberg, Neural Word Embedding as Implicit Matrix Factorization (NeurIPS 2014)
  • Levy, Goldberg & Dagan, Improving Distributional Similarity with Lessons Learned from Word Embeddings (TACL 2015) — the hyperparameters-matter paper
  • Pennington et al., GloVe (EMNLP 2014) — the count-based contemporary
  • Gonen & Goldberg, Lipstick on a Pig (2019) — debiasing's limits
  • Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), ch. 6 — the best textbook treatment
  • Illustrated word2vec — Jay Alammar