Embeddings

Phase 9 · Document 03 · RAG Prev: 02 — Chunking · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Embeddings are the engine of semantic retrieval — they turn text into numbers such that similar meaning → nearby vectors, which is what lets RAG find the right chunk even when the query uses different words than the document ("how do I get my money back?" → a chunk about "refund policy"). The embedding model you choose, and how you use it, directly sets your retrieval recall — and retrieval dominates RAG quality (00). Picking the wrong embedder, mismatching query vs document handling, ignoring the model's max input, or comparing vectors with the wrong metric are silent quality killers. Embeddings also power clustering, dedup, classification, and recommendations — so this is foundational well beyond RAG.

2. Core Concept

Plain-English primer: text → a point in meaning-space

An embedding is a fixed-length list of numbers (a vector, e.g. 384–3072 floats) that represents the meaning of a piece of text. An embedding model is a neural network trained so that texts with similar meaning get vectors that are close together, and unrelated texts get vectors far apart. Think of it as placing every chunk as a point in a high-dimensional "meaning-space": "refund policy", "return my purchase", and "get my money back" land near each other; "GPU memory" lands far away.

Retrieval is then just geometry: embed the query, find the document points nearest to it.

emb = embed(["refund policy", "how do I get my money back?", "GPU memory bandwidth"])
# emb[0] and emb[1] are CLOSE (same meaning); emb[2] is FAR

How "closeness" is measured (the metric matters)

The standard similarity is cosine similarity — the cosine of the angle between two vectors (it ignores magnitude, focusing on direction/meaning):

import numpy as np
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))   # 1 = identical, 0 = unrelated, -1 = opposite

Many models output normalized vectors (unit length), in which case cosine = dot product and Euclidean distance is monotonic with it — so they rank the same. Use the metric the model was trained for (almost always cosine); using the wrong metric quietly degrades ranking. Your vector DB must be configured for the right space (04).

Dimensions and the quality/cost trade-off

The vector length is the dimensionality (e.g., 384, 768, 1536, 3072). More dimensions can capture more nuance but cost more storage, memory, and slower search (04). Some models support Matryoshka / dimension truncation — you can shorten the vector (e.g., 3072→512) for cheaper search at a small quality cost. Dimensionality is a real knob, not just a spec.

Symmetric vs asymmetric (query ≠ document)

A subtle but important property: in RAG, the query ("how do I return something?") and the document ("Our return policy permits returns within 30 days…") are different kinds of text. Asymmetric retrieval models are trained for exactly this (short question ↔ longer passage), often via instruction prefixes — e.g., prepend "query: " to questions and "passage: " to documents (E5, BGE, Nomic models require this). Using the wrong prefix, or none, measurably hurts recall. Symmetric models (for similarity/dedup) treat both sides the same. Know which your model is and follow its prompt convention.

Choosing an embedding model

Decisions: API vs self-hosted (privacy/cost/control — Phase 5.01, Phase 6), multilingual needs, max input (must fit your chunk size, 02), dimensionality, and domain fit (code/legal/bio). Compare candidates on MTEB (the Massive Text Embedding Benchmark leaderboard) and on your data — leaderboard ≠ your domain.

Operational facts

• Same model for index and query. You must embed documents and queries with the same model (and version) — different models live in different, incompatible spaces. Changing the embedder means re-embedding the whole corpus (01 incremental).
• Max input. Inputs longer than the model's limit are truncated — your chunks must fit (02).
• Batch for throughput/cost; cache embeddings (they're deterministic per text+model).
• Cost is per token, but embeddings are far cheaper than generation; the index is computed once + on changes.

3. Mental Model

   text → EMBEDDING MODEL → vector (point in high-dim "meaning-space")
        similar meaning → NEARBY points;  retrieval = embed query, find NEAREST chunks (geometry)
        closeness = COSINE similarity (angle); normalized → cosine = dot product [04 metric]

   KNOBS:
     dimensionality (384…3072): more nuance vs more storage/slower (Matryoshka truncation)
     symmetric vs ASYMMETRIC: query↔passage need the right INSTRUCTION PREFIX (E5/BGE/Nomic)
     model choice: API vs self-host · multilingual · MAX INPUT (fit chunk [02]) · domain · MTEB + YOUR data

   ★ same model+version for index AND query; changing it = RE-EMBED the corpus

Mnemonic: embeddings place text as points where meaning = nearness; retrieval is nearest-neighbor by cosine. Match the metric, the prefix (asymmetric), and the model+version across index and query.

4. Hitchhiker's Guide

What to look for first: the model's max input (must fit your chunks), whether it's asymmetric (needs query/passage prefixes), and the similarity metric (set the DB to it). Then pick a strong default (text-embedding-3-small/BGE) and validate on your data.

What to ignore at first: chasing the #1 MTEB model and exotic dimensions. A solid default + correct usage beats a "better" model used wrong.

What misleads beginners:

• Ignoring asymmetric prefixes. Forgetting query:/passage: on E5/BGE-style models silently tanks recall.
• Wrong similarity metric. Using Euclidean when the model wants cosine (on un-normalized vectors) mis-ranks (04).
• Mixing models/versions. Documents embedded with model A can't be searched with model B — different spaces.
• Chunks exceeding max input. Silent truncation → embeddings miss the tail of the chunk (02).
• Trusting the leaderboard over your data. MTEB is a guide; your domain may rank differently.

How experts reason: they pick an embedder by domain fit + max input + multilingual + API-vs-self-host, follow the model's prompt/prefix convention, set the correct metric in the DB, normalize if recommended, and benchmark candidates on their own labeled queries (recall@k) — not just MTEB. They plan for re-embedding if they ever switch models.

What matters in production: retrieval recall (the embedder's real KPI), correct query/passage handling, metric consistency, embedding cost/latency at ingest and query time, and a migration plan if the model changes.

How to debug/verify: if semantic recall is poor, check (1) asymmetric prefixes applied, (2) metric matches the model, (3) chunks within max input, (4) same model index+query; then A/B two embedders on your labeled set (09).

Questions to ask: symmetric or asymmetric (prefixes)? max input tokens? dimensionality (truncatable)? recommended metric/normalization? multilingual? API or open-weight? MTEB and domain performance?

What silently gets expensive/unreliable: missing prefixes, metric mismatch, model/version drift between index and query, oversized chunks (truncation), and over-large dimensions (storage/latency).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• An embedding maps text to a vector where similar meaning = nearby; retrieval is nearest-neighbor in that space.
• Cosine similarity is the standard metric; for normalized vectors, cosine = dot product — set the DB to the model's metric (04).
• Asymmetric models need query/passage prefixes (E5/BGE/Nomic) — missing them hurts recall.
• Use the same model + version for index and query — different models are incompatible spaces; switching means re-embedding (01).
• Chunks must fit the model's max input or they're truncated (02).
• Dimensionality trades quality vs storage/latency; some models allow truncation (Matryoshka).
• Pick by domain fit + multilingual + max input + API-vs-self-host; validate on MTEB and your data.
• Embeddings are cheap and cacheable/deterministic (per text+model) — batch and cache.

9. Observations from Real Systems

• text-embedding-3-small is a common production default (cheap, strong, dim-truncatable); BGE/GTE/E5 dominate open-weight (Phase 6).
• The classic open-model bug is forgetting the query:/passage: prefix — recall jumps once added.
• Self-hosted embeddings are common for privacy/cost (embed sensitive data in-house, Phase 8.09).
• Switching embedders triggers a full re-embed — teams pin the model+version and treat changes as migrations (01).
• MTEB guides selection but teams confirm on their own labeled queries — domain rank often differs from the leaderboard (09).

10. Common Misconceptions

11. Engineering Decision Framework

CHOOSE + USE an embedder:
 1. REQUIREMENTS: domain (general/code/legal), multilingual?, privacy (API vs self-host [5.01/6]),
    max input ≥ chunk size [02], dimensionality budget [04].
 2. SHORTLIST via MTEB (retrieval tasks) → VALIDATE on YOUR labeled queries (recall@k) [09].
 3. USE CORRECTLY: apply query/passage PREFIXES if asymmetric; set DB metric to the model's (cosine);
    normalize if recommended.
 4. CONSISTENCY: same model+version for index AND query; pin it (switching = re-embed).
 5. OPERATE: batch + cache embeddings; mind cost/latency at ingest and query.

12. Hands-On Lab

Goal

Build intuition + a real comparison: see semantic nearness, then A/B two embedders (and correct usage) on a labeled query set.

Prerequisites

• pip install numpy sentence-transformers openai; the chunks from 02; ~10 labeled queries (known answering chunk).

Steps

1. See the space: embed ["refund policy", "how do I get my money back?", "GPU memory bandwidth"]; print pairwise cosine; confirm the first two are close, the third far.
2. Index the corpus: embed your chunks with embedder A (e.g., text-embedding-3-small); store vectors (+ keep the text) — a simple NumPy top-k is fine for the lab.
3. Retrieve + recall@k: for each labeled query, return top-k by cosine; record whether the gold chunk is in the top-k (recall@k).
4. A/B a second embedder (e.g., a BGE/E5 model) — with the correct query/passage prefixes — re-index, re-measure recall@k. Then deliberately drop the prefixes and show recall falls (the asymmetric lesson).
5. Metric check: compare ranking with cosine vs raw Euclidean on un-normalized vectors; note differences (and that normalized → identical).
6. Dimensionality: if using a truncatable model, truncate 3072→512 and measure the recall vs storage trade-off.

Expected output

A recall@k comparison across embedders + correct/incorrect prefix usage, plus a metric and dimensionality observation — concretely showing what drives retrieval quality.

Debugging tips

• Recall low for an open model → missing query/passage prefix.
• Rankings look random → metric mismatch or you embedded query and docs with different models.

Extension task

Add a multilingual query against English docs with a multilingual embedder and confirm cross-lingual retrieval works.

Production extension

Wire the chosen embedder into ingestion (01) + a real vector DB with the correct metric (04); cache embeddings and plan a re-embed migration path.

What to measure

recall@k per embedder; prefix on/off effect; cosine vs Euclidean ranking; dimensionality recall/storage trade-off; embedding latency/cost.

Deliverables

• A cosine-similarity demonstration of meaning-space.
• A two-embedder recall@k comparison on your labeled queries.
• A prefix on/off result (asymmetric lesson) + a metric/dimension note.

13. Verification Questions

Basic

1. What is an embedding, and what property makes retrieval work?
2. What metric ranks embeddings, and when does cosine equal dot product?
3. Why must index and query use the same model+version?

Applied 4. Your open embedder underperforms. What usage detail (E5/BGE) is the likely culprit? 5. When would you truncate dimensionality, and what's the trade-off?

Debugging 6. Semantic recall is poor. List four things to check. 7. You switched embedding models and retrieval broke. Why, and the fix?

System design 8. Choose + configure an embedder for a private, multilingual, code+prose corpus; justify metric, prefixes, dims, hosting.

Startup / product 9. Why does the embedding choice + correct usage have outsized impact on RAG quality and cost?

14. Takeaways

1. Embeddings place text as points where similar meaning is nearby; retrieval is nearest-neighbor by cosine.
2. Match the metric (usually cosine; normalized → dot) and the asymmetric prefixes (query/passage).
3. Use the same model+version for index and query — switching means re-embedding.
4. Chunks must fit max input; dimensionality trades quality vs cost (truncatable in some models).
5. Choose by domain + multilingual + hosting + max input, validated on MTEB and your data.

15. Artifact Checklist

• A cosine-similarity demo of meaning-space.
• A two-embedder recall@k comparison on labeled queries.
• A prefix on/off (asymmetric) result.
• A metric and dimensionality note.
• A chosen embedder + a re-embed migration plan.

Up: Phase 9 Index · Next: 04 — Vector Databases