Token Embeddings and Positional Encoding

Phase 2 · Document 01 · Transformer Foundations Prev: 00 — The Transformer Big Picture · Next: 02 — Attention and Self-Attention

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Embeddings are where discrete text becomes continuous math the network can work with — and the same idea powers two things you'll use constantly: the input layer of every LLM, and the embedding models behind RAG and semantic search (Phase 9). Positional encoding — especially RoPE — is the unglamorous component that decides whether a model's long-context claims hold up and whether context extension works. Understanding both lets you reason about why two texts are "similar," why context windows can sometimes be stretched, and why a model degrades past its trained length.

2. Core Concept

Token embeddings

After tokenization (Phase 1.01), each token ID indexes a learned embedding matrix of shape (vocab_size, hidden_dim). Row i is the vector for token i.

token_id 9906 → embedding_table[9906] → [0.13, -0.44, ..., 0.02]   (hidden_dim floats)

This turns a meaningless integer into a dense vector positioned in a high-dimensional space where geometry encodes meaning: tokens used in similar contexts during training end up near each other. The vectors are learned — training nudges them so that next-token prediction works.

The same mechanism, taken to the sentence/document level by a dedicated embedding model, produces one vector per text whose cosine similarity measures semantic closeness — the retrieval engine of RAG. (An LLM's internal token embeddings and a standalone embedding model are related ideas but different artifacts; see Phase 1.04.)

Positional encoding

Self-attention is permutation-invariant — on its own it can't tell "dog bites man" from "man bites dog." Position must be injected.

• Absolute positional encodings (original transformer): add a position-dependent vector to each embedding. Simple, but extends poorly past trained length.
• RoPE (Rotary Position Embedding) — the modern standard (Llama, Qwen, Mistral, etc.). Instead of adding a vector, it rotates the Query and Key vectors by an angle proportional to position, so attention naturally depends on relative distance between tokens. This generalizes better and enables context extension tricks.
• ALiBi — adds a distance-based bias to attention scores; another relative scheme favoring extrapolation.

Context extension

Because RoPE encodes position as rotation frequency, you can rescale those frequencies (e.g. position interpolation, YaRN, NTK-aware scaling) to make a model accept longer contexts than it was trained on — usually with some quality cost and often best after light fine-tuning. This is why you sometimes see "8K model extended to 32K," and why such claims need verification (Phase 0.02).

3. Mental Model

WORD → ID → POINT IN SPACE
  "king", "queen", "prince" land near each other (learned geometry)
  similarity(a,b) = cosine(vec_a, vec_b)            ← the RAG primitive

POSITION = ORIENTATION, not location
  RoPE rotates Q,K by an angle ∝ position
  attention(i,j) ends up depending on (i − j): RELATIVE distance
  rescale the rotation frequencies → accept longer context (with care)

Two embedding worlds:
  • token embeddings  = input layer INSIDE an LLM
  • embedding models  = whole-text vectors for SEARCH/RAG (same idea, different artifact)

4. Hitchhiker's Guide

What to understand first: embeddings = learned vectors where distance ≈ meaning; position is injected separately (usually RoPE).

What to ignore at first: the trigonometric derivation of RoPE. Know what it buys you (relative position, extension) before the math.

What misleads beginners:

• Confusing an LLM's internal token embeddings with a standalone embedding model for RAG.
• Assuming you can set any context length you like — exceeding the trained/extended length degrades output.
• Thinking embedding similarity is "truth" — it reflects training distribution and the specific model.

How experts reason: for RAG they pick an embedding model deliberately and use the same model for queries and documents; for long context they check whether the model uses RoPE and whether the provider applied a real extension method (vs just raising the limit).

What matters in production: embedding model choice (dimension, domain fit, cost) for retrieval; position-encoding type and trained length for long-context reliability.

How to verify: embed similar vs unrelated texts and check cosine gap; for long context, run a recall test at increasing lengths to find where quality falls off.

Questions to ask: What position encoding does this model use? What's its trained context length vs the advertised one? For an embedding model: what dimension, what training domain, what max input?

What silently breaks: mixing two embedding models for queries vs docs (retrieval collapses); trusting an "extended" context that wasn't properly fine-tuned; assuming cross-lingual similarity without a multilingual embedding model.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Token embeddings are learned; geometry (distance/direction) encodes meaning from training.
• The embedding matrix is part of the model's parameters: vocab_size × hidden_dim values.
• Self-attention is permutation-invariant — position must be injected (RoPE/ALiBi/absolute).
• RoPE encodes relative position by rotating Q and K and is the modern default.
• Context can be extended by rescaling RoPE frequencies (position interpolation, YaRN) — usually with quality cost and best with fine-tuning.
• For RAG, always embed queries and documents with the same embedding model.
• Cosine similarity is the standard semantic-closeness measure for embeddings.
• An LLM's internal token embeddings ≠ a standalone embedding model (related idea, different artifact).

9. Observations from Real Systems

• Llama, Qwen, Mistral, Gemma model cards specify RoPE and often the rope-scaling config used for long-context variants.
• OpenAI/Cohere/BGE embedding models publish a fixed output dimension — the vector size you store in a vector DB (Phase 9.03).
• vLLM / llama.cpp expose rope-scaling parameters so operators can serve extended-context variants.
• Vector databases (Qdrant, Chroma, pgvector) index embedding vectors and rank by cosine/dot product — the direct application of Section 2.
• "8K → 32K extended" releases demonstrate context extension in the wild — and why recall must be re-tested at length.

10. Common Misconceptions

11. Engineering Decision Framework

Choosing an embedding model (RAG/search):
  domain fit (general vs code vs multilingual) → dimension (storage/latency) →
  max input length → cost → MEASURE retrieval quality (Phase 9/13).
  Rule: same model for queries AND documents.

Long-context decision:
  Need more context than trained length?
    → check position encoding (RoPE?) and whether a real extension (YaRN/PI) was applied.
    → re-test RECALL at target length; don't trust the headline number.
    → if recall fails → use RAG to bring only relevant tokens into context instead.

12. Hands-On Lab

Goal

See meaning-as-geometry directly: embed words/sentences, measure cosine similarity, and observe how a tokenizer+embedding turns text into vectors.

Prerequisites

• Python 3.10+; an embedding API key OR sentence-transformers for local embeddings.

Setup

pip install numpy openai            # or: pip install sentence-transformers

Steps

import numpy as np
from openai import OpenAI
client = OpenAI()

def embed(t):
    return np.array(client.embeddings.create(model="text-embedding-3-small", input=t).data[0].embedding)

cos = lambda a,b: float(a@b/(np.linalg.norm(a)*np.linalg.norm(b)))

words = ["king","queen","man","woman","banana"]
vecs = {w: embed(w) for w in words}
print("king~queen:", round(cos(vecs['king'],vecs['queen']),3))
print("king~banana:", round(cos(vecs['king'],vecs['banana']),3))

# Sentence-level (the RAG primitive)
q = embed("How do I reset my password?")
docs = ["Steps to recover account access","Our refund policy","Reset credentials guide"]
for d in docs:
    print(round(cos(q, embed(d)),3), d)

Expected output

• king~queen clearly higher than king~banana.
• The password query ranks the credentials/recovery docs above the refund doc.

Debugging tips

• All similarities ~equal? You may be normalizing wrong or using a tiny/!matched model.
• Cross-lingual fails? Use a multilingual embedding model.

Extension task

Try the analogy king − man + woman and find its nearest word among candidates — a classic embedding-geometry demo.

Production extension

Embed 100 short docs, store vectors in a list, and build a top_k(query, k) retriever ranking by cosine — the seed of your Phase 9 RAG index.

What to measure

Cosine gaps (related vs unrelated); retrieval ranking correctness; embedding dimension and latency.

Deliverables

• A similarity table for related vs unrelated terms.
• A working top_k cosine retriever over ≥20 docs.
• A note: which embedding model/dimension you'd choose for a real RAG and why.

13. Verification Questions

Basic

1. What does an embedding turn a token ID into, and why?
2. Why does a transformer need positional encoding at all?
3. What does RoPE encode, and why is "relative" useful?

Applied 4. For RAG, why must queries and documents share an embedding model? 5. You need 32K context from an 8K-trained model. What do you check and test?

Debugging 6. Retrieval quality collapsed after you "upgraded" the query embedding model. What happened? 7. A long-context model gives great short answers but misses facts deep in long inputs. Likely cause and mitigation?

System design 8. Design the embedding layer of a multilingual document-search product: model choice, dimension, storage, and eval.

Startup / product 9. Your RAG costs balloon as documents grow. Explain how embeddings + retrieval keep cost bounded vs stuffing everything into a long context.

14. Takeaways

1. Embeddings turn token IDs into learned vectors where geometry encodes meaning.
2. The same idea powers RAG via standalone embedding models and cosine similarity.
3. Attention is order-blind; position is injected — modern models use RoPE (relative).
4. Context can be extended by rescaling RoPE frequencies, but verify recall.
5. Same embedding model for queries and documents, always.
6. Prefer retrieval over blindly extending context for very long inputs.

15. Artifact Checklist

• Code: embedding similarity + top_k cosine retriever.
• Similarity table (related vs unrelated).
• Notes: RoPE/relative-position and context-extension trade-offs.
• Decision record: embedding model choice for a real RAG use case.
• Recall test at increasing context lengths (if exploring long context).
• Diagram: "meaning as geometry" + position-as-rotation sketch.

Next: 02 — Attention and Self-Attention