Vector Databases

Phase 9 · Document 04 · RAG Prev: 03 — Embeddings · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A vector database is where your embeddings (03) live and get searched — it answers "which chunks are nearest to this query vector?" in milliseconds over millions of vectors, which a brute-force scan can't. Picking and configuring it correctly determines your retrieval latency, recall, scalability, and cost, and — critically — whether you can do metadata filtering (ACL, freshness) at search time, which production RAG requires (01, 10). The two ideas you must own: approximate nearest-neighbor (ANN) search (the speed/recall trade-off that makes vector search feasible) and filtered search (combining "nearest" with "allowed/fresh"). Choose wrong and you get slow queries, missed results, or a leak.

2. Core Concept

Plain-English primer: nearest-neighbor at scale

Retrieval = find the chunk vectors nearest to the query vector (03). The naïve way — exact k-NN — compares the query to every vector (O(N) per query). At a few thousand vectors that's fine; at millions it's far too slow. So vector DBs build an index that finds the nearest neighbors approximately but fast — Approximate Nearest Neighbor (ANN) search. You trade a little recall (you might miss a true neighbor occasionally) for orders-of-magnitude speed.

EXACT k-NN: compare query to ALL N vectors → perfect recall, O(N), too slow at scale
ANN (HNSW/IVF/…): smart index → ~milliseconds over millions, ~95–99% recall (tunable)

A "vector database" = an ANN index + vector storage + metadata storage/filtering + the usual DB concerns (persistence, updates, scaling, backups).

The main ANN algorithms

• HNSW (Hierarchical Navigable Small World) — a multi-layer graph you greedily traverse toward the query. The default in most modern vector DBs: excellent recall/latency, supports incremental inserts. Knobs: M (graph connectivity), ef_construction (build quality), ef_search (search effort → higher = better recall, slower).
• IVF (Inverted File) — cluster vectors into buckets (nlist); search only the nearest nprobe buckets. Fast, memory-light, but needs training and is weaker on incremental updates. Often combined with PQ (Product Quantization) to compress vectors (huge memory savings at some recall cost) → IVF-PQ.
• ScaNN / DiskANN / others — Google's ScaNN (fast, quantization-aware); DiskANN (serve billions from SSD). Specialized scaling options.

The universal trade-off: recall ↔ latency ↔ memory. Tuning ef_search/nprobe moves you along the recall/speed curve; quantization (PQ/int8) trades memory for a little recall. Pick the point your SLO needs (Phase 7.08).

Metadata filtering (the production essential)

Real retrieval isn't just "nearest" — it's "nearest among allowed, fresh chunks": filter by acl/tenant (security, 10/Phase 8.09), updated_at (freshness), source/doc_type. Filtered ANN is subtle: naïvely filtering after the ANN search can return too few results (the top-k were filtered out); good DBs do filtered search that respects the filter during traversal (pre-filtering / filterable HNSW). Confirm your DB supports efficient metadata filters at scale — it's a common gap.

The database landscape

Decision drivers: scale (thousands vs billions), filtering needs (ACL!), hybrid support (05), managed-vs-self-host (Phase 5.02), and "do I already have Postgres?" (→ pgvector avoids a new system). For ≤ a few hundred-K vectors, even in-memory FAISS/NumPy is fine.

Configure it right (the gotchas)

• Set the distance metric to match the embedder (cosine usually; 03) — a mismatch silently mis-ranks.
• Tune ef_search/nprobe to your recall SLO; benchmark recall vs latency.
• Plan upserts/deletes for incremental ingestion (01) — HNSW handles inserts well; some IVF setups need rebuilds.
• Capacity: memory ≈ N × dim × bytes (+ index overhead); quantization (PQ/int8) cuts it. Size before you load (03 dimensionality).

3. Mental Model

   retrieval = nearest vectors to the query [03]
   EXACT k-NN O(N) (too slow at scale)  →  ANN INDEX: ~ms over millions, ~95–99% recall (tunable)
        HNSW (graph, default; knobs M/ef_construction/EF_SEARCH) · IVF(+PQ) (buckets nprobe; compress) · DiskANN(SSD)
        trade-off triangle: RECALL ↔ LATENCY ↔ MEMORY  → tune ef_search/nprobe + quantization to your SLO

   a VECTOR DB = ANN index + vector store + METADATA FILTERING + persistence/updates/scale
        ★ filtered search (acl/freshness) must respect the filter DURING traversal, not naïvely after
   set DISTANCE METRIC = embedder's (cosine) [03]; size memory ≈ N×dim×bytes

Mnemonic: ANN trades a little recall for huge speed (HNSW default; tune ef_search). A vector DB adds metadata-filtered search — and filtering must happen during the ANN traversal, not after. Match the metric.

4. Hitchhiker's Guide

What to look for first: does the DB do efficient metadata-filtered search (for ACL/freshness), and is the distance metric set to your embedder's? Then your scale tier (thousands → in-memory; millions → a real DB; billions → Milvus/DiskANN).

What to ignore at first: exotic indexes and billion-scale tuning. For a prototype, Chroma or even FAISS/NumPy is plenty; choose a production DB when scale/filtering/ops demand it.

What misleads beginners:

• Assuming ANN = exact. It's approximate — recall < 100% and tunable; if recall matters, raise ef_search/nprobe and measure (09).
• Filtering after search. Post-filtering the ANN results can drop you below k — use the DB's filtered search.
• Metric mismatch. DB on Euclidean while the embedder wants cosine (un-normalized) silently mis-ranks (03).
• Ignoring memory. N × dim × 4 bytes adds up fast; quantize or you OOM at scale.
• New system for no reason. If you're on Postgres, pgvector avoids operating a separate DB.

How experts reason: they pick the DB by scale × filtering × hybrid × ops, set the correct metric, tune ef_search/nprobe to a recall SLO, use quantization at scale for memory, ensure filtered search for ACL/freshness, and plan upserts/deletes for incremental ingestion. They benchmark recall@k and p95 latency on their own data, not vendor claims.

What matters in production: recall@k at target latency, filtered search correctness (ACL never leaks, 10), memory/cost at scale, incremental update support, and HA/persistence/backups (it's a database).

How to debug/verify: measure recall@k vs an exact baseline on a sample; sweep ef_search/nprobe to map the recall/latency curve; verify a filtered query never returns out-of-ACL docs; check memory vs N×dim.

Questions to ask vendors: ANN algorithm + tunables? efficient pre-filtered (metadata) search? hybrid/sparse support (05)? metrics supported? upsert/delete + incremental? scale ceiling + memory/quantization? managed vs self-host?

What silently gets expensive/unreliable: low recall from un-tuned ANN, post-filtering returning too few results, metric mismatch, unbounded memory at scale, and ACL filters that aren't actually enforced in the index (Phase 8.09).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• ANN search trades a little recall for huge speed — exact k-NN is O(N) and too slow at scale.
• HNSW is the default index (great recall/latency, incremental inserts); IVF(+PQ) is memory-light and scales; DiskANN serves from SSD at billion scale.
• Tune ef_search/nprobe to a recall SLO; the trade-off triangle is recall ↔ latency ↔ memory (Phase 7.08).
• A vector DB = ANN index + vector storage + metadata filtering + DB concerns (persistence, updates, scale).
• Filtered search must respect the filter during traversal — naïve post-filtering can return fewer than k.
• Set the distance metric to the embedder's (usually cosine) — mismatch mis-ranks (03).
• Memory ≈ N × dim × bytes (+ index) — quantization cuts it; size before loading.
• Choose by scale × filtering × hybrid × ops; pgvector avoids a new system if you're on Postgres.

9. Observations from Real Systems

• Chroma is the common dev/prototype default; Qdrant/Weaviate/Milvus for production; pgvector when teams want to stay on Postgres (Phase 5.02).
• Most modern DBs default to HNSW; Milvus/FAISS expose IVF/PQ/DiskANN for billion-scale.
• Metadata-filtered search is where ACL-aware retrieval lives — the make-or-break feature for enterprise RAG (10, Phase 8.09).
• Recall is silently below 100% unless tuned — teams that don't measure recall@k ship worse retrieval than they think (09).
• At small scale, in-memory FAISS/NumPy beats standing up a DB — don't over-engineer early.

10. Common Misconceptions

11. Engineering Decision Framework

CHOOSE + CONFIGURE a vector store:
 1. SCALE: ≤~100k → in-memory FAISS/NumPy/Chroma · millions → Qdrant/Weaviate/pgvector · billions → Milvus/DiskANN.
 2. FILTERING: need ACL/freshness? → DB with efficient PRE-FILTERED search (must enforce ACL) [10,8.09].
 3. HYBRID: need keyword too? → DB with sparse/BM25 support (Qdrant/Weaviate/pgvector) [05].
 4. OPS: managed (Pinecone) vs self-host; already on Postgres? → pgvector (no new system) [5.02].
 5. CONFIGURE: metric = embedder's (cosine) [03]; tune ef_search/nprobe to a RECALL SLO; quantize at scale.
 6. UPDATES: ensure upsert/delete for incremental ingestion [01]; HA/persistence/backups.
 7. BENCHMARK recall@k + p95 latency on YOUR data [09].

12. Hands-On Lab

Goal

Build retrieval over a real vector store, measure ANN recall vs an exact baseline, tune the recall/latency knob, and prove metadata-filtered (ACL) search.

Prerequisites

• The embedded chunks from 03; pip install qdrant-client numpy (or chromadb, or pgvector).

Steps

1. Exact baseline: with NumPy, compute exact top-k (cosine) for ~10 labeled queries — the ground-truth neighbors.
2. Index in a DB: load the same vectors into Qdrant/Chroma with the metric set to cosine (03); attach acl + updated_at metadata from 01.
3. ANN recall: retrieve top-k via the DB; compute recall@k vs the exact baseline. Then sweep ef_search (or nprobe) low→high and plot recall vs latency — see the trade-off.
4. Filtered search (ACL): issue a query with a metadata filter (e.g., acl contains "team:research"); verify results are nearest and all within the allowed ACL — and that out-of-ACL docs never appear (10).
5. Freshness filter: filter updated_at >= cutoff; confirm stale docs are excluded.
6. (Scale/memory) estimate memory ≈ N × dim × 4 bytes; if available, enable int8/PQ quantization and re-check recall + memory.

Expected output

A recall@k-vs-latency curve for the ANN index, plus a verified ACL/freshness filtered search — concretely demonstrating the approximate nature and the production filtering requirement.

Debugging tips

• Recall low → raise ef_search/nprobe, or metric mismatch (not cosine) (03).
• Filter returns too few → DB is post-filtering; use its pre-filter/filterable-index mode.

Extension task

Compare two DBs (e.g., Chroma vs Qdrant) on recall@k, p95 latency, and filtering ergonomics for your corpus.

Production extension

Wire incremental upsert/delete from ingestion (01), enforce ACL filters as a hard constraint, and add recall@k + p95 latency to your dashboards (Phase 7.08).

What to measure

recall@k vs exact; recall–latency curve (ef_search/nprobe); filtered-search correctness (ACL/freshness); memory vs N×dim; quantization effect.

Deliverables

• A recall@k-vs-latency curve for your ANN index.
• A verified ACL + freshness filtered search.
• A DB choice + metric/tuning note justified by the data.

13. Verification Questions

Basic

1. Why use ANN instead of exact k-NN at scale, and what's the trade-off?
2. What is HNSW, and what knob trades recall for latency?
3. What does a vector DB add beyond an ANN index?

Applied 4. Choose a vector DB for (a) a prototype, (b) Postgres-based prod with ACL, (c) billion-scale. Justify. 5. Why must metadata filtering happen during traversal, not after?

Debugging 6. Recall is poor despite good embeddings. Two causes and fixes. 7. A filtered query returns fewer than k results. What's wrong?

System design 8. Design vector storage for a multi-tenant RAG product with ACL-filtered search and incremental updates.

Startup / product 9. How do index choice, recall tuning, and filtered search affect your latency SLO, cost, and security posture?

14. Takeaways

1. Retrieval is nearest-neighbor; ANN makes it fast by trading a little recall — HNSW is the default, tune ef_search.
2. The trade-off is recall ↔ latency ↔ memory; quantization buys memory at scale.
3. A vector DB = ANN index + storage + metadata filtering + DB ops; filtered search must respect the filter during traversal.
4. Set the metric to the embedder's (cosine); size memory ≈ N×dim×bytes.
5. Choose by scale × filtering × hybrid × ops (pgvector if already on Postgres); benchmark recall@k + p95 on your data.

15. Artifact Checklist

• A recall@k-vs-latency curve (ANN vs exact baseline).
• A verified ACL + freshness filtered search.
• Correct distance metric matching the embedder.
• A memory estimate (+ optional quantization result).
• A DB choice justified by scale × filtering × hybrid × ops.

Up: Phase 9 Index · Next: 05 — Hybrid Search