Brother Talk — RAG & Vector Search

Off the record. The stuff I'd tell you over a beer, not in a design review.

RAG is the most over-demoed, under-engineered thing in AI right now. Anybody can wire up "embed some PDFs, stuff the top-5 into a prompt" in an afternoon and demo it to applause. Then it hits real data and real users and it's confidently wrong half the time, and nobody on the team knows why — because they never separated the retriever from the generator. The engineer who can walk in and say "recall@10 is 0.4, that's a retrieval problem, here's the chunking fix" is worth ten people who can build the demo. Be that person. The demo is the trap; the evaluation is the job.

Almost every RAG bug is a retrieval bug, and almost everyone blames the model. I promise you'll sit in a room where smart people are tweaking the prompt and swapping to a bigger LLM because "the answers are bad," and the actual problem is that the relevant chunk was never in the top-50. The reflex that makes you senior: before you touch the generator, compute recall against the labelled set. If the evidence wasn't retrieved, no prompt on earth saves you. Measure first, argue second.

Don't fall for the vector-DB shopping spree. There's a whole industry trying to sell you that the vector database is the hard, important choice. It mostly isn't — they all wrap the same FAISS-class algorithms. The hard part is chunking, embedding choice, reranking, and eval. I've watched a team burn a month migrating databases to fix a recall problem that was actually 500-token chunks blurring the facts. Pick pgvector if you already run Postgres, pick a managed DB if you hate ops, and spend your real energy on retrieval quality.

The recall/latency/memory triangle is the cheat code. Once you internalize that you can't max all three — that nprobe/ef is a recall dial, that PQ is a memory dial, that flat is exact-but-slow — you stop being mystified by ANN and start tuning it. Most people treat the vector index as a magic box; you'll treat it as three knobs with a known trade. That framing alone makes you sound like you've shipped this before.

Embedding skew will get you, and it's silent. The single nastiest production failure in this space: someone upgrades the embedding model and forgets to re-embed the corpus. No exception, no alert — recall just quietly dies and the answers get mysteriously worse. Pin your embedding version like it's a database migration, because it is one. Tattoo this: change the embedder → re-embed everything.

RAG vs fine-tuning is a question people get backwards constantly. Fine-tuning is for teaching the model a skill, format, or style it doesn't have. RAG is for giving it facts — especially facts that change or need a citation. People keep trying to fine-tune facts into a model: it's expensive, it goes stale the day you ship, and it still hallucinates with no source. When someone says "let's fine-tune so it knows our docs," that's usually a RAG problem wearing the wrong hat.

Reranking is the highest-leverage thing nobody bothers to add. A two-stage retrieve-then-rerank pipeline — cast a wide net with fast ANN, then re-score the top-50 with a cross-encoder — is the single biggest quality jump available, and it's a few lines of code. Teams ship raw top-5 from the bi-encoder, watch the answers be mediocre, and conclude "RAG doesn't work for us." It works; they skipped the part that makes it good. Add the reranker. And add MMR so you're not feeding the model five copies of one sentence.

What's actually worth caring about: retrieval evaluation (recall@k / nDCG), good chunking, a reranker, and treating retrieved text as untrusted. What's not worth losing sleep over: squeezing the last 1% of ANN recall, the exact vector DB brand, or whether your distance metric is cosine or normalized-dot (they're the same once you normalize).

The career framing. This is the applied-AI system-design phase — the one that comes up in every interview and every real product, because every company wants their LLM to answer over their own data. The differentiator isn't that you can build a RAG pipeline; everyone claims that. It's that you can debug one — name the failure mode, measure the right half, and fix the retriever. Get this cold and you'll be the person teams call when their RAG "just doesn't work." Now go make pytest green and watch that recall climb.