« Phase 06 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a retrieval request
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy the retrieval mechanics, build everything that encodes a decision. Chunking encodes a decision about your documents. Topology encodes a decision about your risk. The grounding threshold encodes a decision about what you will defend.

One specific warning: it is tempting to adopt a framework's default RAG chain end to end. It will work in a demo and it will have no namespaces, no classification filter, no freshness contract and no citations. Those are the four things a bank actually needs, and they are exactly the four the default does not have.

2. A decision framework for a retrieval request

A team wants to add a corpus. Seven questions, in order:

1. Whose data is it? One tenant, several, or genuinely shared? This decides the namespace, and it decides it before anything else.
2. What is its classification, and does a barrier apply? If a barrier applies, the tenant boundary is too coarse and you need a finer namespace.
3. What is the freshness contract? How stale may an answer be before it is wrong rather than merely old? If nobody can answer, the answer is that there is no contract and answers will be confidently superseded.
4. What does a citation look like to a reviewer? If the source has no stable addressing — no document id, no version, no section — that is a data problem to fix at ingestion, not a retrieval problem to work around.
5. What are the ten questions people will actually ask? That is your golden set, and it takes an afternoon. Without it, every subsequent decision is taste.
6. Does it need lexical retrieval? If the corpus contains identifiers people search by — references, LEIs, product codes — yes, and dense-only will disappoint in a way that is hard to diagnose.
7. Who owns it, and what happens when the source is deleted? Deletion propagation and erasure requests are cheap at ingestion and expensive later.

If the answer to (1) is "shared" and to (2) is "confidential", stop and ask again. Those two are rarely both true, and the combination is how a pooled index ends up with someone's customer data.

3. Review red flags

In a design document

• One index, tenant as a metadata filter.
• Any mention of filtering after search.
• Chunk size stated with no mention of document structure.
• No doc_version on chunks.
• No freshness contract.
• No citations in the answer contract.
• A weighted score fusion with a hand-tuned constant.
• Reranking with no candidate limit.
• No relevance floor, and therefore no way to say "nothing relevant".
• "We'll evaluate it once it's built."
• No plan for changing the embedding model.
• Traces and evaluations pooled across tenants ("it's just telemetry").
• Information barriers described in prose rather than as a retrieval constraint.

In code

# Red flag: the tenant boundary as a WHERE clause
results = index.search(q, top_k=100)
return [r for r in results if r.tenant == user.tenant]     # the cliff, and the leak

# Red flag: fixed-size chunking
chunks = [text[i:i+2000] for i in range(0, len(text), 2000)]

# Red flag: unsigned feature hashing / unnormalized vectors mixed with normalized
vector[hash(tok) % d] += 1

# Red flag: unclamped IDF
idf = math.log(n / df)                    # negative for common terms

# Red flag: magic fusion weight
score = 0.6 * norm(bm25) + 0.4 * cosine   # where did 0.6 come from?

# Red flag: reranking everything
scores = [cross_encoder(q, c) for c in all_chunks]

# Red flag: no floor
return ranked[:5]                          # always returns 5, even for nonsense

# Red flag: budgeting the pieces
if sum(len(p) for p in parts) < budget:    # headers and separators are tokens too

# Red flag: silent drop
context = "\n".join(chunks[:3])            # what happened to 4 and 5?

In an incident review

• "The answers got worse after we added the filter" → the filtering cliff.
• "It cited a policy that had been replaced" → no freshness contract, no doc_version.
• "We can't tell what the agent saw" → retrieved chunk ids are not in the trace.
• "Only tenant X complains about quality" → per-tenant recall, not aggregate.

4. Production war stories

The answer from the wrong desk. One index, tenant as a post-filter. A refactor moved the filter up a call stack. No error rate moved, hit rate was unchanged, and a Legal user got a Payments incident report inside a summary. Found by a human, months later, because a 200 OK with a plausible answer is invisible to every monitor you have.

The week spent blaming the embedding model. ANN returned 100 neighbours across 10 million chunks; the tenant filter kept 3; classification and barrier filters kept none. The team tried three embedding models before someone drew the funnel. It was the filtering cliff, and the fix was a namespace.

The half table. Fixed-size chunking split a fee schedule between its header row and its data rows. The agent read amounts with no idea which column they belonged to and answered confidently. Structure-aware chunking fixed it in an afternoon; the incident took a week to understand.

The negative IDF. "Account" appeared in 80% of the corpus. Unclamped IDF made its contribution negative, so documents containing the user's own search term ranked lower than those without it. Undetected for a quarter, because the results were still plausible — just subtly worse.

The magic weight. 0.6 * bm25_normalized + 0.4 * cosine. Nobody could say where 0.6 came from. An embedding model upgrade degraded retrieval; the weight was hand-tuned again; six months later it was tuned a third time. Moving to RRF ended the cycle.

"I don't know" was unreachable. No relevance floor. Asked about a product the bank does not offer, the retriever returned its nearest neighbours — three unrelated policy chunks — and the model answered from them, fluently.

The telemetry that was customer data. Traces pooled across tenants "because it's observability". They contained prompts, retrieved chunks and answers, which means they inherited the classification of the most sensitive thing each request touched. The finding was substantial and entirely avoidable.

The migration nobody planned. A better embedding model shipped, everyone wanted it, and there was no runbook. Re-embedding 12 million chunks took a quarter, during which the index was frozen and two other projects waited.

5. The interview signal

Signal 1 — you say "retrieval must be authorized, not merely relevant." Unprompted. It reframes retrieval from an IR problem to a security problem, and it is the sentence that most distinguishes someone who has run this in a regulated environment.

Signal 2 — you name the no-detection property. "It's the only failure in the platform that produces a 200 OK." Then the conclusion: controls that fail silently and severely deserve prevention, not detection.

Signal 3 — you connect the filtering cliff to the isolation decision. The performance argument and the security argument point at the same fix. Very few candidates notice this, and it is a genuinely useful thing to have in your pocket during a design review.

Signal 4 — you explain why hybrid works. Not "more retrievers is better", but "they fail on complementary inputs, and in a bank the identifier case is most of what users ask about."

Signal 5 — you refuse score fusion and can say why. Incomparable scales, corpus-specific normalization, invalidated by a model change. RRF reads positions.

Signal 6 — you treat an embedding-model change as a migration, with the five steps, and you say "plan it before you need it, because the day you need it is the day a better model ships."

Signal 7 — you mention the relevance floor. That "I don't know" is unreachable without one is a subtle observation, and it demonstrates thinking about the absence of an answer as a first-class outcome.

Anti-signals:

• Describing a tenant filter as isolation.
• Chunk size discussed with no mention of structure.
• "We use LangChain's default retriever."
• Proposing a tuned fusion weight.
• No answer to "how do you know retrieval is good?"
• Treating citations as a UX feature.
• Not knowing what happens when the embedding model changes.

The question to ask them: "How do you isolate tenants in the vector store, and what's your recall@k on a golden set?" Two answers, and together they tell you whether retrieval is engineered or assembled.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Build the golden set first. Twenty real questions with known-correct source spans. It takes an afternoon and it converts every subsequent argument from taste into measurement. Teams that skip it argue about chunk size for months.
2. Demonstrate the filtering cliff. Load a realistic corpus, apply a 1-in-500 filter after ANN search, and measure recall@10 collapse. Then re-run with a namespace. Seeing recall go from 0.2 to 0.9 with no other change is the moment topology stops being abstract.
3. Run the chunking bake-off. Fixed-size vs structure-aware, measured as recall@k. The effect size is usually large enough to end the conversation permanently, and it teaches the habit of measuring the boring parts.

And the framing for the platform team: retrieval is where the platform touches the bank's information, so it is where the platform's biggest silent risk lives. Every other failure announces itself. This one hands a plausible answer to the wrong person and waits. That argument is how namespaces, ingestion tagging and a second output-side gate get funded — not as nice-to-haves, but as the only controls for a failure mode you cannot monitor.