« Phase 06 · Warmup · Track Overview

Principal Deep Dive — Architecture, Tradeoffs & Blast Radius

Table of Contents

• 1. The three tradeoffs
• 2. Topology as a per-corpus decision
• 3. Scaling envelope
• 4. Failure modes and blast radius
• 5. The ingestion pipeline nobody designs
• 6. Retrieval in the latency and cost budgets
• 7. Decisions that look wrong but are intentional
• 8. What changes at 10×

1. The three tradeoffs

Tradeoff 1 — isolation vs cost. A namespace per tenant gives structural isolation and costs you index overhead per tenant, worse recall for small tenants (fewer neighbours to find), and more things to operate. A shared index is cheap and puts your entire tenant boundary inside a WHERE clause.

The resolution is per corpus, not per platform. Genuinely shared content — product documentation, published policy, regulatory text — is pooled, because there is no boundary to enforce. Anything carrying a tenant's business data is siloed. That is the bridge model, and it is what almost every bank converges on once someone asks "what happens if the filter is wrong?"

Tradeoff 2 — recall vs precision vs latency. Retrieve more candidates and recall rises; rerank more and precision rises; both cost latency, and the reranker costs a model call per pair.

The resolution is to fix the latency budget first (Phase 00) and derive the depths from it. Typically: retrieve 50, rerank to 5. Then measure recall@50 and precision@5 separately, because a single end-to-end number cannot tell you which stage is the constraint — and the fix for each is different (better chunking or embeddings vs a better reranker).

Tradeoff 3 — freshness vs stability. Frequent re-ingestion keeps answers current and makes them non-reproducible: the same question yesterday and today retrieves different evidence, which is a problem when someone asks you to explain a decision from six months ago.

The resolution is versioned documents plus a retrieval snapshot recorded in the trace. The chunk carries doc_version; the execution chain records which chunk ids and versions were retrieved. Then "what did the agent see" is answerable even after the corpus has moved on — and that is a Phase 15 requirement, not a retrieval nicety.

2. Topology as a per-corpus decision

The question is not "silo or pool" but "which corpora, and why". A workable classification:

The last row surprises people. Traces contain prompts, retrieved content and answers — which means they inherit the classification of the most sensitive thing they touched. A trace store treated as "telemetry" and pooled across tenants is a data leak with an observability label on it.

The operational consequence of the bridge model: a query may need to hit two namespaces (the tenant's own, plus the shared corpus) and fuse. That is fine — RRF handles it — but it must be deliberate, because the alternative is somebody "simplifying" by pooling everything.

3. Scaling envelope

Two that bite in practice.

The reranker is the throughput constraint, not the index. ANN search over a million vectors is single-digit milliseconds; a cross-encoder over 50 pairs is a model call with real latency and real cost. That inverts the intuition that "the vector database is the expensive part" — and it is why rerank depth is the first knob to turn under load, and why the degradation ladder puts it first.

Small tenants retrieve worse. A namespace with 200 chunks has fewer good neighbours than one with 200 000. Per-tenant silos therefore produce uneven quality across tenants, which is invisible until a small tenant complains. Mitigations: fuse with the shared corpus (so everyone has a floor), and monitor recall per tenant rather than in aggregate — an average hides exactly the tenants who are suffering.

4. Failure modes and blast radius

Three of the top four have no runtime detection, which is the defining property of this phase. Everything else in the platform fails loudly; disclosure through retrieval fails with a 200 OK.

That is the argument for two things a design review should insist on:

1. Structural isolation where the blast radius is a customer boundary — a namespace cannot be forgotten the way a filter can.
2. An independent second gate on the output side. Ingestion tagging will be wrong sometimes; MNPI and PII detection on the way out is the control that does not share a failure mode with it.

The freshness row is worth a note because it is the one useful leading indicator retrieval produces. A rising freshness-exclusion rate means ingestion has stalled — before anyone notices that answers are stale, and long before someone acts on one.

5. The ingestion pipeline nobody designs

Retrieval design gets attention; ingestion gets a script. Then it becomes the source of most production problems. What a real pipeline owes you:

• Idempotency. Re-ingesting a document must not duplicate chunks. Key on (doc_id, version, chunk index), not on arrival.
• Versioning, not overwrite. A superseded document's chunks are retired, not deleted, so a six-month-old trace can still be explained.
• Classification and barrier tagging at ingestion, validated — a typo must fail the ingest, not silently create an unfilterable chunk. The lab enforces exactly this in Document.__post_init__.
• Backpressure. Embedding endpoints are rate-limited; a bulk re-ingest must not starve live query traffic. Separate the capacity or throttle explicitly.
• Dead-letter handling. A document that fails to parse must land somewhere visible. Silently skipped documents are the most common cause of "the agent doesn't know about X."
• Freshness telemetry. Per-corpus last-successful-ingest, exposed as a metric, because the freshness contract is unenforceable without it.
• Deletion propagation. A document deleted at source must have its chunks retired — and in a bank, a customer's erasure request must be executable, which requires knowing which chunks derive from which source record.

That last point is a data-governance obligation, not an engineering nicety, and it is much cheaper to build at ingestion than to retrofit.

6. Retrieval in the latency and cost budgets

From Phase 00's 3-second budget, retrieval's share:

Two observations that matter architecturally:

Everything except rerank is parallelizable. The query embedding, BM25 and (once embedded) dense search have no data dependency on each other beyond embed→dense. Serializing them is the most common self-inflicted latency wound in a retrieval pipeline, and it roughly doubles the stage.

Rerank is 60% of the budget and the only genuinely optional part. That is what makes it the top of the degradation ladder, and it is why the lab makes shedding it a first-class, recorded outcome rather than an exception path.

On cost: retrieval affects the model bill more than its own. Every retrieved chunk is input tokens on every turn, and — from DEEP-DIVE §8 — more retrieval lowers your prefix-cache hit fraction, because retrieved content is volatile and sits after the stable prefix. So retrieval depth is a cost decision with a non-obvious second-order term, and "just retrieve 20 chunks to be safe" is more expensive than it looks.

7. Decisions that look wrong but are intentional

Tenant uses a namespace; classification uses a filter. Looks inconsistent — why not enforce both structurally? Because blast radius differs: a tenant bug crosses a customer boundary, a classification bug does not leave the tenant. Structural isolation has a real cost (index overhead, worse small-tenant recall), and spending it where the blast radius is largest is the trade. Making everything structural would mean a namespace per (tenant × classification × barrier), which multiplies indexes and makes every one of them worse.

excluded_by_entitlement counts per ranking, so one chunk can count twice. Looks like a bug. The counter measures filter actions, which is what tells you which retriever is surfacing ineligible content — useful when a barrier tag is missing and only the dense retriever finds it.

The pre-filter runs on each ranking, not on the fused set. Looks like duplicated work. Filtering after fusion lets an ineligible chunk shift the ranks of eligible ones, which leaks information through ordering even when it is ultimately removed.

Zero-scoring BM25 chunks are dropped, but the retriever still returns something for a nonsense query — until the rerank floor. Looks redundant to have both. They cover different retrievers: BM25 naturally returns nothing on no term match; dense retrieval always has a nearest neighbour. The floor is what makes the dense path able to say nothing.

Grounding is checked against the retrieved set, not the corpus. Looks weaker. It is the point: a claim supported by something the model never saw is a coincidence, not evidence. Checking against the corpus would let a hallucination pass because the fact happens to be true somewhere.

The assembler re-renders on every drop. Looks \( O(n^2) \) and wasteful. It is exact, and the incremental version is not — headers and separators are real tokens, and the lab's first implementation over-ran its budget by exactly that amount.

8. What changes at 10×

At 10 000 chunks and 3 tenants, the lab is close to shippable. At 10 million chunks and 40 tenants:

• ANN is mandatory, and with it the filtering cliff becomes a live concern. This is the point at which the topology decision stops being theoretical.
• Per-tenant recall monitoring replaces aggregate recall, because an average hides the small tenants who are suffering.
• Ingestion becomes a platform with backpressure, dead-lettering, versioning and deletion propagation — not a script.
• Re-embedding becomes a scheduled capability, not an emergency. Once you have done it once with a runbook, the next model upgrade is a week rather than a quarter.
• The reranker needs its own capacity plan, because it is the throughput constraint and it is a model call (Phase 05).
• Chunking changes need an eval gate, because they shift recall across the entire corpus and the effect is invisible per-query.
• Retrieval snapshots go into the trace, because reproducibility at six months is a governance requirement and cannot be reconstructed.
• Query rewriting and multi-hop retrieval start paying for themselves, and both need their own evaluation because both can make things worse.

Seams to build now, cheap today: doc_version on every chunk; classification and barrier validated at ingestion; per-tenant namespaces even when a tenant is tiny; retrieved chunk ids in the execution chain; and the freshness-exclusion counter, which will be your first useful ingestion alert.