Design 03 — GenAI Assistant Platform

"Design an enterprise RAG assistant over internal knowledge, with safety controls, for 50k employees." Phases 04 and 09–12 built the parts; capstone 03 assembled them.

1. Requirements

Functional: Q&A over internal docs (wikis, policies, tickets) with citations; freshness within hours of doc changes; per-user access control on sources; structured actions later (out of scope v1 — say so and why: the risk class changes, see §6).

Non-functional: TTFT < 1.5 s, streaming; groundedness gated (unsupported- claim rate ≤ threshold on the golden set); injection-tested; cost budget per 1k queries; auditability (who asked, what was retrieved, what was answered).

The framing that wins the room: this is a failure-management design — hallucination measured not anecdotal, injection as an attack surface, evaluation as CI. (Phase 10 Ch. 1.)

2. Architecture

docs ──► ingestion: contracts → chunking → embeddings + BM25 index   (P01, P04)
              │  ACL metadata carried per chunk
              ▼
query ──► input guards (injection scan, policy) ─► retrieval (hybrid BM25+dense
              │                                     + RRF, ACL-filtered)  (P04)
              │                                          │ top-k → rerank
              │                                          ▼
              │                              context assembly (token budget,
              │                              source attribution, freshness)
              │                                          ▼
              │                              generation (provider-abstracted,
              │                              prompt registry, streaming)
              │                                          ▼
              └─► output guards: groundedness · policy · schema (P10)
                                       │
                         respond-with-citations │ extractive fallback │ refuse
                                       │
            logs: full trace (query, chunks, prompt version, answer, guard verdicts)

Release machinery around it: eval harness (golden + canary + false-refusal suites) as the CI gate on every prompt/retrieval/model change (P10 + P08); canary rollout with guardrails (P09); per-release system card (P12).

3. Deep dives

Retrieval (quality bound for everything): hybrid BM25 + dense with RRF (rank-based fusion — no score calibration needed); chunking by structure (headings/sections) with overlap, chunk size traded against context budget; reranking (cross-encoder) on the top-50; ACL filtering inside retrieval (post-generation filtering leaks via the answer — access control must bind before the model sees the text). Measure retrieval recall@k on the golden set separately from end-to-end quality — it localizes every "hallucination" debug (Phase 10's triangle).

Groundedness gating: sentence-level claims scored for support against the retrieved context (overlap baseline → NLI as budget allows); below-threshold answers degrade to the extractive fallback (quote the chunks with citations) — degrade, don't fail. The unsupported-claim rate over the versioned golden set is the release gate and the trended KPI.

Injection defense in depth (the corpus is user-writable — indirect injection is the threat model): least-privilege architecture first (v1 has no tools — the strongest defense is having nothing to hijack), context filtering on retrieved chunks, untrusted-span marking, output leak filtering, and the canary attack suite (including poisoned-document canaries planted in a test corpus) with attack-success-rate tracked per release.

Freshness: doc-change events → re-chunk/re-embed pipeline (streaming, Phase 02 pattern); index versioning with atomic alias flips; staleness as a monitored metric per source.

Cost & latency: prompt caching for the stable system prefix; context-budget discipline (retrieval precision over chunk count); the cascade (head queries → small model/extractive path; tail → big model) with the router itself evaluated; streaming with sentence-buffered output guards. Quote the shape: context tokens dominate cost; caching + assembly discipline routinely cut 30–50%.

4. Evaluation & operations

Three cadences (Phase 10 Ch. 7): rule-based guards per request (ms); the harness per change (golden Q/A + canaries + false-refusal benign suite — over-refusal eats the product silently); human-labeled audits monthly calibrating the scorers. Online: thumbs + citation-click rates as weak labels; A/B for major changes (Phase 11 — with SRM on the router); drift on query-topic distribution (Phase 12) catching "users started asking about things the corpus lacks" — which is a content gap surfaced as an unanswerable-rate metric, the single most actionable GenAI dashboard.

5. Failure modes

6. Evolution

Tool use / actions (the risk-class jump: allow-listed tools, parameter validation, confirmation UX, per-user authorization, action audit — interview- prep/04's last question is the script); multi-turn memory with summarization; fine-tuned domain models where retrieval can't close the gap (with the eval harness as the constant across all of it).