Capstone 03 — GenAI Assistant Platform with Guardrails & Eval CI

Composes: Phase 04 (hybrid retrieval) · Phase 10 (guardrails + eval harness) · Phase 09 (canary) · Phase 11 (experiment readout) · Phase 12 (governance card) Time: 2–4 weeks

The enterprise-GenAI capstone: a RAG assistant whose every prompt/retrieval change passes an evaluation gate, whose attack surface is canary-tested, and whose release is a guarded canary — the JD's GenAI paragraph as a running system.

What you ship

corpus (provided synthetic KB + a poisoned-document injection set)
   │
hybrid retrieval (P04: BM25 + dense + RRF) ──► context assembly (budgeted)
   │                                                │
   │                                          generation (mock LLM by default;
   │                                          real provider behind a flag)
   │                                                │
   └────────► guardrail layer (P10): groundedness · injection · policy · schema
                                                     │
                              eval harness as CI gate (P10 + P08 pattern)
                                                     │
                  canary release (P09) + experiment readout (P11)
                                                     │
                            model/system card (P12) generated per release

Stage list

Corpus + golden set: a synthetic knowledge base with known facts (so groundedness has ground truth), a golden Q/A set, a canary attack set, and a poisoned-document subset for indirect injection.
Walking skeleton: BM25-only retrieval → extractive answer → harness run. The extractive path is also your fallback and your baseline row.
Retrieval depth: Phase 04 hybrid + RRF; measure recall@k on the golden set vs BM25-only — retrieval quality bounds everything downstream; prove it.
The guarded loop: Phase 10's full layer wired in; the harness becomes a gate (Phase 08's pattern): groundedness ≥ threshold AND attack detection ≥ threshold AND zero policy violations AND false-refusal rate ≤ threshold.
Release machinery: two assistant configs (e.g., different context budgets) through the Phase 09 canary router on simulated traffic; Phase 11 analyzer for the readout — including the SRM check on the router.
Governance: the Phase 12 card generated per release: config versions, gate results, known limitations, attack-suite status.

Money table (fill with your runs)

Metric	BM25-only baseline	Full system
retrieval recall@5 (golden set)
groundedness / unsupported-claim rate
canary-attack detection rate
indirect-injection success rate (poisoned docs)
false-refusal rate (benign suite)
canary experiment readout (metric, CI)

Injected-failure demo (required)

Ship a deliberately degraded prompt template (drops the grounding instruction); show the eval gate blocking the release with the specific failed cases — the "hallucination regression caught in CI" story.

Honest-limitations paragraph (required)

Mock-LLM determinism vs real-model variance; rule-based groundedness scoring limits (paraphrase); synthetic corpus scale; judge-free evaluation (and what Extension A of Phase 10 adds).