GenAI Production Questions

The beyond-the-demo loop. Each answer maps to Phase 10 (+ Phases 04, 09, 11, 12 where they compose).

Q: Users say the assistant "hallucinates." Engineer a response. Instrument before fixing: build the golden set, measure the RAGAS-style triangle — context relevance (was the answer in the retrieved chunks?), groundedness (unsupported-claim rate via sentence-level support scoring), answer relevance. Low context relevance → it's a retrieval bug (hybrid search, reranking, chunking — Phase 04), the most common case. Context fine but support low → generation: grounding instructions, citation requirements, temperature, and the groundedness output-guard with extractive fallback. Then gate the metric in CI so the fix can't regress. Skeleton: measure → localize (retrieval vs generation) → fix the right stage → gate.

Q: Define your hallucination metric precisely enough to gate releases on. Unsupported-claim rate: decompose answers into sentence-level claims; score each for support against the retrieved context (token-overlap baseline → NLI → LLM judge as budget allows); rate = unsupported/total over a versioned, content-hashed golden set. Gate = rate ≤ threshold with the eval harness run on every prompt/retrieval/model change. Name the scorer's failure modes (paraphrase → false alarms; coincidental overlap → false passes) and the calibration anchor: periodic human-labeled samples measuring scorer agreement.

Q: Prompt injection — what's the threat model and what actually defends? Two classes: direct (user instructs the model to misbehave — bounded by the user's own privileges) and indirect (instructions embedded in retrieved documents/emails — the attacker programs your system through your corpus, with the system's privileges; any RAG over user-contributed content has this surface). Defense in depth, honestly ranked: (1) privilege separation — the only layer that holds against novel attacks: least-privilege outputs, allow-listed tools with validated parameters, confirmation on side effects; (2) input/context filtering with pattern detectors; (3) output filtering for leakage; (4) canary regression suites — known attacks run on every change, attack-success-rate tracked like accuracy, because detectors decay as attacks evolve. 100% canary detection ≠ safe; it's a regression floor, not a ceiling.

Q: How do you make LLM outputs safe for downstream systems to consume? The reliability ladder: constrained decoding where you control inference (grammar/schema at the token level — malformed output becomes impossible); otherwise validate-then-repair — schema-validate with aggregated path-specific errors, re-prompt with those errors (one round fixes most failures), bounded at 1–2 repairs, then fallback. Never raw json.loads into business logic. Design detail that matters: error-message quality is now system quality — the errors are the repair prompt.

Q: When do you trust LLM-as-judge evaluation? After validating against human labels on a sample (report agreement); with bias mitigations — position (swap orders), verbosity (length-normalize), self-preference (judge ≠ generator family); judge version pinned and re-validated on change. Use for relative comparisons (prompt A/B) where biases partially cancel; not for absolute quality claims. A judge is a model in production — it gets a golden set too.

Q: Your GenAI feature costs $40k/month in tokens. Cut it without losing quality — and prove you didn't. Levers in ROI order: (1) prompt caching for stable prefixes (system prompt + few-shot header — often 30–50% of tokens); (2) context-assembly discipline — retrieval precision up, chunks down (tokens are bought per request); (3) the cascade — router sends head queries to a small model/cache/extractive path, escalates the tail (enterprise traffic is head-heavy; 2–5× cuts are normal); (4) response caching keyed on (model version, canonicalized query) for repeated questions; (5) output-length budgets. "Prove it": the eval harness as the constant — every lever change passes the same golden + canary gates, and the cascade's router gets its own evaluation (its errors are silent quality loss). Cost per request and quality metrics on one dashboard.

Q: Design the rollout for swapping the underlying LLM (provider upgrade). Treat it as a model deployment, not a config change: full eval-harness run (golden + canaries + false-refusal suite) against the new model offline; prompt adjustments expected (models respond differently to the same template — re-tune, re-gate); shadow on mirrored traffic comparing groundedness/latency/ cost distributions; canary with guardrails (Phase 09's router + Phase 11's significance discipline); model/system card updated with the new version's known limitations. The trap to name: judge-based metrics may shift because the judge relates differently to the new model — keep the judge pinned and human-calibrate the delta.

Q: The PM wants the assistant to also execute account actions (refunds, cancellations). What changes? The risk class changes: from "wrong words" to "wrong actions," and indirect injection now has side effects. Requirements before shipping: allow-listed tool registry with parameter validation (amount caps, account-scope checks), confirmation UX for irreversible actions, per-tool authorization tied to the user's privileges (the model never exceeds its principal), full action audit logs, and the canary suite extended with action-hijack attacks. Plus governance (Phase 12): this moves the system up a risk tier — model card, approval chain, human override path, and an incident playbook. The one-sentence version: tools turn an LLM app into an agent, and agents get platform-grade safety engineering, not prompt engineering.