Production RAG fails quietly when evaluation is treated as a launch checklist instead of an operating system. The useful harness is small, repeatable, and wired into the same delivery loop as application code.
Start with the questions that matter
We keep gold datasets close to real customer journeys: policy lookup, ambiguous search, conflicting sources, and edge cases where the assistant should refuse or ask for clarification.
Measure retrieval and generation separately
Retrieval quality, answer faithfulness, latency, and citation coverage move independently. Splitting them keeps the team from chasing the wrong fix when scores drop.
Make regressions visible
Every release compares the current model, embedding strategy, chunking, and reranking setup against a locked baseline. If the delta is not explainable, the release pauses before users feel it.