Here is the pattern that sinks most enterprise AI projects: the prototype is impressive, it ships, and for three weeks it is fine. Then someone updates the model, or edits the prompt, or the knowledge base grows — and quality quietly drops. Nobody notices until a customer does. The thing that separates a durable AI agent from a demo that aged badly is not a better model or more retrieval. It is an evaluation harness: the test infrastructure that tells you, continuously, whether the agent is still good.
This is the companion to our voice AI architecture reference. That post builds the agent; this one keeps it honest.
Why RAG is not enough
Retrieval-augmented generation is necessary and oversold. It grounds the model in your data, but it does not tell you whether the answer was right, whether the tool call was appropriate, or whether last week's fix quietly broke this week's case. RAG improves the average; evaluation protects the tail. Enterprise AI fails in the tail — the rare, high-stakes cases — which is exactly what an average-case improvement hides.
Three layers of evaluation
A production harness works at three levels, each catching a different class of failure.
1. Offline test sets (the regression gate)
A curated set of inputs with known-good expectations, run on every prompt change, model change, and retrieval change — before it ships. This is the unit test of AI systems. The trap is grading: exact-match scoring is useless for open-ended output. You grade with a rubric, and increasingly with a model-as-judge that scores each response against explicit criteria, validated against human labels so you trust the judge.
// A single offline eval case. The harness runs hundreds of these
// on every change and blocks the deploy if the pass rate drops.
type EvalCase = {
id: string;
input: string;
context?: string[]; // retrieved docs, pinned for reproducibility
criteria: string[]; // what "correct" means, in plain language
mustNot?: string[]; // e.g. "invents an account number"
};
const gate = { minPassRate: 0.95, maxHallucinationRate: 0.01 };
// CI fails the build if the suite regresses below the gate.2. Online scoring (what actually happened)
Offline sets can't cover the messy reality of production traffic, so you score real interactions too — every one, or a large sample. Each response is judged for task success, faithfulness to retrieved context, tool-call appropriateness, and safety. These scores stream into dashboards with alerts, so a drop shows up in hours, not in a quarterly review. This is the same layer that catches a voice agent going quietly worse after a model update.
3. Human review (the ground truth)
Automated judges drift. A steady trickle of human-labelled cases keeps them honest: you sample scored interactions, have a human grade them, and measure how well the automated judge agrees. When agreement slips, you retune the judge before you trust its numbers again. Humans don't grade everything — they calibrate the machine that does.
Regression gates in CI
The harness earns its keep when it is wired into the deploy pipeline. A prompt edit or a model bump is a code change, and it goes through the same gate: run the offline suite, and if the pass rate or hallucination rate crosses the threshold, the deploy fails. This is what makes AI systems safe to iterate on quickly — you can change the prompt daily because the gate catches the change that made it worse.
Multi-tenant isolation in evaluation
If you run one agent across many customers, evaluation has to respect tenancy. Test sets and retrieved context are scoped per tenant, judges never leak one customer's data into another's scoring, and quality is tracked per tenant because an aggregate pass rate can hide one customer's agent quietly failing. The same per-tenant isolation you enforce in the vector store and memory has to extend into the eval layer, or your numbers lie by averaging.
// Quality is reported per tenant, never only in aggregate —
// a healthy fleet average can hide one tenant in free-fall.
for (const tenant of tenants) {
const score = await evaluate(tenant.id, sample(tenant.traffic));
metrics.record({ tenant: tenant.id, ...score });
if (score.taskSuccess < tenant.sla) alert(tenant.id, score);
}Prompt injection is an evaluation problem too
Any agent with tools and untrusted input has an attack surface: content that tries to hijack the model into calling a tool it shouldn't. You defend it in the architecture (least-privilege tools, output filtering, sandboxing) — but you verify it in the harness, with an adversarial test set of injection attempts that runs on every change. Security you don't continuously test is security you're assuming.
“RAG makes the average answer better. An evaluation harness protects the tail — and enterprise AI is judged on the tail.”
Where to start
You do not need all three layers on day one. Start with a small offline set of the cases that matter most and wire it into CI as a gate — that alone stops the worst regressions. Add online scoring once you have traffic, then human calibration once you rely on the judges. The point is to build the harness alongside the agent, not after the first incident. It is a standard part of how we ship AI transformation work, because it is the difference between an agent you can trust in a year and one you have to keep apologising for.
Written by
RMC Engineering