Testing an AI Assistant Before It Talks to Customers

Why does demo performance not predict production performance

A demo is a curated set of inputs shown to people who want the demo to succeed. Production is an unfiltered stream of real people, some of them confused, some of them adversarial, some of them asking something the assistant was never scoped to handle. These are different populations of input, and a model tuned on the first will predictably struggle with the second.

In our experience, the gap shows up in three predictable places: questions phrased nothing like the training examples, requests that fall just outside the assistant's intended scope, and edge cases specific to the domain that nobody thought to write a prompt for. None of these show up in a demo, because a demo is, by construction, the set of inputs someone already checked work.

Eval set: An eval set is a fixed collection of test inputs and expected outcomes used to measure a model's performance after training, separate from the data used to build or tune it. A good eval set is built from real usage patterns, not invented from imagination.

The single biggest predictor of this gap is where the eval set came from. An eval set invented by the people who wrote the prompts tests whether the model does what its authors expected. It says almost nothing about what happens when a real customer types something nobody expected.

How do you build an eval set from real user language

If the assistant is replacing or supplementing an existing channel — a support inbox, a chat log, a call transcript archive — that history is the best source of test cases available. Real users phrase things oddly, ask two questions at once, provide incomplete context and switch topics mid-conversation. None of that shows up if your eval set was written by an engineer trying to think like a customer.

1. Pull a representative sample from real historical interactions, stratified by topic and complexity, not just the most common cases
2. Include the awkward 10%: incomplete questions, multi-part requests, and messages that don't map cleanly to a single intent
3. Set the correct or acceptable outcome for each case before running the model, so pass criteria aren't adjusted after seeing results
4. Have someone who wasn't involved in building the assistant review the expected outcomes for bias toward what the model is likely to do
5. Refresh the eval set periodically as real usage patterns shift, rather than treating it as a one-time artefact

If no historical interaction data exists yet, the fallback is structured interviews with the people who currently handle these interactions manually — support agents, account managers, whoever fields the questions today. They know which requests are actually common and which ones only look common to someone unfamiliar with the workflow.

What should you actually test beyond accuracy

Accuracy on the happy path is the easiest thing to measure and the least predictive of whether an assistant is safe to launch. A useful test programme covers several categories, each catching a different failure mode.

Out-of-scope handling deserves particular attention. An assistant that answers confidently outside its intended remit is often more damaging than one that answers incorrectly within it, because the user has no signal that anything went wrong. Testing refusal behaviour means deliberately sending requests the assistant should decline and checking it declines gracefully, not just checking it doesn't refuse things it shouldn't.

How do you red-team for domain-specific failure

For assistants handling regulated or sensitive interactions, our AI safety and model testing work is built specifically around this kind of domain-specific red-teaming.

Generic adversarial testing — jailbreak prompts, prompt injection attempts, requests for the system prompt — is worth running but it's not domain-specific. A support assistant for a financial services firm needs to be tested against a different set of failures than a customer assistant for a consumer retailer, and neither set is covered by generic jailbreak libraries alone.

The people best placed to write domain-specific adversarial cases are the people who understand where the domain's actual risk sits: a compliance lead for a regulated product, a support manager who knows which customer complaints escalate fastest, a clinician if the assistant touches anything health-adjacent. Red-teaming built by the same team that built the assistant is systematically blind to exactly the failures that team didn't anticipate, for the same reason a self-reviewed eval set is blind to unanticipated real-world input.

• Identify the two or three worst plausible outcomes specific to your domain, not generic ones
• Write test cases designed explicitly to trigger those outcomes, not to confirm the assistant behaves well
• Involve someone with no stake in the launch date reviewing the results
• Re-run the domain-specific red-team suite after any material prompt or model change, not just once before launch

What is a drift detection baseline and do you need one

Drift detection baseline: A drift detection baseline is a fixed set of inputs and expected outputs, re-run on a schedule after launch, used to catch quiet degradation in performance over time as the model, prompt, or the underlying data it draws on changes.

Assistants degrade after launch for reasons that have nothing to do with the original test programme: an underlying model provider updates their model, a knowledge base gets edited, a prompt gets tweaked by someone trying to fix an unrelated issue. Without a fixed baseline re-run on a schedule, these regressions surface only when a customer complains, which is the most expensive way to find out.

A drift baseline doesn't need to be large. It needs to be fixed, so results are comparable over time, and it needs an owner who actually looks at the results rather than a dashboard nobody checks. Weekly or monthly re-runs, depending on how often the underlying system changes, are usually sufficient for a mid-market deployment.

What happens when the assistant gets it wrong

The question that matters before launch is not whether the model is good enough. Every model is wrong some percentage of the time. The question is what happens on that percentage: does the user know something went wrong, is there a clean path to a human, and does anyone on your side see the failure and learn from it.

This is the bounded-AI argument in practice: task-scoped, approval-gated where the stakes justify it, and audit-trailed so failures are visible rather than silent. An assistant that fails occasionally but escalates cleanly and gets logged is a workable production system. An assistant with a marginally lower failure rate but no escalation path and no logging is not, regardless of the headline accuracy number.

• Design the escalation trigger before launch: confidence threshold, explicit user request, or specific topic flags
• Make the handoff to a human visible to the user, not silent or ambiguous
• Log every escalation and every low-confidence response for review, not just errors a user reported
• Review escalation logs on a schedule and feed recurring patterns back into the eval set

If your team is deciding whether to build this evaluation capability internally or bring in outside review, our companion post on how to evaluate an AI training data vendor covers most of the same due-diligence questions, and our piece on what model-ready data actually means is worth reading alongside this one since a weak underlying dataset produces exactly the eval failures described here. A quick readiness check can also tell you early whether your current test coverage has the obvious gaps this article describes. Our AI safety and quality service and customer support assistant work are both built around this pre-launch programme, and teams evaluating a broader deployment should also look at our notes for AI teams generally.