How to Evaluate an AI Agent Before It Goes Live
November 11, 2026 · 4 min read · Etzal Group Team
A demonstration proves that an agent can complete a task. It says nothing about how often. Before an agent answers customers or changes records, you need a way to evaluate an AI agent that produces a number, and a rule for what that number has to be.
What a test set is
A test set is a list of realistic inputs, each with a description of what a correct result looks like. Build it from real material:
- Past conversations and tickets. The questions people actually ask, in the words they use.
- Corrections from the pilot. Every time a reviewer changed the agent's output, that case belongs in the set.
- Known hard cases. Ambiguous requests, missing information, two requests in one message.
- Cases to refuse. Requests outside the agent's scope, attempts to extract information it should not share, instructions hidden inside a document it reads.
Write the expected result for each case in a form that can be checked: the right category, the fields that must be extracted, the tool that should be called, or the facts a reply must contain.
Score outcomes and actions
For an agent that only writes text, you score the text. For an agent that acts, you also score what it did.
- Outcome. Was the final answer correct and complete?
- Actions. Did it call the right tools with the right arguments? Did it call any it should not have?
- Honesty. Did it claim to have done something it did not do? An agent that says "I have updated your address" without calling the update tool has failed, however polite the sentence.
- Refusals. Did it decline what it should decline, and hand over to a person when it should?
- Cost and time. How many steps, how many tokens, how many seconds?
Record tool calls in a log and check claims against the log. This one check catches a class of error that reading replies will miss.
Three kinds of checks
- Code checks. Exact, cheap and stable. Is the output valid JSON? Does the extracted total equal the sum of the lines? Was the refund tool called for an amount above the limit?
- Model-graded checks. A second model scores the answer against a rubric, for example "does the reply answer the question asked, using only the provided policy text?" Useful for tone and completeness.
- Human review. A person reads a sample. Slow, and still the reference against which the other two are calibrated.
Use code wherever a rule can decide. Use a grading model for what code cannot judge. Use people to check that the graders agree with reality.
Repeat every case
Run each case several times. Report results as a pass rate per case and overall. A case that passes three times out of five is a warning: the behaviour exists but is not dependable. For actions that carry risk, treat anything below a consistent pass as a failure.
Set the bar before you look
Decide the release threshold in advance and write it down. For example:
- Overall pass rate of at least 90% on the full set.
- 100% on the safety cases: no forbidden tool calls, no invented actions.
- No more than a set number of steps on average.
A threshold chosen after seeing the results will always turn out to be the result you got.
Make it a gate
The evaluation should run automatically whenever the prompt, the tools, the knowledge sources or the model version changes. If the pass rate drops below the bar, the change does not ship. This is what turns a one-time test into protection against regressions, which are common: a prompt edit that fixes one case often breaks two others.
Keep a history of results. A slow decline over several changes is easy to miss release by release and obvious on a chart.
After launch
Production is the larger test set. Sample real conversations every week, score them the same way, and add every new failure to the set. Track the rate at which people override or correct the agent, because that is your users grading it for free.
Common mistakes
- Testing only the happy path. Agents fail at the edges.
- Checking the reply and not the actions. The text can be right while the action is wrong.
- A test set that never grows. It should absorb every production failure.
- One run per case. It hides unreliability.
- No fixed bar. Without one, every release is an argument.
Evaluation harnesses are part of every agent we deliver through Enterprise Software, and an AI Consulting engagement can design the test set for an agent you already have. Talk to us if you are preparing a launch.
Summary
Build a test set from real cases, including the ones the agent must refuse. Score outcomes, actions and honesty. Run each case several times. Fix the bar before testing, and run the evaluation on every change. An agent released this way has a known reliability, not a hoped-for one.
Frequently asked questions
How many test cases do we need to evaluate an AI agent?
Fifty well-chosen cases are enough to start and far better than none. Aim for one hundred to two hundred as the agent matures. Coverage matters more than size: include common requests, rare ones, incomplete ones and cases the agent must refuse.
Why run each test case more than once?
Language models do not always give the same answer to the same input. A case that passes once may fail on the next run. Running each case three to five times shows whether a behaviour is reliable or merely possible.
Can an AI judge the answers of another AI?
Yes, for qualities that are hard to check with code, such as tone or completeness. Give the judging model a clear rubric and compare its scores with human scores on a sample first. Use code checks wherever a rule can decide, because they are cheaper and do not drift.
Related service
WhatsApp AI Chatbot
We build, train, and run a WhatsApp AI agent that answers customers 24/7, qualifies leads, and follows up.
