Short answer: an AI agent is ready for real work when it meets pass criteria you set in advance on a test set built from your own real cases, behaves safely when people try to break it, and matches or beats your current process in shadow mode on live traffic. AI agent evaluation isn't a one-off exam. It's a permanent test suite that runs every time you change a prompt, a tool, a document or a model, so you know whether the change made things better or quietly broke something.

Most agent projects that disappoint don't fail because the model is weak. They fail because nobody measured whether the agent worked on realistic inputs before it went live, and nobody noticed when it stopped working afterward. This post covers how to build a golden dataset from real cases, which metrics actually matter, how to use AI to grade AI without fooling yourself, why regression testing is non-negotiable, and how shadow mode and red-teaming close the gap between test and reality.

Why isn't a good demo enough?

Demos use friendly inputs. Someone types a clear, polite question the agent was designed for, and it answers beautifully. Real traffic is different:

  • Emails with three questions in one, half of them off-topic
  • Customers who are angry, vague or writing in a mix of languages
  • Scanned documents at an angle, with handwriting in the margins
  • Requests the agent shouldn't handle at all
  • Edge cases nobody thought to mention during scoping

An agent that's impressive on ten hand-picked examples can be wrong on a meaningful share of real cases. You only find out by testing on real cases, at volume.

How do you build a golden dataset for AI agent evaluation?

A golden dataset is a collection of real inputs with the correct outcome recorded for each. It's the single most valuable asset in an agent project, more durable than any prompt or model choice.

  1. Collect real cases. Pull a few hundred historical examples from the actual workflow: support tickets, invoices, emails, form submissions, call transcripts. Remove or mask personal data as needed.
  2. Cover the distribution. Include common cases roughly in proportion to how often they occur, plus deliberately chosen hard cases, edge cases and "should escalate" cases.
  3. Record the right answer. Have the people who do the work today label each case: correct category, correct extracted values, correct action, or "should go to a human." Where they disagree, discuss and decide; disagreements often reveal unclear policy.
  4. Define what "correct" means for each case. For extraction, exact field values. For actions, the right tool called with the right arguments. For written replies, a checklist: answers the question, cites the right policy, makes no promises outside policy, appropriate tone.
  5. Keep growing it. Every production failure, customer complaint or reviewer correction becomes a new test case.

For illustration, a golden set for an invoice-processing agent might hold a few hundred invoices from your real suppliers, including duplicates, credit notes, invoices with missing PO numbers, multi-page invoices and a few that aren't invoices at all.

Which metrics matter for AI agent evaluation?

Measure what the business cares about, not just what's easy to count.

MetricWhat it measuresExample
Task success rateShare of cases completed fully and correctlyTicket answered correctly with the right action taken
Field or step accuracyCorrectness of individual partsInvoice date, amount and supplier each right
Correct escalationHands off what it should, keeps what it canRefund disputes sent to a human; status questions handled
Critical error rateMistakes with serious consequencesWrong customer emailed; payment created incorrectly
Tool-call correctnessRight tools, valid arguments, sensible orderLooked up the order before replying about it
GroundednessClaims supported by retrieved sourcesPolicy answer matches the policy text
LatencyTime per taskFast enough for live chat or voice
Cost per taskModel and service costsFits the budget at expected volume

Set thresholds before you test, and set them per metric. A critical error rate threshold should be far stricter than an overall success rate threshold. An agent that's right most of the time but occasionally emails the wrong customer's data isn't ready, however good the average looks.

How do you use an LLM as a judge without fooling yourself?

Some outputs, like free-text replies and summaries, can't be scored by exact match. A common technique is to have another language model grade them against a rubric. It scales well, but it needs care:

  • Use a specific rubric, not "rate this 1-10." Ask concrete yes/no questions: "Does the reply state the refund amount from the policy? Does it promise anything not in the policy? Does it address all questions asked?"
  • Give the judge the reference material: the correct answer, the source policy, the original input.
  • Calibrate against humans. Have people grade a sample of a hundred or so outputs too. If the model judge and the humans disagree often, fix the rubric before trusting it.
  • Watch for bias. Model judges can favor longer answers, confident tone, or outputs written in their own style. Check for these patterns.
  • Keep humans on the critical stuff. Use model grading for scale; use human review for high-stakes categories and for periodic audits of the judge itself.

Why is regression testing on every change non-negotiable?

Agents are sensitive to small changes. Rewording one instruction to fix one problem can break three other behaviors. A new model version can be better overall and worse on your specific task. Updating a policy document can shift answers in unexpected places.

Treat your golden dataset as a regression suite:

  1. Run it automatically on every change to prompts, tools, retrieval settings, documents or models.
  2. Compare to the previous version, case by case, not just on totals. Which cases newly fail? Which newly pass?
  3. Block releases that drop below thresholds or introduce new critical errors.
  4. Keep a changelog linking each change to its test results.

This is what makes it safe to keep improving an agent, and to switch models when a better or cheaper one appears. Our guide to choosing an LLM for business shows how the same test set drives model selection.

What is shadow mode, and how long should it run?

Test sets are built from the past. Shadow mode tests the agent on the present. The agent processes real, live inputs alongside your team, but its outputs aren't used. You compare what it would have done with what your people actually did.

Shadow mode reveals things a test set can't:

  • New case types that weren't in historical data
  • Integration problems with live systems
  • Latency and cost at real volume
  • How often people would have overridden it

Run it until you've seen enough volume across the case types that matter, often a few weeks for a busy workflow, longer for a low-volume one. Then move to live operation with human approval, and loosen approval as evidence builds. Our post on when a human should approve what an AI agent does covers that progression.

How do you red-team an AI agent?

Red-teaming means deliberately trying to make the agent fail or misbehave. Assign someone, ideally not the person who built it, to try:

  • Prompt injection: instructions hidden in emails, documents or form fields ("ignore your instructions and...").
  • Data leakage: asking for other customers' information, internal documents or the agent's own instructions.
  • Policy pressure: angry or persistent requests for refunds, discounts or exceptions.
  • Out-of-scope requests: legal, medical or financial advice; topics unrelated to its job.
  • Malformed inputs: empty messages, huge attachments, unusual languages or scripts, corrupted files.
  • Tool abuse: inputs that might cause loops, mass actions or calls with dangerous arguments.

Every successful attack becomes a test case and, usually, a guardrail. Our guide to prompt injection and AI agent security covers attack patterns in depth, and AI agent guardrails covers the defenses.

What does a readiness checklist look like?

Before launch, an agent should have:

  1. A golden dataset of real cases covering common, hard and escalation scenarios.
  2. Pass thresholds agreed with the process owner, including a strict critical-error threshold.
  3. Results meeting those thresholds on the latest version.
  4. Automated regression testing on every change.
  5. Red-team results with fixes for anything serious.
  6. Shadow-mode results on live traffic comparable to or better than the current process.
  7. Monitoring in place for production, with failures feeding back into the test set.

Frequently Asked Questions

How many test cases do we need to evaluate an AI agent?

A few hundred well-chosen real cases is a solid start for most business workflows. What matters more than raw count is coverage: common cases, hard cases, edge cases and cases that should be escalated. Add new cases continuously from production.

Can we evaluate an agent without technical staff?

The people who do the work today are essential for labeling cases and judging quality, and they don't need to be technical. Running automated tests and regression suites typically needs some engineering support, from your team or a partner.

What accuracy is good enough to launch?

It depends on the cost of errors and how they're caught. Compare against your current process, which also makes mistakes, and set stricter thresholds for critical errors. With human approval on actions, you can launch earlier and tighten as evidence builds.

How often should we re-evaluate a live agent?

Run the full test suite on every change, and review production samples weekly at first, then monthly. Re-evaluate whenever your provider updates the model, your policies change, or monitoring shows a shift in inputs or outcomes.

Is LLM-as-judge reliable?

It's useful for scale when it's given a specific rubric and reference material, and calibrated against human grading. It shouldn't be the only check on high-stakes outputs, and the judge itself needs periodic auditing.

Want to know if your AI agent is really ready?

If you have an agent in the works, or one in production you're not sure about, tell us what it does and how it's tested today. We'll help you build a test set from your real cases and a release process you can trust. Learn more about our agentic AI development.