Short answer: the AI agent KPIs that tell you whether an agent is working are the ones tied to finished work and business outcomes: task completion rate, first-pass accuracy, escalation rate, human time spent per task, cost per successful outcome, and the customer or business result the agent exists to move, such as resolution time, conversion or days sales outstanding. Activity numbers like "conversations handled" or "messages sent" measure busyness, not value.
This post sets out a practical KPI set in three layers, explains how to define each metric so it can't be gamed, shows what a simple dashboard looks like, and suggests a review cadence that works for a small or mid-sized business.
Why do most AI agent dashboards measure the wrong things?
Because the easy numbers are activity numbers. Platforms report conversations, messages, tokens and uptime out of the box. Those are useful for operations but say nothing about whether the work was done well.
An agent can handle thousands of conversations and still fail: by giving answers that send customers to a human anyway, by creating records staff have to fix, or by deflecting people who then churn. A support team celebrating "conversations handled by AI" while repeat contacts climb is measuring the wrong thing.
The fix is to define, for each workflow, what a successful outcome is, and measure from there.
What are the core AI agent KPIs?
Think in three layers: quality (is the work right?), efficiency (what does it take?), and outcome (does the business benefit?).
| KPI | Layer | Definition | What it catches |
|---|---|---|---|
| Task completion rate | Quality | Share of tasks the agent finished end to end without human takeover | Agents that start well but stall |
| First-pass accuracy | Quality | Share of completed tasks accepted without correction | "Completed" work that's wrong |
| Escalation rate | Quality | Share of tasks handed to a human, by reason | New case types; over- or under-confidence |
| Error severity mix | Quality | Count of errors by impact (cosmetic, rework, customer-facing, financial) | A few serious errors hidden among many trivial ones |
| Human time per task | Efficiency | Minutes staff spend reviewing, correcting or finishing each task | Agents that shift work rather than remove it |
| Cost per successful outcome | Efficiency | Model, infrastructure and human review cost divided by successful tasks | Cheap calls that create expensive rework |
| Cycle time | Efficiency | Time from trigger to finished task | Speed gains, or bottlenecks at review steps |
| Business outcome metric | Outcome | The number the agent was built to move | Whether any of this matters |
Task completion rate
Define "complete" precisely. For an invoice-processing agent: a draft entry created with all mandatory fields, validated, and ready for approval. For a support agent: the customer's issue resolved without a human, confirmed by no repeat contact within a set window. Vague definitions make every number look good.
First-pass accuracy
The share of the agent's output accepted as-is. For drafts that people approve, log whether they edited before approving, and what they changed. This is often the single most informative quality number, because it's measured by the people who know the work. Track it alongside completion: high completion with low first-pass accuracy means the agent is finishing jobs badly.
Escalation rate, by reason
A single escalation rate is ambiguous. Break it down: escalated by design (policy says a human handles refunds over a threshold), escalated for low confidence, escalated because a tool failed, escalated because the customer asked. The by-design ones are healthy. Rising tool failures or confidence escalations point to problems. A sudden drop in escalations deserves as much scrutiny as a rise. Our post on human-in-the-loop AI agents covers designing those handoffs.
Human time per task
The most underused metric. If staff spend almost as long checking the agent's work as doing it themselves, the agent isn't saving much. Estimate it by sampling: time a set of reviews each month. It's also the number that ties most directly to capacity freed up.
Cost per successful outcome
Total cost (model usage, infrastructure, tools, plus human review time) divided by successful tasks, not by attempts. It's the honest way to compare designs and to track whether optimization is working. See how to control a growing AI bill for the levers.
Which outcome metrics should you tie each agent to?
Every agent should have one primary business metric agreed before launch. Some examples by function:
| Agent | Primary outcome metric | Supporting metrics |
|---|---|---|
| Customer support agent | Resolution time; repeat-contact rate | Customer satisfaction on AI-handled tickets |
| Lead qualification agent | Speed to first response; qualified-meeting rate | Sales acceptance of AI-qualified leads |
| Accounts payable agent | Invoices processed per person per day; payment errors | Early-payment discounts captured |
| Collections agent | Days sales outstanding; promise-to-pay kept rate | Disputes raised |
| Recruitment screening agent | Time to shortlist; hiring manager acceptance of shortlists | Candidate drop-off; fairness checks across groups |
| Internal helpdesk agent | Tickets resolved without IT or HR | Employee satisfaction; time saved for the team |
Measure a baseline before launch. Without the "before" number, you can't show the "after" improvement, and you'll end up arguing from anecdotes. Our guide on how to measure AI ROI walks through baselining.
Be careful attributing changes. If resolution time improves while you also hired two people and changed your returns policy, the agent isn't the only cause. Where volume allows, compare AI-handled cases against similar human-handled ones over the same period.
What does a practical KPI dashboard look like?
For illustration, here's what a one-page weekly view might contain for a support agent at a mid-sized online retailer:
- Headline outcome: median resolution time and repeat-contact rate, this week against the pre-launch baseline.
- Quality row: completion rate, first-pass accuracy on sampled conversations, escalation rate split by reason.
- Efficiency row: human minutes per AI-handled conversation, cost per resolved conversation.
- Risk row: count of serious errors (wrong refund information, wrong order details shared), complaints mentioning the AI, any incidents.
- Trend lines: eight weeks of each, so drift is visible.
- Top failure reasons: the three most common causes of escalation or correction this week, with an owner for each.
That's enough to run the agent. Engineering will have more detailed telemetry underneath, covered in how to monitor AI agents once they're live, but business owners shouldn't need to read traces to know if their agent works.
How often should you review AI agent KPIs?
A cadence that suits most SMBs:
- Daily (automated): alerts only, for error spikes, tool failures, cost anomalies or a jump in escalations. No meeting.
- Weekly (30 minutes): the workflow owner reviews the dashboard and a sample of real conversations or outputs, decides on fixes, and assigns them.
- Monthly: outcome metrics against baseline, cost per outcome, and whether to expand the agent's scope, hold, or pull back.
- Quarterly: a business review of all agents: which ones earn their keep, what to retire, what to build next.
Name one owner per agent, someone on the business side who is accountable for its results. Without an owner, dashboards decay into wallpaper.
Which KPI mistakes should you avoid?
- Counting activity as value. Conversations handled, emails sent and documents processed are volume, not success.
- Rewarding deflection. If the agent's goal is "fewer tickets reach humans", it will learn to make reaching a human hard. Measure resolution, not avoidance.
- Ignoring the review burden. An agent that needs constant checking can look productive while costing staff time.
- Averaging away serious errors. One wrong payment matters more than a hundred slightly clumsy replies. Track severity separately.
- Changing definitions mid-stream. Keep KPI definitions stable, or note changes clearly, so trends stay meaningful.
- No baseline. Every claim of improvement becomes debatable.
How should you set KPIs for your first agent?
Before you build, write one page: the workflow, the definition of a successful task, the primary outcome metric and its current baseline, the three or four quality and efficiency metrics you'll track, and who owns the weekly review. If you can't fill in that page, the use case isn't ready yet. If you're still choosing where to start, AI agent use cases by department is a good place to look.
Frequently Asked Questions
What is a good task completion rate for an AI agent?
It depends on the workflow's complexity and risk; there's no universal benchmark. What matters is that the rate is measured against a strict definition of "complete", improves or holds steady over time, and comes with high first-pass accuracy. A modest completion rate on hard tasks can still be very valuable.
How do you measure AI agent accuracy?
Sample completed tasks regularly and have someone who knows the work grade them against clear criteria, and capture edits automatically wherever staff approve drafts. Combine that with a fixed evaluation set you re-run after any change, so you can separate real quality shifts from changes in incoming work.
How do you calculate ROI for an AI agent?
Compare the value of the outcome change (time saved, revenue gained, errors avoided) against the total cost of building and running the agent, including human review time. You need a pre-launch baseline to make the comparison credible.
Should customer satisfaction be an AI agent KPI?
Yes, for any customer-facing agent, measured specifically on AI-handled interactions and compared to human-handled ones. Pair it with repeat-contact rate, because satisfaction surveys have low response rates and can miss customers who simply gave up.
Who should own AI agent KPIs?
A business owner for each agent, typically the manager of the team whose work it does, supported by whoever builds and maintains it. Technical teams own system health; business owners own whether the agent is delivering results.
Want help defining what success looks like for your agent?
If you're planning an AI agent, or running one without a clear way to judge it, tell us about the workflow. We'll help you set KPIs, baselines and a review routine that show whether it's working. Here's how we approach agentic AI development.