Method · The numbers behind a pilot · updated 1 November 2026

What a pilot measures, and how the number is taken

A pilot proves two things: that the workflow metric moved against a baseline taken the same way for two weeks before, and that the safety metrics held. For a front-desk or lead agent the workflow metric is median first-reply time and enquiry-to-booking rate; for a document agent it is documents-complete rate and days to complete; for a back-office agent it is staff minutes per item and error rate. The safety metrics are the same everywhere: zero unapproved actions, an escalation rate that is high enough to be real and low enough to be useful, every opt-out honoured, and the share of drafts a person corrected. The decision rule is written down before go-live, and the report is one page: baseline, pilot, what broke, what it cost, and whether to continue.

Before a buyer pays for a pilot they should know exactly what proof they will get. This page is that: the metrics by kind of workflow, the safety metrics that have to hold regardless, the two-week baseline method, the decision rule agreed before anything is built, and the one-page report the pilot ends with. It is the method we use; it is also the method to hold any vendor to.

Why the measurement is the product

An agent that "works" is a demo. An agent that moved a number you measured yourself, with the same method before and after, is a result you can defend to a partner, a board or a sceptical front desk. Most pilots that fail to renew did not fail; they were never measured, so nobody could say whether they worked.

Two weeks is short. It is long enough to measure response time, hours, completion rates and escalations on a workflow that happens a few dozen times a week. It is not long enough to measure retention, lifetime value or revenue. A pilot report that claims a revenue uplift in two weeks is modelling, not measuring, and should say so. Ours does.

The workflow metrics, by kind of agent

Pick the two or three that fit the workflow. Each has a definition, so the before and after mean the same thing.

Median first-reply time
Minutes from the customer’s first message to the first substantive reply, median over all enquiries in the period, reported separately for business hours and after hours. The headline metric for any enquiry, lead or front-desk agent.Measure yours now
Share answered within 5 minutes
Enquiries with a first reply inside five minutes, divided by all enquiries. The research bar; the median can hide a long tail.
Enquiry-to-booking (or -consultation, -viewing) rate
Enquiries that reached a confirmed booking within the period, divided by enquiries. The conversion metric for lead and reservations agents. Define "confirmed" once and keep it.
No-show rate
Confirmed appointments where the person did not arrive, divided by confirmed appointments. The metric for reminder and confirmation agents. Needs at least two weeks and a few dozen appointments to mean anything.
Recall list worked
Share of the overdue list that received a contact attempt in the period, and share that booked. The metric for recall agents; most practices’ baseline is close to zero.
Documents-complete rate and days to complete
Matters or files where every checklist item was received, divided by matters opened; and the median days from first request to complete. The metric for document-chasing agents.
Staff minutes per item
Minutes of a person’s time per enquiry, booking, request or document, logged on the baseline sheet, averaged over the period. The hours metric for every agent; the one owners recognise.Estimate yours
Error rate
Items the agent handled that a person later had to correct, divided by items handled. For back-office agents (categorisation, data entry, reconciliation) this is the quality metric and it is measured by sampling, not by assumption.
Direct share
For hotels and anyone paying channel commission: bookings made directly, divided by all bookings. The money metric when the agent answers at night.

The safety metrics, every pilot

These hold regardless of the workflow. A pilot that moves the workflow metric and fails one of these does not continue.

Unapproved actions
Actions the agent took that the rulebook marked approval-only or never. The target is zero, and one is an incident with a root cause in the report, not a footnote.
Escalation rate and reasons
Conversations handed to a person, divided by all conversations, with the top reasons. Too low (under about 5%) usually means the escalation list is incomplete; too high (over about 40%) means the approved facts are thin. Both are fixable in week one.
Opt-outs honoured
Every "stop" request, and whether messaging to that contact ceased immediately. The target is 100%, checked from the log, not assumed.
Draft correction rate
For draft-only actions: the share of drafts a person edited before sending, and how much. The metric that decides whether a line moves from draft to act.
Wrong-fact count
Statements the agent made that were not in the approved facts or the connected systems, found by reading a sample of transcripts weekly. Each one becomes a line in the approved facts list or a rule.
Running cost
Model usage, hosting and messaging fees for the period, from the actual bills, against the band quoted at scoping.

The baseline method and the decision rule

Same sheet, same definitions, same length of time, before and after. Agreed before anything is built.

The baseline method, step by step

Week one of the pilot, and the two weeks before it if the business can start early.

BASELINE METHOD — [workflow], agreed [date] between [business owner] and [builder]

1. PERIOD: [14] consecutive days before go-live, including weekends. The same length after go-live, starting on day [3] of live operation (the first two days are shakedown and are logged but not counted).
2. UNIT: one row per [enquiry / booking / request / document]. Everything that arrives on [channels] counts, including the ones nobody replied to.
3. SHEET: the pilot baseline sheet (columns: date, channel, time received, time first replied, replied by, handled fully by agent, outcome, escalated + reason, staff minutes, notes). Kept by [name]; checked by [builder] twice a week.
4. DEFINITIONS, fixed now: "first reply" = first substantive message, not an auto-acknowledgement; "booked" = [confirmed in system]; "complete" = [all checklist items received]; "after hours" = [19:00–09:00 and Sundays].
5. STAFF MINUTES: recorded by the person at the time, in whole minutes, including system entry. Where that is impractical, a two-day time sample is taken in each period and scaled.
6. SOURCE OF TRUTH: timestamps from [WhatsApp export / system log], not memory. Where the before-period has no log, the sheet is filled from the chat history, and the report says so.
7. WHAT IS NOT MEASURED: revenue, retention, lifetime value. These are modelled in the report from the measured rates and labelled as models.

If the business cannot log the two weeks before, the pilot’s week one is the baseline and the report says the comparison is one week against one week.

The decision rule

Signed before go-live. One sentence each; no reinterpretation after.

DECISION RULE — [workflow] pilot

CONTINUE to a production build if, over the measured period:
- [Primary metric, e.g. median first-reply time] improves from [baseline value] to [target], AND
- Unapproved actions = 0, AND
- Opt-outs honoured = 100%, AND
- Escalation rate is between [5]% and [40]% with no single reason above [50]% of escalations, AND
- Wrong-fact count ≤ [3] in the period, each fixed within 2 working days.

PAUSE AND FIX (one more week at no extra fee) if the primary metric improved but one safety metric failed once.

STOP, keep the artefacts (rulebook, escalation list, data-flow diagram, baseline) if the primary metric did not improve or a safety metric failed twice.

Signed: [owner], [builder], [date]

The one-page pilot report

What the pilot ends with. If a vendor’s report is longer than this, the extra pages are usually where the honesty went.

Pilot report template

The last day of the pilot. Filled from the sheet and the log, not from impressions.

PILOT REPORT — [Business], [workflow], [dates]

1. WHAT WAS BUILT: [one paragraph: channel, systems connected, what the agent does alone / with approval / never]

2. THE NUMBERS
   Metric                      Baseline ([dates])    Pilot ([dates])    Change
   [Primary metric]            [value]               [value]            [±]
   [Secondary metric]          [value]               [value]            [±]
   Staff minutes per item      [value]               [value]            [±]
   Items handled fully by agent    —                 [n] of [N] ([%])
   Escalation rate                 —                 [%]; top reasons: ___
   Unapproved actions              —                 [0]
   Opt-outs received / honoured    —                 [n] / [n]
   Wrong facts found / fixed       —                 [n] / [n]
   Running cost for the period     —                 [amount]

3. WHAT BROKE AND WHAT WE CHANGED: [dated list, incident → cause → change]

4. WHAT THE NUMBERS DO NOT SHOW: [retention, revenue, anything modelled rather than measured, labelled]

5. DECISION AGAINST THE RULE: [continue / pause and fix / stop], because [the rule, applied]

6. IF CONTINUING: scope, price band and timeline for the production build; the pilot fee credited: [amount]. Running cost band at expected volume: [band].

7. HANDED OVER: rulebook v[n], escalation list, approved facts list, data-flow diagram, baseline sheet, the log, account access.

What this means if you are comparing vendors

Ask every vendor for their baseline method and their decision rule before the price. A vendor who measures will have both written down and will be glad you asked. A vendor who does not will offer a dashboard instead, and a dashboard without a baseline is a picture of the present with no before in it.

Our pilot is scoped around this page: the measure is agreed on the free call, the baseline is taken in week one, and the report on the last day follows the template above. If the number does not move, the report says so and the pilot stops, with the artefacts kept.

Cite this page: SigmaticAI, "What a pilot measures, and how the number is taken", appsigmaticai.com/library/what-a-pilot-measures, updated 1 November 2026.

Link to it rather than copying the table; the numbers are refreshed here and the sources are one click away.