An AI proof of concept proves something only when it is built to answer one specific question with a yes or a no. Scope it around a single hypothesis ("an agent can correctly code 85% of our supplier invoices without human edits"), agree the success criteria before any building starts, test on your real, messy data including the hard cases, cap it at a few weeks, and schedule the go, adjust or stop decision on day one. A PoC without those pieces turns into a demo that impresses everyone and decides nothing.
This post walks through each ingredient, the most common ways PoCs go wrong, how big and how long one should be, and a one-page brief template you can fill in before talking to any vendor or developer.
Why do so many AI proofs of concept prove nothing?
Because they're scoped to show that AI can do something, which almost nobody doubts anymore. Of course a language model can read an invoice, summarize a call or draft an email. The questions that actually block a business decision are narrower and harder:
- Can it do our version of this task, with our documents and our exceptions?
- How often is it wrong, and what does being wrong cost?
- How much human checking does it still need?
- Will the people who do this work today trust and use it?
- What would it cost to run at our real volume?
A PoC that answers none of those ends with a meeting where everyone agrees the demo was impressive, and then the project drifts. We see this pattern often enough that it's one of the mistakes businesses keep repeating with AI projects.
What should an AI proof of concept actually test?
Start with a hypothesis written as a sentence a finance person could check. Good hypotheses have a task, a measurable outcome, a threshold, and a condition.
| Weak hypothesis | Strong hypothesis |
|---|---|
| "AI can help our support team." | "An AI agent can fully resolve at least 40% of 'where is my order' tickets using our order system, with zero incorrect delivery dates in a 300-ticket test." |
| "Let's see if AI can read our contracts." | "AI can extract renewal date, notice period and liability cap from our last 200 vendor contracts with 95% field accuracy, flagging the ones it isn't sure about." |
| "Try an AI sales assistant." | "AI-drafted follow-up emails, reviewed by reps, cut average drafting time per lead from about 10 minutes to under 3, with reps rating at least 70% as 'send with minor edits.'" |
The numbers in the right-hand column are illustrations; pick thresholds that reflect your economics. The point is that each strong version can fail. If a hypothesis can't fail, it isn't testing anything.
Choose the riskiest assumption to test, not the easiest. If the uncertain part of your project is whether the AI can handle handwritten delivery notes, the PoC should focus on handwritten delivery notes, not the clean typed ones.
How do you set success criteria before building?
Write them down and get sign-off from the person who will make the final decision. Agreeing criteria after seeing results invites everyone to move the goalposts toward whatever happened.
Useful criteria usually cover four angles:
- Quality: accuracy, error rate, or the share of outputs a reviewer accepts without edits. Define what counts as an error. A wrong date on an invoice is different from a slightly awkward sentence in an email.
- Safety: the maximum tolerable rate of serious mistakes. For many workflows this is close to zero, which is why the AI's ability to say "I'm not sure" and route to a person matters as much as its accuracy.
- Efficiency: time saved per item, or the share of items handled without human effort. Measure the current baseline first, or you'll have nothing to compare against. Our guide to measuring AI ROI without fooling yourself covers baselines in detail.
- Cost to run: rough per-item model and infrastructure cost at production volume, so you know whether success would be affordable.
Also write down the stop condition. "If field accuracy is below 80% after two rounds of improvements, we stop or rethink the approach." It feels pessimistic, but it's what makes the PoC worth paying for.
Why does the PoC need real data and hard edge cases?
Demos are usually run on clean examples. Production is not clean. The gap between the two is where AI projects fail.
Pull a real sample from the last few months, not hand-picked favorites. Then deliberately build an edge-case set: the 30 to 50 examples your team groans about. For an invoice PoC, that might be scanned invoices at an angle, credit notes, invoices in two currencies, multi-page invoices with totals on page three, and vendors who put the PO number in the notes field. For a support PoC, it's angry customers, multi-issue tickets, messages in mixed languages, and requests that sound routine but aren't.
Score the AI on the general sample and the edge-case set separately. A system that scores well overall but fails badly on the hard cases needs a good escalation path, not just a better prompt. For more on building a test set that reflects reality, see how to know an AI agent is ready for real work.
Sort out data access early. Getting permission to use real customer data, anonymizing it if needed, and getting a read-only connection to the relevant system often takes longer than the AI work. Start that on the first day.
How long and how big should an AI proof of concept be?
Small and short. For most SMB use cases, two to six weeks is enough, and the build should be the minimum needed to test the hypothesis.
A PoC usually includes the core AI step, a realistic data sample, a simple way for reviewers to see and score outputs, and basic logging of cost and errors.
It usually excludes polished user interfaces, single sign-on, full integration with every system, scale testing, and coverage of every task variant. Those belong to the pilot and production phases. If you find yourself building them during the PoC, scope has crept.
A typical timebox, for illustration:
- Week 1: confirm the hypothesis and criteria, secure data access, measure the current baseline, assemble the sample and edge-case sets.
- Weeks 2-3: build the minimal version, run it on the sample, review results with the people who do the work, improve.
- Week 4: final run on a held-back set the team hasn't tuned against, score it, estimate running costs, write up the result.
Holding back a set you never tuned against is important. If you only test on examples you've been iterating on, your accuracy number will look better than reality.
What should happen at the end of the proof of concept?
A decision, in a meeting that was booked at the start. There are only three honest outcomes:
- Go: the criteria were met. Move to a limited pilot with real users and real volume. Our post on getting AI pilots into production covers that next stage.
- Adjust: results were close, and you know why they fell short (a specific document type, missing data, an unclear policy). Run one more focused round with a new timebox.
- Stop: the approach doesn't work well enough for this task right now. That's a useful result. You spent a few weeks instead of a few quarters finding out.
Write a short results memo: the hypothesis, the criteria, the scores, examples of failures, estimated running cost, and the recommendation. That memo is the actual deliverable of a PoC, more than the code.
What are the most common AI PoC anti-patterns?
- The demo PoC: built to impress stakeholders, tested on a handful of examples, no criteria. It always "succeeds" and never leads anywhere.
- The forever PoC: no end date, so it keeps expanding. Three months in, it's an unfinished product.
- The wrong-process PoC: automating a task that's rare, low-value, or about to change. Run a quick process mapping exercise first to pick a task worth testing.
- The orphan PoC: nobody on the business side owns the outcome, so nobody acts on the result.
- The vendor-sandbox PoC: run entirely on the vendor's sample data, which tells you about their data, not yours.
- The users-not-invited PoC: the people who'd use it daily see it for the first time at the end, and reject it.
A one-page AI proof of concept brief you can copy
Fill this in before you talk to a developer. If you can't complete it, you're not ready to start building yet.
- Business problem: what's slow, costly or error-prone today, in one or two sentences.
- Hypothesis: task, measurable outcome, threshold, condition.
- Current baseline: volume per month, time per item, current error rate (rough is fine).
- Success criteria: quality, safety, efficiency, cost to run.
- Stop condition: the result that means "don't proceed."
- Data: which sample, how many items, who approves access, anonymization needed.
- Edge-case set: the hard examples you'll include on purpose.
- Out of scope: what this PoC will deliberately not build.
- People: decision owner, two or three reviewers from the team doing the work, technical owner.
- Timebox and decision date: start date, end date, decision meeting booked.
Frequently Asked Questions
How much does an AI proof of concept cost?
It depends on data access and complexity, but a focused PoC is a fraction of a full build. The biggest cost drivers are messy data and integrations, which is why PoCs should use exports or read-only access where possible. Be wary of PoCs priced so cheaply that they can only be run on sample data.
What's the difference between a proof of concept, a pilot and an MVP?
A proof of concept tests whether an approach works on your data at all. A pilot puts a working version in front of real users at limited volume to see whether it works in practice. An MVP is the smallest product version you'd actually ship. Each answers a different question, so don't skip straight from demo to rollout.
Can we run an AI proof of concept without sharing sensitive data?
Often yes. You can anonymize or mask records, use a provider that doesn't train on your data, keep processing in your own cloud account, or start with the less sensitive part of the workflow. Plan this in week one, because data approvals tend to be the slowest step.
What accuracy should an AI proof of concept target?
There's no universal number. It depends on what an error costs and whether a person reviews the output. A drafting assistant with human review can be useful at modest accuracy; an agent that posts entries to your ledger unattended needs very high accuracy plus reliable "I'm not sure" detection.
Who should be involved in an AI PoC?
At minimum: a decision owner with budget authority, two or three people who do the work today and will review outputs, and whoever builds it. The reviewers are the most overlooked and the most important, because their judgment defines what "correct" means.
Want a PoC that ends in a clear decision?
If you have an AI idea and want to know whether it'll hold up on your real data, tell us the problem you're trying to solve. We'll help you write the hypothesis and criteria first, then build only what's needed to test them. See how we work on agentic AI development projects, or start with our no-hype guide to AI for business.