What an AI Pilot Should Prove Before You Fund Production

Most AI pilots end with a good demo and a hard question. The system worked on the examples everyone chose, the room was impressed, and now someone has to decide whether to fund the real thing. Without an agreed answer to what the AI pilot was meant to prove, that decision turns into a debate about impressions rather than evidence.

The numbers suggest this happens often. Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value (Gartner, July 2024). Research from MIT’s NANDA initiative, reported by Fortune, found that only about 5 percent of enterprise generative AI pilots achieved rapid revenue acceleration, while most delivered little measurable impact on the profit and loss (Fortune, August 2025).

Neither finding says AI does not work. Both say that many pilots are not designed to produce a funding decision. This guide sets out what a pilot should prove, how to measure each part, and how to agree the go or stop rule before any work begins.

The short answer

A pilot earns production funding when it shows five things on your own material: it handles your real input, its accuracy can be measured and meets an agreed bar, the people it is for actually use it, it fits the systems and handovers around it, and it costs less per good result than the way the work is done today. Agree the thresholds for each before the pilot starts, in writing. If the pilot meets them, fund it. If it misses them, you have learned something cheaply, which is what a pilot is for.

Proof of concept, pilot and production are different questions

Much of the confusion starts with treating these three stages as one. Each answers a different question, and each needs different evidence.

	Proof of concept	AI pilot	Production
The question it answers	Can this be done at all?	Does it work here, for these people, at an acceptable cost?	Does it keep working, every day, as things change?
Data	Sample or cleaned examples	Your real input, including the messy cases	Everything, as it arrives
Users	The build team	A small group of the people who will use it	Everyone the process touches
Integration	Usually none	The systems the workflow depends on, often read-only	Full, with permissions, monitoring and support
What it produces	Confidence that the idea is possible	A measured result and a funding decision	The business outcome itself
Typical failure	Mistaken for a pilot	Run without a decision rule, so it never ends	Nobody owns it after launch

A proof of concept that impresses is not evidence for production. It shows the model can do the task on chosen examples. The pilot is where you find out whether it can do the task on yours.

The five things an AI pilot should prove

1. It works on your real input

Demos run on clean examples. Your business runs on scanned invoices, forwarded email chains, half-completed forms and documents in three versions. A pilot that never meets that material has not tested the thing that usually breaks.

Gartner puts the stakes plainly: it predicts that through 2026, organizations will abandon 60 percent of AI projects unsupported by AI-ready data (Gartner, February 2025). The pilot is the cheapest place to find out whether your data is ready. Take a representative sample of real input, including the cases your team complains about, and keep it aside as the test set. If the system only works once the input has been tidied by hand, you have found a data project that has to come first, and that is a useful result.

2. Its accuracy can be measured, and it meets the bar

“It looked right” is not a measure. Before the pilot, decide what a correct result is for this task, and how you will check it: against a reviewer’s judgment, against a known answer, or against source material. Then decide how good is good enough. A first draft that a person edits can tolerate more error than an action that changes a customer record.

The strongest pilots go a step further and build the measurement into the system, so it keeps running after launch. That is the difference between a test you pass once and a control you keep. When we built an internal AI platform for Pfizer, where junior researchers consult AI avatars grounded in senior researchers’ own datasets and documents, every answer was designed to be checked against that material and shown with a match percentage and a rating of good, average or below average. Answers rated below average are flagged and reviewed by the team. The researcher sees the answer and its evaluation side by side, so trust is earned answer by answer rather than assumed.

3. The people it is for actually use it

A system that is accurate but unused has proved nothing about value. The MIT research points to a “learning gap” rather than model quality as the core problem: general-purpose tools work well for individuals but struggle inside companies because they do not adapt to how the work is actually done.

So the pilot should put the system in front of a small group of real users, inside their normal working day, and watch what happens. Do they come back to it without being reminded? Do they trust the output enough to act on it, or do they redo the work alongside it? What do they ask for that it cannot do? Usage over the pilot period, and what users say when the novelty has worn off, tell you more than any satisfaction score collected at the end of a demo.

4. It fits the systems and the handover

An AI step that works on its own but cannot read from the CRM, write to the ticketing system or hand a case to a person has not been piloted. It has been demonstrated. The pilot should connect to the systems the workflow depends on, even if only with read access, and it should test the handover: what happens when the system is unsure, who receives that case, and with what context.

Handover is often where the value sits. For Wordibly, a US transcription company, we built a platform that combines AI transcription with human correction across three service tiers, with Zoho, QuickBooks and Zapier connected. The AI does not replace the transcriptionist; it changes where their time goes. Client-reported, the result was 60 percent faster turnaround. A pilot that had tested the AI step alone would have missed the part that made the difference.

5. It costs less per good result than the current way

Model charges are the visible cost. The full cost includes infrastructure, the time people spend reviewing and correcting outputs, the rework when something goes wrong, and the support the system will need once it is live. The useful comparison is cost per accepted result: total operating cost divided by the number of results that met the agreed bar.

Compare that figure with the cost of doing the same work the current way, including the people and the delay. A pilot is the first moment you can measure this on real volume, so record it. If the AI version costs more per good result and does not bring a benefit worth paying for, such as speed or capacity, production will not fix the economics. For the build side of the budget, see how much it costs to build an AI agent.

Agree the funding decision before the pilot starts

The single change that saves most pilots is writing the decision rule down first. When the thresholds are agreed in advance, the result decides the next step. When they are not, the pilot drifts, the scope grows and the final meeting becomes a negotiation.

The table below is a template, not a benchmark. The example thresholds are there to show the shape of a rule; set your own from what the process needs and what it costs today.

What to prove	How to measure it	Example go threshold	Stop signal
Works on real input	Share of real test cases the system can process without manual clean-up	Handles the agreed share of the test set, including the hard cases	Only works after the input is tidied by hand
Accuracy	Results checked against reviewers, known answers or source material	Meets the accuracy bar agreed for this task, with errors that are caught before they cause harm	Errors that reach customers or records, or no agreed way to check
Adoption	Repeat use by pilot users over the pilot period	Users return to it unprompted and act on its output	Users redo the work alongside it or stop using it
Fit and handover	Connected to the core systems; unsure cases reach the right person with context	Handover works end to end on real cases	Cases fall between the system and the team
Cost per good result	Total operating cost divided by accepted results, against today’s cost	Lower than today, or similar with a benefit worth paying for	Higher than today with no offsetting benefit

Name one person who makes the call when the pilot ends, and agree what each outcome leads to: fund production, run one more defined round on a specific gap, or stop. “Extend the pilot and see” should be the exception, with a reason and an end date.

How big a pilot should be

Small enough to finish, real enough to count. In practice that usually means one workflow, one team and one clear outcome, run on real data for long enough to see normal variation rather than a good week. If you are still choosing which workflow to start with, our checklist of which business processes to automate with AI first helps narrow the field.

Two warning signs on scope are worth watching. A pilot that needs months of build before anyone can use it is a project in disguise; it is carrying risk the pilot was meant to remove. A pilot that tries to cover several workflows at once usually proves none of them well. Narrow scope is not a lack of ambition. It is what makes the result readable.

Signs a pilot will not survive production

Some pilots look successful and still fail at the next step. These are the patterns that most often explain why.

1. The test data was chosen, not sampled. Results on examples picked to work say little about the rest.
2. Accuracy was judged by the people who built it. The users who will rely on it, or an agreed answer set, should do the judging.
3. Nobody tested what happens when it is wrong. Every system will be wrong sometimes. Production needs a path for that, not a promise that it will not happen.
4. The cost was measured on the model bill alone. Review time, rework and support are part of the cost from the first day.
5. No one owns it after the pilot. Models change, data drifts and the business changes around the system. If no one is responsible for keeping it right, it will not stay right.
6. The model cannot be swapped. If the design depends on one provider’s model, the system is tied to that provider’s pricing and release schedule. A structure that lets the model change protects the investment.

What we have learned building systems that last

We still build and run several of the platforms we have delivered for clients, some since 2022, so we see what happens long after the pilot report is filed. Three lessons have held across that work.

Treat evaluation as a feature, not a phase. The checks that convince you in the pilot are the same checks that keep the system honest in production. On the Pfizer platform, the answer scoring that shows researchers how far to rely on each response is part of the product itself, delivered in three months by three Appson AI specialists and in use since 2024.

Design the handover before the model. Where the AI stops and a person takes over decides most of the value, as the Wordibly platform shows. Getting that boundary right matters more than the last few points of model accuracy.

Plan for the second year. Releases, data refreshes and model changes are the normal life of an AI system. A pilot that ends with a clear owner and a plan for that work has already answered the question most production projects forget to ask.

Before you sign off an AI pilot

Whether you are reviewing a vendor’s proposal or your own team’s plan, these questions are worth answering on paper first.

1. What decision will this pilot make? If the answer is not “fund production or not”, the pilot needs a sharper purpose.
2. Which real input will it run on? Agree the test set, and keep the hard cases in it.
3. How will a correct result be checked, and by whom? Name the method and the reviewers.
4. What are the go thresholds and the stop signals? Write them down before the build starts.
5. Who will use it during the pilot? Real users, in their normal working day.
6. Which systems will it touch, and with what permissions? Read-only is a sensible start.
7. What will production cost to run, and who owns it? If nobody can answer yet, that is the next thing to find out.

Frequently asked questions

What is the difference between an AI proof of concept and an AI pilot?

A proof of concept tests whether something can be done at all, usually on sample data with no real users. A pilot tests whether it works in your business: on your real input, with the people who will use it, connected to the systems around it, at a cost you can measure. A proof of concept builds confidence; a pilot should produce a funding decision.

How long should an AI pilot run?

Long enough to see normal variation in the work, not just a good week, and short enough that the decision does not drift. The scope matters more than the calendar: one workflow, one team and one outcome, with the end date agreed at the start.

What metrics should an AI pilot measure?

Five, at minimum: how much real input it handles without manual clean-up, how often its results are correct against an agreed check, whether pilot users keep using it, whether it connects to the core systems and hands unsure cases to the right person, and its cost per accepted result compared with the current way of doing the work.

Why do so many AI pilots fail to reach production?

The common causes named in the research are poor data quality, unclear business value, rising costs, weak risk controls and tools that do not fit how the work is actually done. Many of these show up in a pilot that runs on chosen examples, is judged by its builders and has no agreed decision rule.

Should we build the pilot ourselves or use a partner?

Either can work. The MIT research found that purchased tools and partnerships succeeded more often than internal builds, but the deciding factor is usually the design of the pilot rather than who builds it. If you use a partner, ask them to agree the thresholds and stop signals with you in writing before any work begins.

Funding the right next step

A good AI pilot is not a smaller version of the final system. It is a deliberate test with a decision at the end of it: real input, measured accuracy, real users, working handovers and a cost per good result you can compare with today. Agree what success looks like before the build starts, and the pilot will tell you whether to fund production, fix one gap or stop, which is exactly what you needed it to do.

That is how our AI development services team scopes every pilot: one workflow, on your own data, with the thresholds written down before any work begins. Tell us about one process, what comes in, who handles it and what has to come out, and we will tell you plainly whether it needs AI and what a pilot should prove. Talk to our AI team to start.