Why AI pilots die between the demo and the deadline

Why AI projects fail

Most AI pilots fail for reasons that have nothing to do with the model. They fail because nobody wrote down what a good answer looks like, so no one can tell whether a change helped. Because the demo ran on clean, hand-picked files that the real archive does not resemble. Because no one owned the thing once it worked. Because the scope kept widening until it could not be finished. And because the system had no way to say "I don't know", so one confident wrong answer in front of the wrong person ended it.

Those five account for nearly every stalled pilot we get called into, and each is visible early. By week nine of a twelve-week pilot the symptom is usually obvious to the people doing the work and still unsaid in the steering meeting. The last one does the most damage per incident, and gets its own treatment in our guide to stopping an AI system from making things up.

What the 95 percent number actually says

The claim that 95 percent of AI pilots fail comes from "The GenAI Divide: State of AI in Business 2025", published in July 2025 by MIT's Project NANDA. We read it. It does not say that.

Its executive summary says "95% of organizations are getting zero return" on generative AI investment, which is a statement about return, not about pilots dying. The report's own funnel exhibit puts numbers on the pilot question: 60 percent of organizations investigated a custom or task-specific AI tool, 20 percent reached pilot stage, 5 percent reached production. Read off those figures, roughly one organization in four that piloted a custom tool got one into production. Not one in twenty.

The caveats are the report's own. It is labelled "Preliminary Findings" and rests on 52 organizations, a review of more than 300 publicly disclosed AI initiatives, and 153 survey responses "collected across four major industry conferences". It counts an implementation as successful when "users or executives have remarked" on a "marked and sustained productivity and/or P&L impact", so success is a remark in an interview rather than a measurement. It says its own figures are "directionally accurate based on individual interviews rather than official company reporting", and that its views are the authors' own and "do not reflect the positions of any affiliated employers". The report also had a short public life. The Wayback Machine's capture index shows the PDF serving a 200 at nanda.media.mit.edu/ai_report_2025.pdf on 18 August 2025 and returning a 404 at the same address later the same day. As of August 2026 that address redirects to a group overview page that does not mention the report at all, and we have found no retraction or correction. We read the Internet Archive copy captured on 18 August 20251. We are not the first to check this: 80,000 Hours published a similar correction2 and reached the same conclusion about how the figure is quoted. A statistic that is repeated everywhere, sourced to a document that is no longer published, is worth being careful with.

Two figures are better documented. Gartner predicted on 25 June 20253 that "over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls", and estimated that only about 130 of the thousands of vendors selling agentic AI are real. That is a forecast, not a measurement. S&P Global Market Intelligence reported in October 20254, from a 451 Research survey of 1,006 respondents, that the share of companies abandoning most of their AI initiatives rose from 17 percent to 42 percent year over year, with the average organization scrapping 46 percent of its proof-of-concept projects before production. That survey was fielded in late 2024.

A lot of pilots do die. The exact rate is not well established, and no published number tells you anything about yours. The patterns below do.

Five ways a pilot dies

1. Nobody can tell whether a change helped

Someone reports a bad answer, an engineer changes a prompt or a retrieval setting, the bad answer goes away, and the change ships. Three weeks later two other things are quietly worse and nobody connects them. Progress is measured by whether the last complaint was fixed, which means the project has no memory and no direction.

The root cause is that no one wrote down what a good answer is, in a form a machine can check. Without that, every change is a coin flip dressed as engineering judgement.

What prevents it is a graded test set built in week one, before the first feature: a few dozen real questions with the answers a competent human would give, run automatically on every change, with a threshold that blocks a release when it drops. It is unglamorous, and it does more for a pilot than anything else on this list. The practical version is in how to evaluate a RAG system.

2. The demo ran on data somebody chose

The demo used forty documents. They were recent, clean, consistently named, and picked by the person building the demo. Week nine, the system is pointed at the real archive: scans of faxes, three naming conventions, a folder from 2019 that nobody will vouch for, spreadsheets where the real answer lives in a merged cell. Accuracy falls off a cliff and the team blames the model.

The root cause is that the pilot tested the model when the risk was in the data. A curated demo is a rehearsal, not a pilot.

What prevents it is running against the real corpus in the first fortnight, ugly parts included, and treating what breaks as the actual project plan. The S&P Global survey above supports this: organizations with below-average project failure rates were more likely to weigh data availability when choosing use cases, 54 percent against 42 percent.

3. Nobody owns it in production

This one is the cruellest, because the software works. Everyone agrees it works. It still does not ship. Ask who gets paged when it returns nonsense at 2am on a Saturday and the room looks at each other. The security review has not started. There is a budget line for building it and none for running it. The pilot ends, the demo link stays up a while, and it dies of neglect rather than failure.

The root cause is that the pilot was scoped to prove something, and proving something creates no obligation to operate it.

What prevents it is naming the production owner and the operating budget before the build starts, not after the demo lands. Our own engagements are built that way: a fixed-scope first piece, then a flat monthly rate to operate it, with the handover in the plan from the start. The sequence is in how an engagement runs.

4. The scope has no edges

It started as "answer questions about our leases". By week nine it also flags anomalies, drafts the follow-up email, and needs to talk to a system nobody has API credentials for. Every addition was reasonable on the day it was made. The result is a project that cannot be finished, only abandoned, because the finish line moved four times.

The root cause is the absence of a written definition of done: when nobody has said which questions the system must answer, every question is in scope.

What prevents it is a one-page list, agreed before the build, of the questions it must answer well and the questions it is explicitly allowed to refuse. The refusal list matters more than the answer list: it is what lets you decline a reasonable request without an argument. Gartner's stated cancellation reasons, escalating costs and unclear business value, are usually this failure showing up on a finance report a year later.

5. It has no way to say "I don't know"

The pilot is good. Not perfect, but good. Then in the steering committee demo an executive asks something the data does not cover, and the system invents a figure that sounds entirely plausible. Someone in the room knows it is wrong. The demo continues politely and the project is finished.

That is the asymmetry nobody prices in. A system right 90 percent of the time and silent about the rest is useful. One right 90 percent of the time and confident about the rest is worse than nothing: a reader who cannot tell which is which has to check everything, and checking everything is the job you were trying to remove.

The root cause is a system with no abstain path and no citations. It was built to always produce an answer, so it always does.

What prevents it is grounding and an honesty gate from the first week: answers pulled from real records, a visible link from every claim to its source, and a hard refusal when retrieval does not support an answer. On one production system, over more than 29 million indexed chunks, we hold abstain precision at 1.000: every time it declined to answer, it was right to decline. Grounding does not make mistakes impossible. It makes them visible and traceable instead of silent, which is the difference between a system people check and one they abandon.

The week nine checklist

Hold your own project against this list.

The decision rule: three or more no's at week nine and the pilot is not behind schedule, it is on the wrong track. Stop adding features and close the no's first.

What a pilot that ships does differently

Very little, and all of it early. It picks one workflow with clear edges. It writes down what a good answer looks like before the first line of code. It runs on real data in the first fortnight, names its production owner while there is still time to change their mind, and ships with a refusal path.

None of that requires a better model. It requires deciding, in week one, that you are building something to run rather than something to show.