The demo went well.
The output was good, everyone in the room agreed it was impressive, and somebody said it was going to save the team a lot of time. That was in March. It is September now, and the work is being done exactly the way it was being done before.
What makes this situation so frustrating is that none of the usual explanations fit. The technology worked. Nobody dug in and refused to use it. There was no blowup, no incident, no moment you can point at and say that is where it went wrong. The effort just never turned into a different way of operating, and if someone asks you what happened, you find that you cannot really say.
There is a hidden assumption underneath all of this, and it is worth dragging into the open. A pilot that proves AI can do the work is not the same thing as proof that the business will do the work differently. Two separate claims. Most pilots only ever test the first one.
So “was the pilot successful?” turns out to be close to useless as a question. The better one is narrower and more annoying to answer: what specifically was supposed to be different afterward, and what stopped it?
In practice that has a fairly short list of answers.
Six stalls that look alike from the outside
Every stalled AI effort looks identical from a distance. Money went in, attention went in, people were genuinely enthusiastic, and operations did not move. Underneath, though, these are six quite different situations, and they need six different responses. Lumping them together is a large part of why the second attempt tends to die the same death as the first.
1. The pilot was never attached to a decision
This is the common one, and it is usually decided before anyone writes a line of code.
The pilot was built to show that AI could handle a kind of work. It was not built to change a particular decision, a particular handoff, or a particular thing somebody is accountable for producing. So when the pilot ends there is nothing to switch over, because nothing was ever scheduled to switch. The work was running alongside the business, not inside it.
From the inside this feels like a success that inexplicably failed to spread. The usual response is to run another pilot.
2. The process downstream still wants the old thing
The model produces good output. The next step cannot use it in the form it arrives in.
Maybe the format is different. Maybe it handles most of the cases well and the reviewer still has to open every single one to find the handful it got wrong. Whatever the specific mismatch, somebody ends up quietly reformatting or re-checking, and the total hours go up instead of down.
The system is doing its job. Nobody changed the workflow that receives what it produces.
3. Nobody accepts the output
The output is good enough to act on. No one has trust in it because of unreliable AI stigma. So every case still lands on the desk of the person the whole thing was supposed to relieve.
That review is not irrational, by the way. Nobody wants to be the first person to wave through something a machine produced and then be wrong about it. Checking everything is the safe posture, and it stays the safe posture until somebody gives a reason to stop. But if checking a result costs about as much as producing one, then a system that is right almost all of the time still saves you nothing.
What makes this a stall of its own is that the distrust is not coming from the results. It arrived before them. You can put a month of accurate output in front of people and still find every case being checked, because what they are guarding against is not this system in particular, it is a general sense that AI output cannot be relied on and that being caught trusting it would look careless. Accuracy alone does not clear that. Somebody has to say out loud which cases no longer need a second pair of eyes, and be willing to own it the day one of those cases goes wrong.
4. The pilot did not run on your actual information
Somebody built a clean sample. They picked the documents, fixed the obvious errors, and left out the weird cases and the records that live in an inbox rather than in a system. This is a reasonable thing to do when you are trying to find out whether an idea has any legs at all.
On that sample, results were strong. On the real pile, with its duplicates and its blank fields and its years of people doing things three different ways and the parts that only exist in one person’s head, results are worse. Sometimes much worse. The model did not change. What you fed it did.
5. It works, and nobody can tell when it is wrong
The system is right most of the time. There is no practical way to know which cases are the exceptions.
That leaves two options and both are bad. Review everything, which puts you straight back in stall three. Or review nothing, which is fine right up until a bad answer reaches a customer, at which point the whole effort tends to get suspended.
It is worth being precise here, because this one gets flattened into the last one a lot. In stall three the checks keep coming back clean and people keep checking anyway. Here the checking is the only thing standing between a bad answer and a customer. Reliability is not just how often a system is correct. It is also whether you can tell that it is wrong while there is still time to do something about it.
6. It works, and it costs more than it saves
Nobody talks about this one, which is odd, because it is the only stall where everything functions exactly as designed.
The system does the work. It also needs watching. It needs correcting now and then. It needs somebody who understands it, and a plan for the week the vendor changes something under you, and a person who answers questions about it when they come up. Add all of that up, set it against the hours you actually saved rather than the hours you projected, and sometimes the number just does not clear.
That is not bad execution. It is an opportunity that was not worth taking, found out late.
Why the model is the wrong suspect
When something stalls, the first fix anyone proposes is technical. Switch models. Wait for the next release. Bolt on retrieval. Fine-tune it. Rebuild the whole thing on a different framework.
Go back through those six, though, and look for the one that a better model fixes.
Stall one is a scoping failure. Two is workflow. Three is a trust problem that was in the room before the system was. Four is an information problem. Five is evaluation. Six is arithmetic. A stronger model leaves every one of them precisely where it was.
And there is a piece of evidence pointing away from the model that everybody in the room already agrees on: the demo worked. A working demonstration tells you the model can do this task, at least on the kind of input it was handed. If capability were the thing holding you back, you would have hit it during the pilot, not after.
This is not an academic distinction, because the technical fix is the expensive one. Swapping models means rebuilding, re-testing, and re-earning everyone’s confidence. If the actual problem is that nobody may act on the output without a review, you can spend a quarter and a great deal of goodwill arriving at exactly where you started, with a better model.
Working out which of the six you are in is nearly always cheaper.
Where the evidence actually lives
You cannot get at this by asking, because all six get described in the same words. “It never really got adopted” covers every one of them.
What separates them is what people did.
Follow the artifact, not the opinion
Pull real cases the system produced output for, and trace what happened next. Did anyone open it? Was it edited before it got used, and how much? Did somebody redo the same work somewhere else? Did it reach the customer or the report or the decision, or did it just stop?
Output that gets rewritten every time is a different problem from output that gets ignored every time. That distinction alone separates stall two from stall one.
Look at the exception path
Ask what happens to a case the system handles badly.
If you get a specific, confident answer, somebody built a real process. If you get a vague one, or if it emerges that nobody can actually identify which cases were handled badly, you are almost certainly in stall five.
Find out whose week changed
If work was genuinely handed over, somebody’s week looks different now. If nobody’s week looks different, then either it never got handed over, or it got handed over and quietly handed back.
That is a factual question with a factual answer, and it beats asking people whether they found the tool useful.
Ask what was supposed to change, and by when
Which measure, which decision, which output. By when. Who would have been the one to notice.
If nobody can answer that without looking at each other first, you are in stall one, and you have been since before the work started.
Compare the pilot data to the live data
Ask who picked the pilot inputs and what got left out. People will usually tell you, and the answer is usually revealing. Stall four tends to be sitting right there in the gap between the sample and the real thing.
Locating your own stall
Here is a rough order to work through, arranged so the questions that eliminate the most possibilities come first. This is Gecko Road’s reasoning rather than any kind of standard, and the point is to narrow things down quickly, not to score anybody.
- Name the decision or output that was supposed to change. If you cannot name it, stop. You are in stall one, and no amount of technical work will touch it.
- Check whether the output gets used as it arrives. If it is routinely reformatted, re-timed, or re-keyed, that is stall two.
- Check whether anyone acts on the output without reviewing it. If everything is still reviewed, ask what the reviewers have actually caught lately. If the checks keep coming back clean and the checking continues regardless, that is stall three, and the thing to settle is which cases are allowed through unchecked.
- Compare the pilot input against the live input. If the pilot ran on a selected sample, some of stall four is in play no matter what else is true.
- Ask how a wrong answer gets noticed. If there is no answer, that is stall five, and trust is never going to form, however accurate the system is.
- Put the real operating cost next to the hours actually saved. If it does not clear, that is stall six, and stopping may well be the right call.
Two caveats. You will often find several of these at once, and the early ones tend to produce the later ones, so working in order usually costs less than chasing whichever symptom is loudest. And finding a stall is not a verdict on whether trying AI was sensible. Several of the six are ordinary operational gaps that would have shown up with any new system.
What this cannot tell you
Knowing where an approach runs out is part of using it properly.
It will not tell you whether the opportunity was worth chasing at all. A pilot can stall for a perfectly clear reason and still have been the wrong problem to pick. That is a different question and a better one to ask before you build.
It will not tell you whether some harder version of the task is within reach of what models can currently do. Some tasks really are capability-limited. The claim here is narrower than that: a working demo followed by an unchanged business is not usually the evidence of it.
It does not cover the case where the effort got stopped for reasons that had very little to do with the work. Budget cycle, reorganization, the sponsor moved to another division. Those are real and common and you will not find them by looking at workflow.
And it will not tell you what the fix costs. Working out that stall three is an authority problem is genuinely useful. Reassigning accountability for a whole class of decisions is an organizational change, and it can easily turn out to be harder than the technical work ever was.
What you get instead is a much smaller question. Not “why did our AI effort not work,” which nobody can answer, but something like “the output is fine, and three people downstream still check every case because nobody has said which ones are allowed through unchecked.” That is a sentence a business can actually do something with.
Sources
- AI Risk Management Framework Core, National Institute of Standards and Technology.
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, National Institute of Standards and Technology.
Evidence note: The six stall patterns, the evidence questions, and the locating sequence are Gecko Road analysis and decision guidance. They are not NIST categories, an industry taxonomy, or a measured distribution of how often each pattern occurs. All examples are hypothetical.