The pattern has become consistent enough to be predictable. A pilot demonstrates well in a controlled setting, secures funding, and then spends the next eighteen months not quite reaching production. Nobody cancels it; it simply stops being mentioned.
Having reviewed a good number of these — some our own — the causes are almost always identifiable before any code is written. These are the six questions we ask in assessment, and the answers we have learned to be sceptical of.
1. What decision does this change, and who makes it today?
If a system produces an output that no identified person acts on, it will not survive its first budget review regardless of accuracy. "Improve customer insight" is not a decision. "Route this claim to the right adjuster queue" is.
The strongest signal is a named person who currently makes the decision manually. That person can tell you what information they use, what makes a case difficult, and — critically — what a wrong answer costs. That last point determines almost everything about the design.
When the answer is "leadership wants AI in this area", the project is a solution seeking a problem. That happens more than anyone likes to acknowledge, and it is far cheaper to name it in week one than in month twelve.
2. Does the data exist, and is it any good?
This is where the largest share of AI projects actually fail, and it is rarely the failure that gets reported. The model performs adequately; the data feeding it does not.
Specific things worth checking before committing budget:
- Does the data contain the outcome you want to predict, historically and labelled?
- Is it available at prediction time, or only after the fact? This distinction kills more models than any other.
- How consistently has it been captured — did the definition or the form change three years ago?
- What proportion of the critical fields are null, defaulted, or entered as free text?
- Can you legally use it for this purpose, given how it was collected?
The second point deserves emphasis. We have seen models built on features that are only populated after the event being predicted has already occurred. They perform superbly in evaluation and are useless in production. This is common enough to be worth checking explicitly on every project.
3. What is the cost of being wrong?
This determines the entire architecture, and it is frequently not asked until late.
A recommendation engine that suggests a suboptimal product has a low error cost — the user ignores it. A system that denies a benefits claim has a high error cost with regulatory and human consequences. These require fundamentally different designs, not the same design with different accuracy thresholds.
High error cost implies confidence thresholds, human review paths for uncertain cases, decision audit trails, and the ability to explain any individual outcome. Those requirements should shape the system from the first design session. Retrofitting them is expensive and usually done badly.
Some use cases fail this question outright. If the error cost is high and no human review step is acceptable operationally, the correct answer may be that the project should not proceed. That is a legitimate conclusion.
4. Would a simpler approach work?
A meaningful share of "AI projects" are correctly solved by a SQL query, a rules engine, or a gradient-boosted model on tabular data. These are cheaper, faster, easier to explain to a regulator, and considerably easier to operate.
A reasonable ordering, from simplest:
- A query or report — the answer already exists in the data
- Deterministic rules — the logic is known and stable
- Classical ML — structured prediction on tabular data, which is where gradient boosting still dominates
- Retrieval over language models — unstructured text where the answer exists in documents
- Generative systems with tool use — genuinely open-ended tasks
Start at the top and move down only when the level above demonstrably fails. Teams that start at the bottom because that is where the interest is spend a lot of money rediscovering that churn prediction is a gradient boosting problem.
5. How will you know if it is working?
Build the evaluation set before building the feature. This single practice separates AI systems that improve over time from those that drift silently.
A workable evaluation approach needs a fixed set of representative cases with known good answers, including hard and adversarial ones; a scoring method appropriate to the task; the ability to run the whole thing automatically on every change; and production monitoring, because offline evaluation and live behaviour diverge.
Without this, quality assessment reduces to someone trying a few examples and forming an impression. That approach cannot detect a change that improves nine cases and breaks one — which is exactly the change that reaches production and causes an incident.
If you cannot describe how you would detect a regression, you do not have a system. You have a demonstration.
6. Who operates this in two years?
AI systems require more ongoing attention than conventional software, not less. Models drift as the world changes. Providers deprecate versions. Costs move with usage patterns. Retrieval quality degrades as document corpora grow.
Before funding, establish who monitors quality in production, who owns retraining, what happens when the provider deprecates the model version you built on, and what the budget looks like at ten times current volume. Token economics that are trivial in a pilot are occasionally alarming at scale.
If the answer to all of these is "the vendor", you have a dependency you should price accordingly.
What this adds up to
None of these questions are about model architecture, and that is the point. The projects that reach production are distinguished by clarity on the decision being changed, honesty about data quality, appropriate design for the error cost, and a plan for operating the result.
The technology is genuinely capable and improving quickly. The failures we see are almost never capability failures. They are projects that started without an answer to question one, and everything downstream inherited that ambiguity.