Foxbox Digital is an award-winning engineering-focused product development agency.

Inside the Box Foxbox Digital Blog

Versapay: Digital Payments Mobile Strategy

Know when rapid AI-driven iteration becomes a liability. Learn the key signals your codebase needs structure—and how to transition to real engineering without slowing product momentum.

When to Stop Iterating and Start Engineering | Foxbox Digital 

There's a moment in every AI-assisted product build where the superpower starts to become a liability.


For a while, everything is momentum. You describe what you want, it gets built. You find a problem, you fix it. You add a feature in an afternoon that would have taken a week before. The velocity feels extraordinary — because it is.


And then something shifts. Changes start having unexpected side effects. The codebase develops a gravity that slows things down. A simple feature request turns into a three-day slog. A bug gets fixed and two others appear. The thing that made development fast — iterating quickly without worrying too much about structure — starts working against you.


This is the transition point from building to engineering. Most founders feel it before they can name it. This post is about how to recognise it, what it means, and what to do about it.

When you're building with AI tools — Cursor, Lovable, Replit, or any of the other current leaders in the space — the core workflow is iterative. You describe what you want, you evaluate what you get, you refine. The feedback loop is fast. The results are often surprisingly good.


This workflow is optimised for exploration. You're figuring out what the product should be by building it. The velocity comes from not having to know everything upfront — you can try things, see if they work, and change them if they don't.


What it's not optimised for is accumulation. Every time you add something, modify something, or fix something, you're making a deposit into a codebase that has to support everything you've put in it so far. When the codebase is small and simple, this is fine. As it grows, each change touches more of the system, and the probability of unintended consequences grows with it.


The point at which the cost of unintended consequences starts exceeding the benefit of rapid iteration is the point to switch modes.

Signal 1: You've stopped trusting deployments

In the early days, deploying was fine. You'd push a change, it would go out, the thing would work. Now there's a knot in your stomach when you deploy. You push something out and immediately monitor the error logs. You've been burned before.


This is the system telling you something about its own fragility. The code has accumulated enough state and enough complexity that changes are no longer isolated. They interact.


Signal 2: You're spending more time on bugs than features

Track your last two weeks of work. If more than half of your time is going to fixing things that broke — rather than building things that didn't exist — the codebase has crossed a threshold. You're maintaining it rather than building on it.


This ratio tends to get worse over time if the underlying structure isn't addressed. It doesn't naturally correct itself.

Signal 3: The codebase has knowledge that only exists in the code

This is a subtler one. Ask yourself: if a competent engineer who had never seen this project opened the repository today, how long would it take them to understand how it works?


If the answer is "they'd basically have to reverse-engineer it" — there's no documentation, the code is difficult to follow, the architectural decisions aren't visible anywhere — you've built something that is brittle in a human sense as well as a technical one. It depends on the people who built it to maintain it, rather than on its own coherence.

Signal 4: You're about to put real load on it

You have a launch planned. A press feature is coming out. You're closing a partnership that will bring in a wave of new users. You're about to deliberately grow.


This is a prospective signal rather than a reactive one — you're not yet broken, but you're about to be exposed. This is the best time to make the transition, because you can do it deliberately rather than in response to a crisis.

Signal 5: Money is moving through the system

If your product has real financial transactions — payments, subscriptions, contracts, invoicing — the standards are different. Bugs in a feature are annoying. Bugs in billing are existential. Inconsistent state in a user's profile is recoverable. Inconsistent state in a payment record might not be.


The moment real money is involved is the moment to demand engineering discipline, regardless of whether you've hit the other signals.

What "Starting to Engineer" Actually Means

This is the part that founders sometimes misunderstand. The transition from iterating to engineering isn't:


Stopping product development to do a rewrite

Hiring a team of people who will spend six months on infrastructure before shipping anything

Adopting a slow, bureaucratic development process

Establishing a foundation you can build on.

 This means addressing the structural issues — the missing indexes, the absent error handling, the synchronous operations that should be async, the secrets in the codebase — so that future development is faster and more confident, not slower.

 Good documentation, clear architectural decisions, code that can be understood by someone who wasn't there when it was written. This isn't a luxury — it's what enables you to move fast at scale, because changes don't require the full context of everyone who's worked on the system.


 Not necessarily 100% coverage of everything — that's often a waste of effort at this stage. But tests around the things that matter: the critical paths, the financial logic, the edge cases that have burned you before. Tests that give you confidence to deploy.

 Your deployment process, your monitoring, your alerting, your on-call runbooks — these aren't supporting characters. When production breaks, they're the whole show. Getting them right isn't a distraction from building the product; it's what allows you to build the product sustainably.

There are two ways to get this wrong, and they're mirror images of each other.


The first is staying in iteration mode too long. You keep shipping, the codebase accumulates fragility, you hit a scaling moment unprepared, and you end up doing emergency remediation under pressure. This is more common. It tends to happen because iteration feels productive and engineering feels like it's not building anything.


The second is switching to engineering mode too early. You have a small product, a handful of users, and you start talking about microservices, Kubernetes, and multi-region deployment. This is how startups burn six months on infrastructure for a product that isn't validated yet. The irony is that the people making this mistake often do it in the name of "doing it properly."


The discipline is in timing. You want to switch modes at the point where the system's current structure will actively slow you down — not before, not after.


That point is different for every product, every team, every stage. But it's almost never at the beginning, and it's almost always before the crisis.

If you've recognised the signals above and want to make the transition intentionally, here's what it looks like in practice:

 Understand what you have before deciding what to do about it. What are the structural problems? Where are the risks? What will break first, and under what conditions? This doesn't have to be exhaustive — you're looking for the high-impact issues, not a comprehensive audit of everything.

 Not everything needs to be fixed at once. Security issues (exposed secrets, missing auth checks) are first. Infrastructure reliability (error tracking, monitoring) is next. Performance and scalability come after that. Structural improvements (test coverage, code organisation) can be layered in over time.

 The transition shouldn't require a freeze on feature development. The best approach is usually a parallel track: some capacity dedicated to structural improvement, some dedicated to product development. The proportion shifts as the structural issues are resolved.

 If the people who built the MVP aren't engineers — or if they are, but they're needed on product — there's real value in bringing in a team who does this work specifically. Not to take over the codebase, but to set it up so that your team can maintain it confidently.

There's a version of this story where founders ignore the signals until they can't anymore. The product gets big enough that real users depend on it. The team grows. Revenue is meaningful. And then something breaks in a way that takes days to fix, or a new engineer joins and can't get the development environment working, or technical due diligence for a Series A reveals a codebase that makes investors nervous.


At that point, the options narrow. You're either doing remediation under pressure, or you're doing it under the scrutiny of investors and new team members. Neither is ideal.


The transition is much easier when you choose to make it. The codebase doesn't have to be a liability. It's fixable. But it takes intention — and the earlier you bring that intention to it, the less it costs.

When to Stop Iterating and Start Engineering

A practical breakdown of the multi-agent AI engineering model Foxbox uses in production: how orchestrators, parallel subagents, and critic agents work together to build and verify software at scale.

The Three-Agent Model: How we Accelerate Agentic Engineering

There's a version of AI-assisted development that most people picture: a developer typing a prompt, reading the output, and deciding whether to keep it. One human, one model, back and forth. It's better than writing everything from scratch. It's not how we work.

At Foxbox we use what's become known in the engineering community as the three-agent model. It's a multi-agent architecture pattern, a way of structuring AI agents in a hierarchy so that complex software work can be decomposed, executed in parallel, and verified before it ships. The terminology around this pattern is still settling; you'll see it called orchestrator-subagent, agentic loop, or simply multi-agent workflow depending on who's writing about it and when. The underlying structure is consistent regardless of what it's called, and that structure is what's worth understanding.

This post explains the model, where it comes from, how we apply it in practice, and what it looks like on a real engagement.

The three-agent model didn't originate with any single company or tool. It emerged from multi-agent systems research and has been formalized over the past couple of years by engineering teams at Anthropic, Google, Microsoft, and OpenAI as 

 became capable enough to make it practical.

The core idea is older than any of those companies: in any well-designed system, generation and verification should be separate concerns. You don't have the same person write the code and sign off on its correctness. You don't have the same process that builds the artifact also audit it. This separation of concerns is foundational to software engineering. It's why code review exists, why QA is a distinct function, why tests are written independently of the code they test.

The three-agent model applies that principle to AI agents. Instead of one agent trying to do everything — plan, implement, check, refine — you give each role to a specialized agent with its own context, instructions, and scope.

Microsoft's Azure Architecture Center, Google's Agent Development Kit documentation, and Anthropic's guidance on multi-agent workflows all describe variations of the same pattern. The names differ: orchestrator-subagent-evaluator, generator-critic loop, maker-checker. The structure holds.

The orchestrator is the agent at the top of the hierarchy. It receives the task — a feature, a bug fix, a module to build — and handles the planning work: decomposing the problem, deciding what needs to happen in what order, assigning work to subagents, and synthesizing results.

The orchestrator does not write code directly. Its job is coordination and judgment. It maps the work, specifies what each subagent should produce, and determines when output meets the standard to move forward.

This mirrors how a senior engineer actually operates. When a complex feature lands in a sprint, a senior engineer doesn't immediately start coding. They break it down. They think about dependencies. They decide what the interface should look like before anyone touches an implementation. That planning work is what the orchestrator does.

The builders are the hands. The generators. Each one receives a bounded, well-defined task from the orchestrator and executes it in its own isolated context window.

The isolation matters. One subagent working on the data access layer doesn't need to know about the UI. Another subagent writing the API schema doesn't need to know about the deployment configuration. Narrower context produces more focused output — the agent isn't holding irrelevant information while trying to do a specific job.

Subagents can run in parallel. If the orchestrator determines that the database schema, the API endpoints, and the authentication middleware are independent of each other, it can assign all three to separate builder subagents working simultaneously. This is where significant time savings come from in practice: tasks that would be sequential in a traditional development workflow can run concurrently.

The critic is the quality gate. After a builder produces output, the critic reviews it against a defined set of criteria before the result goes back to the orchestrator.

The critic's job is not to rewrite the output. It's to evaluate it: Does this do what was asked? Does it meet the acceptance criteria? Are there edge cases missing? Is there a security issue? Does it conflict with anything in the broader codebase?

If the output passes, the critic signs off and the orchestrator proceeds. If it fails, specific feedback goes back to the subagent for revision. This loop continues until the output meets the standard, or until a maximum iteration count triggers escalation to a human reviewer.

What this creates is a quality gate that catches problems before they compound. A builder writing a database query that would cause an N+1 problem gets corrected before that query is wired into the API layer. A subagent generating an API endpoint that exposes a sensitive field without proper access controls gets flagged before it ships.

The practical reason is context management. A single AI agent working on a large feature has to hold the entire task in a single context window. As that window fills, quality degrades — the model loses track of earlier constraints, makes decisions that contradict earlier ones, and produces output that works in isolation but conflicts with the rest of the system.

Distributing work across multiple agents, each with a focused context, prevents this. The orchestrator's context stays lean because it only manages plans and summaries, not implementation details. Each builder’ context is scoped to its specific task. The critic's context is structured around evaluation criteria, not generation.

There's also a quality argument independent of context. Generation and evaluation are different modes of work. An agent that wrote the code has already committed to a set of assumptions. A separate agent reviewing that code — with different instructions and no prior context — will catch things the generator missed. This is why pair programming works. It's why code review finds bugs the original author didn't. Separating the roles builds that dynamic into the process.

We don't apply this pattern mechanically to every task. A simple bug fix doesn't need an orchestrator. A single-endpoint change doesn't need parallel builders. The pattern earns its overhead when the work is complex enough that a single agent would hit its limits.

When a feature touches the database, the API, and the frontend — or requires integration with a third-party system alongside product logic — the orchestrator decomposes this into separate workstreams that can be specified and verified independently.

Refactoring a data model, migrating an API version, updating an authentication system — these are changes that affect many files and many layers. Parallel subagents, each responsible for a specific domain, move through these changes faster and with less risk of cross-contamination.

Any work where correctness is non-negotiable gets a critic configured with explicit acceptance criteria, not just a general quality review. In regulated industries — healthcare, fintech, anything with real compliance exposure — the critic's checklist includes the specific requirements that can't be compromised.

A Representative Example: Building a Care Coordination Feature

The following is a composite based on the kinds of engagements we run for 

 The specifics are illustrative, but the pattern and the catches are real.

The scenario: a care coordination platform managing care plans for patients across multiple providers. The feature: a notification system that alerts care team members when a patient's status changes, routed correctly based on the member's role, their relationship to the patient, and the type of status change.

This is a non-trivial feature. It has a rules engine (who gets notified, under what conditions), a delivery layer (push, SMS, in-app), a preference system (notification settings per user), a data model change (new notification event types with audit logging), and it touches HIPAA-covered data throughout.

Here's how the orchestrator decomposed it:

Define the event schema for status change notifications and the audit log format

Build the rules engine mapping patient status changes to notification recipients

Build the delivery service (separate subagent per channel: push, SMS, in-app)

Wire the components together and validate the end-to-end flow

Tasks 1 through 4 were assigned to parallel builder subagents. Each had a scoped context: the event schema agent only saw the existing data model and the requirements doc. The rules engine agent saw the event schema output and the business logic spec. The builder agents each saw only their channel's integration requirements.

The critic for this engagement was configured with specific criteria: every function touching PHI must pass through the existing access control middleware, no patient identifiers in log output, notification content must not include clinical notes.

That last requirement caught something on the first pass. The rules engine subagent had included a brief status summary in the notification payload -  the kind of detail a developer building quickly might add as helpful context, without recognizing it as protected health information in transit. The critic flagged it. The builder revised the payload to pass only identifiers and a status code, with readable content fetched client-side after authentication.

That's a meaningful catch. It's the kind of thing that slips through code review when reviewers are moving fast and aren't specialized in HIPAA requirements. Having an agent whose entire job is to evaluate output against a defined compliance checklist — before anything leaves the development workflow — is a different category of protection.

What This Means for How We Staff Engagements

The three-agent model changes what senior engineers spend their time on, not whether you need them.

Engineers at Foxbox are not writing boilerplate. They're not typing out CRUD endpoints or writing the same authentication middleware they've written a dozen times. What they're doing is the work the model can't replace: designing the orchestrator's decomposition, defining the critic's acceptance criteria, reviewing synthesized output before it merges, and making the architectural decisions that determine whether a system holds up at scale.

This is what senior engineering capacity should be used for. The judgment work. The design work. The work that requires understanding the whole system, not just the current task.

What AI has changed is the ratio. More of a senior engineer's day is judgment. Less of it is execution.

The model has real costs. Each agent invocation adds latency and token usage. Orchestrating five parallel subagents costs more than running one. For small tasks, the overhead isn't worth it.

Poorly specified acceptance criteria undermine the critic. A vague mandate ("make sure this is good") won't catch the specific things you care about. The value of the critic is proportional to the precision of its instructions.

And the model requires engineers who understand it. Setting up effective orchestration, writing good specifications, configuring critics with meaningful criteria — these are skills. The tools are increasingly accessible. The judgment required to use them well is not.

The three-agent model is: an orchestrator that plans and coordinates, specialized builder subagents that execute in parallel isolated contexts, and a critic that evaluates output before it's accepted.

It's a pattern grounded in software engineering fundamentals and multi-agent systems research, now formalized by the major AI labs and increasingly standard in how serious engineering teams work with these tools.

We use it because it produces better software faster than any single-agent approach at the scale we operate. In the domains our clients work in — healthcare, fintech, enterprise software — speed without correctness isn't a benefit. It's debt accumulated quickly.

The Three-Agent Model: How we Standardize and Accelerate Agentic Engineering

Over the past year, I’ve had lots of conversations with CTOs and product leaders about AI readiness. Almost every organization can point to something. A pilot. A chatbot. An internal coding assistant. A proof of concept built on top of a foundation model.


AI readiness often asks how we can use AI. Whether we have access to models. Whether we’re able to conduct experiments, learn new tools and integrate some type of AI functionality into an existing product experience.
AI-native readiness asks something more fundamental. Are our systems, teams, data models, and governance frameworks designed for continuous model-driven evolution? That distinction matters.


Bolting AI onto an existing product can generate short-term gains. But if your architecture tightly couples logic to a single provider, if your data pipelines are not instrumented for feedback, if your evaluation loops are informal, and if security controls were designed for deterministic systems, you will plateau quickly.


AI-native organizations design for adaptation. They assume models will improve monthly. They assume workflows will shift. They assume evaluation and governance are ongoing responsibilities, not launch-phase checklists.
As technical leaders, this is not about chasing hype. It is about making durable architectural and organizational decisions. AI-native readiness is a better measure because it reflects whether your organization can compound value from AI over time, not just demonstrate it once.
We break it down across six dimensions:

Most AI strategies stop at pilots and features. Learn how AI-native readiness helps CTOs design systems, teams, and architecture built for continuous AI-driven evolution.

AI Native Readiness: Beyond Bolted-on AI Solutions

AI-native organizations design for adaptation.

 They assume models will improve monthly. They assume workflows will shift. They assume evaluation and governance are ongoing responsibilities, not launch-phase checklists.
As technical leaders, this is not about chasing hype. It is about making durable architectural and organizational decisions. AI-native readiness is a better measure because it reflects whether your organization can compound value from AI over time, not just demonstrate it once.
We break it down across six dimensions:

1. Product Architecture Designed for Model Evolution

AI-native products assume models will change. Models improve. APIs evolve. Pricing shifts. Providers consolidate. Your architecture should reflect that reality. AI-native systems abstract model layers. They separate orchestration from inference. They support component swapping. They log structured interactions for retraining and evaluation.
In practice, this means:

Clear boundaries between business logic and model interfaces

Observability at the prompt, response, and decision layer

Versioned prompt and evaluation frameworks

The ability to test multiple models against the same workflow

If replacing a model requires refactoring large portions of your system, you are not operating in an AI-native posture.


2. Data as a Strategic Asset, Not an Afterthought

AI-native readiness goes beyond asking whether data exists. It asks whether data is usable, governed, and connected to feedback loops.
You need:


Structured and unstructured data pipelines

Mechanisms to capture user interactions for improvement

Instrumentation that ties model outputs to business outcomes

For example, in a clinical triage workflow, it is not enough to generate recommendations. You must track recommendation accuracy, escalation rates, and user overrides. That feedback must flow back into evaluation. AI-native organizations treat data flows as part of the product surface, not background infrastructure.

AI-native readiness also changes how software is built. Engineering teams should be using AI in development, testing, and QA in a systematic way.
This includes:


AI-assisted code generation and refactoring.

Automated test generation and regression validation

LLM-on-LLM review for critical logic paths

Prompt libraries treated like versioned source code

This is not simply about velocity. It is about building teams that assume intelligent tooling is embedded in the workflow. If your engineering process looks unchanged from several years ago, you are likely under-leveraging what AI-native development enables.


4. Operational Maturity and Continuous Evaluation

AI systems are probabilistic. They drift. Edge cases accumulate. Performance changes over time. AI-native readiness requires operational discipline.
You need:
Formal evaluation frameworks

Benchmark datasets aligned to real use cases

Continuous monitoring of latency, cost, and output quality

In practice, this can mean shadow testing new models before promotion to production. It can mean regression suites that evaluate AI outputs against curated scenarios. It can mean cost observability tied directly to specific workflows. Without this layer, pilots rarely mature into reliable systems.


5. Organizational Capability and Cross-Functional Alignment

AI-native readiness is not just technical.
Product, engineering, data, compliance, and security must align on:
Acceptable risk thresholds

You need leaders comfortable designing probabilistic systems. You need engineers who understand distributed architectures and model behavior. You need product teams who can define success metrics for non-deterministic outputs.
Without cross-functional clarity, AI remains isolated.

6. Data Security and Model Governance by Design

AI introduces new attack surfaces.
Prompt injection. Data leakage. Model inversion. Third-party API risk.
AI-native readiness requires:
Strict data segmentation

Vendor risk assessments specific to AI providers

Encryption and access controls at every inference boundary

In regulated industries, this is foundational. If you cannot clearly trace how data moves through your AI workflows, you should not scale them.
Security cannot be retrofitted onto AI systems built without governance in mind.


AI-native readiness is not about having the latest tools. 

It is about whether your organization is positioned to continuously build, evaluate, and evolve intelligent systems.

As CTOs, we are responsible for durable systems. Systems that adapt without becoming fragile. Systems that compound value rather than degrade.
AI-native readiness is a structural lens. It forces us to examine architecture, data, workflows, operations, organizational alignment, and security as an integrated system.


That is a more meaningful measure than whether we have launched an AI feature. And in the current environment, it is the measure that will separate experimentation from sustained advantage.


Take the self assessment

Learn how to use Claude Code to automate end-to-end test creation. Build, review, and run automated tests with Claude Code, reducing setup time, enforcing project rules, and improving test coverage with human oversight.

Automated QA Testing in Claude Code | Foxbox Digital

AI coding assistants are no longer just for brainstorming or writing boilerplate code. They’re becoming powerful tools in test automation. In this piece we’ll walk through a real QA workflow using Claude to speed up repetitive test creation, highlight the human/AI handoff points, and share lessons learned along the way.

After an initial healthy skepticism, I’m now a firm believer that AI tools can dramatically improve QA workflows. Automated testing powered by tools like Claude can reduce hours of manual setup into minutes. But it’s not “end-to-end automation.” Human expertise is still critical in order to:

Architect processes before they can be automated.

Identify the right balance between reusability and project-specific rules.

Review and refine AI-generated code for quality, security, and maintainability.

Using tools like Claude Code, the QA engineer’s role shifts from operating a repetitive system to designing a new one entirely. The payoff is dramatic — a process that once took 1–2 hours manually can now be completed in 20–30 minutes with the right approach.

Let’s walk through one test case for a telehealth application with an AI symptom checker. To confirm patients are able to get the answers they need, we need to confirm they can upload a photo during an appointment set up flow.

Manually, the process would usually look something like:

Validate the app doesn’t crash and the upload succeeds.

Running this flow manually is repetitive and time-consuming.

Instead of hand-coding every selector and step, Claude can act as your automation assistant at every step of the way. Not too early in the process, not tacked on to the end. The craft is now finding balance. Here’s how I approached it

Using a recorder tool, the QA clicks through the scenario. Each action (tap, input, upload) is captured into a Java file.

Comments mark each step (e.g., // Login, // Select symptom, // Upload photo). Claude uses these annotations to understand context.

Naming conventions (PascalCase, file structures).

When to reuse existing methods (e.g., login flows).

Claude ingests the recordings + rules, and generates test files:

Page object files (for selectors and UI elements).

The engineer reviews the generated files, validates selectors, and fixes any missteps. Common pitfalls:

Misclassified selectors (e.g., ID vs XPath).

Fragile selectors that change with app updates. Security risks (Claude sometimes proposes hard-coded values).

With experience, teams learn to anticipate these errors and adjust rules accordingly.

On this particular project, this process has reduced setup time by ~70%, improved reusability, and has helped us deliver a better product in less time. Still, success requires careful upfront architecture and thorough human oversight, since selectors often trip up the AI and rules must be rebuilt for each new project. It moves at a speed no human could, but also makes mistakes that humans never would.

Understanding how to navigate both realities is what will redefine the “Quality” in QA.

: Begin with simple, repetitive flows like logins, file uploads, or form submissions—these are perfect for automation because they’re high-frequency and low-risk. Starting here helps you validate the process, refine your rules, and build confidence in Claude before tackling more complex scenarios.

Your CLAUDE.md (or equivalent rules file) acts as the brain of the system, guiding Claude on naming conventions, file structures, and method reuse. The clearer and more prescriptive this file is, the less cleanup you’ll face later, making it one of the highest-leverage investments in your QA automation setup.

Claude moves fast, but it’s not flawless—misclassified selectors, fragile locators, or incorrect assumptions will creep in. By budgeting time for debugging, you turn these missteps into learning moments that strengthen both your test suite and your rules over time.

Never merge AI-generated code blindly; human validation is critical. Security is a particular concern—watch closely for leaked credentials, hard-coded values, or unsafe shortcuts that might slip through without careful review.

As you run more scenarios, your rule set will grow into a library of reusable steps, making Claude more accurate and reducing duplication. Over time, this library becomes a strategic asset—accelerating test creation and improving consistency across your QA practice.

Automated QA Testing in Claude Code

Discover how Foxbox Digital unpacks the MIT AI Study and the viral “95% of AI Pilots fail” claim. Learn why the stat is misleading, what AI pilots really deliver, and our perspective on building lasting value with AI.

Rethinking AI Pilot Success | Foxbox Digital 

The problem with that viral "95% failure" stat   

 by claiming that 95% of enterprise AI pilot projects fail to deliver any measurable ROI.The headline quickly went viral and fueled talk of an AI hype bubble and a parade of LinkedIn posts about how to “ensure your pilot is part of the 5% success stories!”

But a closer look reveals that 95% stat isn’t telling the full story. The study defined “success” in extremely narrow terms: only pilots that moved into full deployment and showed a profit-and-loss impact within six months counted as successes. 

By that definition, countless AI initiatives that improved efficiency, cut costs, or yielded valuable learnings were labeled failures. In fact, the MIT report’s authors offered almost no data to back the 95% figure, basing it on a small set of interviews and vague criteria.


Even if the headline was accurate, are we missing the bigger picture about where AI is, where it should fit, and what smart investments look like when nobody knows exactly what to expect next?

Even if many AI pilots aren’t immediately profitable, that doesn’t mean those projects are without tremendous value. On the contrary, constant experimentation is both productive and necessary when working with emerging technologies. 

Experimentation is always about learning and generating new insights, not a binary success/fail paradigm.
A stalled pilot can reveal integration challenges, data quality issues, or user adoption hurdles – knowledge that is invaluable as leaders continue iterating toward viable solutions.

In other words, the technology often “works”, it’s the organizational readiness and implementation strategy that lag behind. Innovation requires iteration, and a high early failure rate is normal for any nascent tech. Many executives are learning that “failure” in AI pilots often just means learning what doesn’t work so you can discover what does.

The problem, then, isn’t that pilots sometimes fail – it’s when organizations fail to learn and adapt. Some companies fall into the “experimentation trap,” endlessly running pilots that never connect to real business value or scale beyond the lab.
The recent MIT report itself concluded that the core issue impeding AI ROI is not model quality at all, but a “learning gap” for both the tools and the organizations using them. 

In practice, this means that companies aren’t effectively absorbing the lessons of their AI experiments. Rather than viewing a pilot that didn’t hit a short-term KPI as a waste, leading organizations ask: What did we 

learn about our data, processes, and people

? How can we apply that knowledge to the next AI project?
The real metric of success is how quickly a team can turn today’s pilot missteps into tomorrow’s strategy. 

When failures prompt tweaks and improvements, they cease to be failures at all – they become stepping stones to innovation.

It’s important to remember that many AI success stories emerged only after some trial and error. Far from 95% of companies getting “zero return,” there are numerous examples of AI pilots that paid off after iterative improvement, case studies via 

JPMorgan Chase – Applied AI in risk and operations and reduced financial losses while improving operational efficiency and customer satisfaction. Early prototypes were refined to achieve these gains.

Nestlé – Developed an internal generative AI assistant (“NesGPT”) that employees across departments use daily. The company says this experiment accelerated idea generation from six months to six weeks by empowering teams to iterate quickly with AI.

Novo Nordisk – Created an AI-powered documentation tool (NovoScribe) after piloting the concept. It now automates report writing, cutting resource needs by 80% and shrinking documentation time from weeks to minutes.

These cases show that real ROI is being achieved through AI, often after multiple tweaks and pilot rounds. In fact, analysts have documented 130+ enterprise AI case studies with tangible benefits and direct ROI - evidence that a blanket “failure” label is pessimistic and short-sighted.

Inside the Box

Articles, updates, and ideas you can put to use.