Did AI Actually Improve Anything? How to Measure It
In my previous post I wrote that the market has started asking AI for proof of effect. The obvious follow-up question is: how do you actually produce that proof inside a company? Not for investors, but for yourself, your management team and the people whose work changed.
I don’t think there is a perfect answer. But there is a practical one, and most of it is old-fashioned measurement discipline applied to a new kind of tool.
Faster is not the same as better
Deloitte’s State of AI in the Enterprise, published 29 October 2026 and based on 3,500 leaders, puts the problem neatly. 81% say AI has made decision-making and execution faster. Only 25% say they use it to deeply change the business. And while 71% track their total AI inference and API spend, only 44% track the cost per unit of inference.
In other words: we can feel the speed and we can see the bill. What sits between them, the cost and value of a single piece of work, is mostly invisible.
G2’s 2026 AI Agents Report (14 October) adds the other half. 31% of buyers say employees spend six or more hours a week overseeing agent work, and 45% say that oversight and rework eat more than a quarter of the gains they expected. If you only measure the agent’s output and not the human time around it, you will overstate the effect.
Step 1: Record a baseline before you roll anything out
This is the step that gets skipped most often, and it cannot be done afterwards. Before AI touches a process, measure for at least a few weeks:
- Volume: how many cases, tickets, invoices, pull requests or orders.
- Cycle time: from start to done, not just the part AI will touch.
- Quality: error rate, rework rate, reopened cases, complaints.
- Cost per unit: people hours, licences and systems divided by volume.
Measure it the same way you plan to measure it later. A baseline taken from a different system or with a different definition is just a number.
Step 2: Compare against something
Before-and-after alone is weak. Seasons change, teams change, a price increase lands in the same month. Where you can, use a comparison:
- A/B or staged rollout: give the tool to one team, region or queue first and keep a similar one as control for a period.
- Before-and-after with a comparison group: if everyone gets the tool, compare the change against a process or unit that was not affected.
- Holdout on a sample: let a small share of cases go the old way, so you keep a living reference.
None of this needs a statistics department. It needs someone to decide, before launch, what will count as a result.
Step 3: Separate leading from lagging indicators
Lagging indicators are the ones management cares about: cost, revenue, churn, margin. They move slowly and are influenced by many things. Leading indicators move within days: acceptance rate of AI output, time to first response, share of cases handled without escalation.
You need both. Leading indicators tell you early whether you are on the right track. Lagging indicators tell you whether it mattered.
Step 4: Adoption is not effect
“80% of employees use the assistant weekly” is an adoption number. It says the tool is used. It does not say anything improved. I have seen high adoption with no change in cycle time, and low adoption with a large effect because the right ten people used it on the right process.
Report adoption, but never as the proof.
Step 5: Count cost per accepted outcome
This is the single metric I would add if I could only add one:
Cost per accepted outcome = (inference + licences + platform + human review and rework time) ÷ outputs accepted without correction
It captures what the G2 numbers warn about. An agent that is cheap per call but needs heavy correction can be more expensive than the manual process. And it forces you to define what “accepted” means, which is often the most useful conversation of the whole project.
Step 6: Count what you removed
Real effect usually shows up as something that is gone, not something that is added:
- licences you cancelled,
- legacy systems or integrations you shut down,
- hours that are no longer spent, and what they are spent on now,
- reports nobody needs any more.
If the AI layer is added on top of everything that already existed, you have a double cost, not a saving.
A simple measurement framework
| Function | Leading indicator | Lagging indicator | Quality / risk | Removed cost to look for |
|---|---|---|---|---|
| Customer service | First response time, share resolved without escalation | Cost per resolved case, CSAT | Reopen rate, wrong answers | Seats in old ticket tools, overtime |
| Sales | Time from lead to first contact, prep time per meeting | Win rate, sales cycle length | Incorrect quotes, CRM data quality | Manual list building, external data services |
| Finance | Invoices auto-matched, days to close | Cost per invoice, DSO | Posting errors, audit remarks | Manual reconciliation hours |
| IT / development | Lead time for changes, PR review time | Change failure rate, incidents | Defects escaping to production, security findings | Unused tools, legacy jobs |
| Operations / supply chain | Forecast error, time to detect deviations | Stock-outs, inventory cost, OTIF | False alarms, missed alarms | Manual status reports, spreadsheets |
| HR | Time to answer policy questions, time to hire | Early attrition, cost per hire | Bias checks, GDPR incidents | Admin hours, overlapping systems |
| Data / analytics | Time from question to answer, alerts acted on | Decisions changed by data, cost per insight | Data quality incidents, wrong numbers in use | Dashboards nobody opens, manual pipelines |
Use it as a starting point, not a template. For each process, pick one or two from each column and write down the baseline.
Step 7: Alert thresholds, not more dashboards
This is where my own experience comes in. At a global manufacturer, we had plenty of dashboards. The problem was not the lack of data, it was that people had to remember to look, and then interpret what they saw. The shift that made a difference was moving from dashboards to alerts: define the threshold that matters, notify the person who can act, and give them a drill-down to the cause.
The same logic applies to measuring AI. Instead of an “AI value dashboard”, define thresholds:
- acceptance rate drops below X% for two days,
- cost per accepted outcome rises above the manual baseline,
- reopen or error rate exceeds the pre-AI level,
- human review time per case grows week over week.
When a threshold is crossed, someone owns the next step. That turns measurement from reporting into management.
In my own projects I now run a data platform where agents do much of the maintenance: pipelines, checks and documentation. What keeps that honest is not a status page, it is a small set of checks that alert me when quality or cost moves the wrong way. Most days nothing happens, which is the point.
Three things to do on Monday
- Pick one AI initiative and write down its baseline, even if it’s late. A partial baseline beats none.
- Define “accepted outcome” for that process and start counting cost per accepted outcome, including review time.
- Replace one dashboard with one alert. Choose the metric that would make you act, set the threshold and name the owner.
I don’t think anyone has fully solved how to measure AI. But the companies that can show what got better, and what they stopped paying for, will be the ones still funding AI next year.