Your AI model has 94% accuracy. Your board does not care.
That is not because the board is unsophisticated. It is because accuracy answers a technical question — how often does the model get the right answer? — while the board is asking business questions. Did we process more claims this quarter? Did cost per transaction fall? Did error rates improve? Did we free capacity for higher-value work? Those are different questions, and a confusion matrix does not answer any of them.
This gap is not a presentation problem. It is where the value actually leaks. McKinsey's State of AI survey makes the point with uncomfortable clarity: adoption is now near-universal — roughly four in five organisations report using generative AI in at least one function — yet only about 39% report any EBIT impact at the enterprise level, and only around 6% qualify as "AI high performers" who can attribute 5% or more of EBIT to AI and point to significant value. The decisive difference McKinsey identifies is not better models. It is whether the organisation has rewired how it works around AI — and, in practice, whether it can measure what the system is actually doing. Where leaders track outcomes against a baseline, the case for expansion writes itself. Where they cannot, programmes stall in a fog of impressive demos and unprovable benefit, and the budget conversation quietly dies.
Closing the gap requires a measurement framework that starts from business outcomes and works backwards to the technical metrics that support them — not the other way around.
The measurement hierarchy
Think of AI measurement as three levels. Each serves a different audience and answers a different question, and most failed programmes try to make one set of numbers do all three jobs.
Level one — business outcomes, for the board and executive team. These are the metrics that justify the investment, and they should be expressible in currency, time, or units that any leader can interpret without a data scientist in the room. Throughput: how many claims, invoices, tickets, or orders move through per week, and has that risen since deployment? Cost per unit: what does it cost to process one case end-to-end, and has it fallen? Cycle time: how long from input to output, and has it shortened? Error rate: what share of outputs require correction or rework, and has it improved? Capacity redeployment: how many hours or FTEs have been freed from repetitive work and redirected to something more valuable? These five cover the business case for the large majority of operational AI workflows. Show improvement on two or more and the investment is justified. Show improvement on none and the workflow is not delivering value — however good the model looks on a test set.
Level two — operational metrics, for the operations team. These tell the team whether the workflow is functioning day-to-day, and they are leading indicators: they move before the quarterly outcome numbers do. Automation rate — the share of cases handled end-to-end without human intervention — tells you how much load the system is actually carrying. Fallback rate — cases routed to a human because model confidence sits below threshold — is an early warning; a creeping rise often signals drift before accuracy visibly degrades. Queue depth and latency expose capacity bottlenecks. Edge-case volume and type reveal whether the live input population is wandering away from what the system was built for. And reviewer agreement rate — how often humans concur with the model when they check it — is one of the cleanest early signals of degradation you will get. The Workflow Owner should be watching these on a dashboard weekly. None of them belong in a board pack.
Level three — technical metrics, for the engineering team. Accuracy, precision, recall, confidence distribution, inference latency, and input-distribution drift matter enormously for maintaining and improving the model — but they are diagnostic tools, not business value. Their job is to explain why a level-two metric moved. A shift in the confidence distribution or a detectable drift in input data is usually the technical story behind a rising fallback rate. Report these in engineering reviews. Keep them out of the board deck, where they generate either false comfort or unnecessary alarm.
Build the baseline before you deploy
You cannot measure improvement without a baseline, which is obvious and yet routinely skipped. Companies deploy a workflow, then discover three months later that they cannot quantify the impact because there is nothing to compare against. The model works; the business case is unprovable; the budget conversation goes nowhere.
Establish the baseline two to four weeks before deployment, measuring exactly the five outcomes you intend to track afterwards: current throughput in units per week; fully loaded cost per unit, including labour, systems, and the cost of correcting errors; cycle time from input to output including wait time; current error rate; and current capacity allocation — how many FTEs work the process and what share of their time it consumes. Document these numbers and protect them. They are the foundation of every ROI calculation for the life of the workflow.
If you are still evaluating which workflows to automate, our AI Operating Diagnostic includes a baseline measurement framework that structures this collection from the start.
The ROI calculation that actually works
AI ROI calculations tend to fail in one of two directions. The oversimplified version — "we saved three FTEs, so ROI is three salaries minus implementation cost" — does not survive a sceptical CFO. The overcomplicated version — a twenty-variable model with Monte Carlo sensitivity analysis — does not survive a board meeting. The version that works for a Mittelstand board has four components.
Direct cost savings: the reduction in labour cost for the automated portion of the workflow, calculated as hours saved per week multiplied by fully loaded hourly cost multiplied by working weeks. Be conservative; use observed hours saved, not theoretical maximum. Throughput value: where higher throughput generates revenue or prevents its loss, quantify it — an insurer that clears claims faster retains more policyholders; a manufacturer that inspects more thoroughly ships fewer defects. Where the throughput gain has no revenue line, leave it qualitative rather than inventing one. Error-cost avoidance: every error carries a cost in rework, customer goodwill, and regulatory exposure; if AI cuts the error rate, the avoided cost is often the single most persuasive figure for a risk-conscious board. And capacity-redeployment value: count freed capacity only when it is genuinely redirected to measurable output — new acquisition, complex-case handling, process improvement. Capacity that is simply absorbed without measurable result is a management problem, not an AI benefit, and counting it will eventually be found out.
Sum the four, subtract the total cost of the workflow — implementation, operations, licensing — and you have a number a board can evaluate and an auditor can follow.
When to measure, and when to report
Cadence matters as much as the metrics. Weekly, the Workflow Owner reviews the level-two operational dashboard and acts only when something sits outside its expected range — no report, just attention. Monthly, for the first six months after deployment, compile the level-one business outcomes against baseline; this tighter loop catches problems during the period when the system is least stable. Quarterly, report business outcomes to the AI Sponsor and executive team with baseline comparison, trend, and any action taken or needed. Annually, calculate full-year ROI against the business case that justified the spend, and use it to decide whether to expand, modify, or retire the workflow — and to build the case for the next one.
There is a compliance dividend hiding in this discipline. Under Article 12 of the EU AI Act, high-risk AI systems must technically allow for the automatic recording of events over their lifetime — automatically, not by hand — to support risk identification, post-market monitoring, and ongoing operational oversight. Deployers must keep those logs for a period appropriate to the system's use and, under Article 26, for at least six months. If your operational metrics already flow from system-generated logs rather than spreadsheets assembled after the fact, the measurement framework that satisfies your board is most of the way to satisfying your obligations as a deployer. Build the logging once; serve both masters.
Metrics that mislead
Some numbers sound useful and are actively misleading in an operational AI context, and a thought-leading operator names them out loud.
Accuracy in isolation is the worst offender. A model at 95% accuracy sounds reassuring until you find the 5% of errors clustered in your highest-value cases — the ones that decide a customer relationship or a regulatory exposure. Always pair accuracy with an analysis of where the errors fall. Time saved without redeployment is a close second: "AI saves the team twenty hours a week" means nothing if those twenty hours dissolve into the day rather than converting into measurable output elsewhere. Percentage automated without a quality check is a vanity metric — "80% of cases fully automated" is impressive only if the automated cases are actually right, so automation rate is meaningless reported apart from error rate. And comparison to a theoretical maximum — "we have captured 60% of the theoretical throughput ceiling" — tells the board nothing about whether the investment paid. Compare to the baseline you measured, never to an ideal nobody can bank.
Measurement is part of the method, not an afterthought
In the AI Operating System methodology, measurement is built into every phase rather than bolted on at reporting time. Discovery establishes the baseline. The Accelerator deploys the workflow and starts measuring against it. OS Build refines the framework as the system matures and the input population reveals itself. Managed Operations keeps measurement running as part of the standing operating rhythm. That continuity is what stops measurement from decaying into a one-off slide that nobody trusts by the second quarter.
The principle underneath all of it is simple. The board does not care that your model is accurate. It cares that the business is measurably better off, and that you can prove it the same way every quarter. Build the measurement before you build the case, and the case builds itself.
A Fit Call defines the two or three business metrics your board will actually accept for a target workflow — before you deploy and discover you can no longer prove the impact.
References: McKinsey & Company, "The state of AI: How organizations are rewiring to capture value," 2025 (https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai); EU Artificial Intelligence Act, Article 12 — Record-keeping (https://artificialintelligenceact.eu/article/12/).