The most common evaluation method for enterprise AI is still "the demo looked good." This is roughly equivalent to choosing a new ERP system by watching the vendor's slide deck. The demo is a curated performance. Production is everything else, all at once, indefinitely — and the gap between the two is where most AI investment quietly dies. MIT's NANDA initiative, surveying 300 public deployments and several hundred practitioners for its 2025 GenAI Divide report, found that roughly 95 percent of enterprise generative-AI pilots delivered no measurable impact on the bottom line. The same year, S&P Global Market Intelligence reported that the share of companies abandoning the majority of their AI initiatives jumped from 17 percent to 42 percent. The model is rarely the reason. The absence of a way to prove the system works in production usually is.
Proper AI evaluation is not a one-time gate. It is an ongoing operational practice that measures whether the system delivers value in production — not whether it impressed stakeholders in a controlled demonstration. This is not merely good engineering hygiene. For systems that fall inside the EU AI Act's high-risk categories, it is becoming a legal expectation. Article 15 requires that high-risk systems achieve an appropriate level of accuracy and robustness and "perform consistently in those respects throughout their lifecycle," with the relevant accuracy metrics declared in the instructions for use. You cannot declare a metric you have never measured, and you cannot prove consistency over a lifecycle you do not monitor.
Why demo performance misleads
Demos use curated inputs. Production receives everything — malformed queries, edge cases, adversarial prompts, formats nobody anticipated, documents from a department that does things differently. A benchmark score earned under controlled conditions does not guarantee behaviour under real traffic. Stanford's HELM project made this point with unusual rigour: it evaluated dozens of models against seven metrics at once — accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency — and showed that a model can look strong on a headline accuracy number while dropping sharply once you also measure calibration or robustness to perturbed inputs. The single score hides the failure modes that matter operationally.
The second problem is metric selection. Demo evaluations typically measure one thing — "did the output look correct?" — on a handful of happy-path examples. Production systems need to measure several things at once, continuously, against traffic that shifts under them.
The six-metric evaluation framework
Task-specific accuracy. Generic accuracy is close to meaningless. A document-classification system needs precision and recall per class, not a single overall figure. A system that correctly classifies the bulk of invoices but quietly misses most credit notes can still report a flattering headline accuracy while hiding a serious business problem in the minority class. Define accuracy metrics that map to the business outcome — the cost of a false positive versus a false negative is rarely symmetric — rather than to a statistical average that flatters the headline.
Hallucination rate. As covered in our analysis of why language models invent facts and what it costs, hallucination is domain-dependent: grounded summarisation behaves very differently from open-ended legal or financial reasoning. The figure that matters is not an industry benchmark but the rate measured on your own inputs, with your own grounding documents and retrieval setup. Measure it where the system actually runs, and track it over time, because it moves.
Latency distribution. Average latency is the wrong metric — it averages away exactly the cases that hurt. Measure p50, p95, and p99. A system with a comfortable 200-millisecond average but a five-second p99 will frustrate roughly one user in a hundred, and in a customer-facing workflow handling thousands of requests a day, that tail is not a rounding error. Set latency targets by use case: a real-time assistant needs a sub-second p95, while overnight batch document processing can tolerate minutes.
Cost per task. Not cost per token — cost per completed business task. A contract review that requires several model calls, two retrieval queries, a re-ranking step, and a verification pass costs far more than the raw token count suggests. Measure the full pipeline cost, including retrieval, re-ranking, verification, and any human-review triggers. For a DACH Mittelstand operation this is the number that actually decides whether to ship: a task that costs a few cents at small volume can quietly become a meaningful monthly line item once it runs across an entire back office, and it is the metric that connects model behaviour to a budget a Geschäftsführer will sign.
Consistency. The same input should produce semantically equivalent outputs across repeated runs. High variance is not a quirk; it is a reliability defect, and it is corrosive for any process that needs an audit trail or reproducible decisions. The EU AI Act's emphasis on consistent performance over the lifecycle is, in effect, a consistency requirement written into law. Measure it with semantic-similarity scoring across repeated runs of the same inputs, and treat a widening spread as an early warning.
Drift indicators. Performance degrades as input distributions shift, business processes change, and source documents are updated underneath the system. Track your accuracy metrics on a regular cadence and compare current performance against the baseline established at deployment. Define the thresholds at which degradation requires intervention — retraining, re-grounding, or rollback — before you are debugging them live.
Building an evaluation pipeline
An evaluation pipeline is not a spreadsheet someone updates when they remember. It is an automated system that runs continuously against production traffic. The NIST AI Risk Management Framework organises this discipline under its Measure function — testing, evaluation, verification, and validation, applied not once but throughout the lifecycle. The framework is voluntary and sector-agnostic, which makes it a practical scaffold for a mid-market organisation that needs structure without a hyperscaler's headcount.
Golden test sets. Curate two to five hundred examples drawn from your actual production inputs, each with a verified correct output. This is your ground truth. Run the system against it on a fixed schedule, and treat any accuracy drop as a signal to investigate before it reaches users. The set is a living asset — every genuinely novel failure you find in production earns a place in it.
Shadow evaluation. Sample a small slice of live traffic, a few percent, and route it through both the production system and an evaluation pipeline. Compare against human judgement on a rotating roster of reviewers. This is where you catch the edge cases a static golden set never anticipated — the new document format, the unusual phrasing, the input from the one team that does things its own way.
A/B testing infrastructure. When you change a model, a prompt, or a retrieval strategy, run the new version alongside the old on split traffic and measure all six metrics on both. Promote only when the new version demonstrably wins on the metrics that matter for that specific use case. "It feels better" is not a promotion criterion; a measured improvement that does not regress latency, cost, or the minority class is.
Automated alerting. Define a threshold for each metric and let the system tell you when one is breached — accuracy below the floor you set, p95 latency past its ceiling, cost per task creeping up by a fifth. The team should learn about degradation from a monitor, not from a customer complaint or a regulator's enquiry. This continuous, systematic collection and analysis of in-service performance data is precisely what the EU AI Act's Article 72 post-market monitoring obligation expects of high-risk system providers — and the Commission's implementing template for those monitoring plans was due at the start of February 2026, which moves this from principle to paperwork.
What most enterprises get wrong
Evaluating once, deploying forever. The model that scored well in March may score considerably worse in June because the input distribution shifted, not because anyone touched the system. Evaluation is not a gate you pass through; it is a monitoring function that runs for as long as the system runs.
Measuring the model instead of the system. The model is one component. The retrieval pipeline, the prompt, the post-processing logic, the confidence thresholds, the human-in-the-loop fallback — all of these shape the output. Evaluate the full system as deployed, not the model in isolation, because that is what your users and your auditors will experience.
Optimising for the wrong metric. A compliance-review system tuned for speed at the expense of accuracy manufactures more risk than it removes. Map each metric to the business outcome it protects, weight them deliberately, and make the trade-off an explicit decision rather than an accident of whatever was easiest to measure.
No baseline. Without measuring current human performance on the same task — accuracy, throughput, and cost on a representative sample — you cannot say whether the AI system is an improvement or an expensive lateral move. Establish that comparison point before you deploy. It is also the most honest number you will ever show the board.
The organisations that get evaluation right are not the ones with the largest model budgets. They are the ones that decided, early, that "the demo looked good" is not a measurement — and built the discipline to prove value continuously, defend it to a regulator, and stay on the right side of the line that separates the 5 percent that ship from the 95 percent that stall.
A Fit Call maps your current AI measurement practices against a six-metric framework and the EU AI Act's accuracy and post-market monitoring obligations — before a drifting system becomes a discovered liability.
References: MIT NANDA, "The GenAI Divide: State of AI in Business 2025," 2025, https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/; S&P Global Market Intelligence, "2025 AI Experiences Survey," 2025; NIST, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," 2023, https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf; EU Artificial Intelligence Act, Article 15 (Accuracy, Robustness and Cybersecurity), https://artificialintelligenceact.eu/article/15/; EU Artificial Intelligence Act, Article 72 (Post-Market Monitoring), https://artificialintelligenceact.eu/article/72/; Liang et al., "Holistic Evaluation of Language Models (HELM)," Stanford CRFM, https://arxiv.org/abs/2211.09110.