The Metric That Creates False Confidence The Metric That Creates False Confidence When we think of model performance, the first thing that usually comes to mind is accuracy. It is easy to compute, sounds very precise, and even small improvements look like a lot (e.g., increasing the accuracy from 94% to 97% looks like a lot of improvement). It is something that can go on a slide, that can be updated on a dashboard, and communicated to stakeholders as actual progress. So yes, it is tempting. In the lab, everything is fine. We know the distribution of the data, we know what the differences between models are, and we know that an improvement in accuracy will yield a better model. Production systems operate differently. The majority of systems will function as intended for quite a long time. The minor effects of design decisions will only become apparent later on. Not in the form of spectacular explosions or power failures. Small movements in a new direction that end up influencing people's behavior. For example, a few more false alarms at night than during the day. Longer queues during the morning than during the evening for samples that have to be reviewed manually. And sometimes, borderline cases arise where the system's decision is in doubt, even though the accuracy indicator shows 99% and suggests that the system is functioning correctly. False confidence begins here. Accuracy measures the performance of a model on a set of carefully selected and validated examples used for evaluation. Production measures how the model performs in the real world, where many things can go wrong. The Difference Between Model Performance and System Behavior The Difference Between Model Performance and System Behavior Most development workflows are built on the assumption of a static world. We have a feature pipeline, a static test dataset, and our models improve within a clearly defined space of possibilities. And in that space, improvements really are improvements. Once deployed, those boundaries dissolve. As the model is deployed and used, schema changes propagate upstream, logging frameworks get upgraded, traffic patterns shift after release, and hardware degradation that was not noticeable during local testing becomes visible under heavy workloads. A workflow originally designed for a simple inference task can end up affecting other parts of the system as well. Accuracy is a measure of how a model performs on the data it was designed to analyze. It does not take into account the real-world noise and variability that the model will be exposed to. So, when a new field was added to the request payload, things changed. Although we aren’t actually using this new field in the model itself, or at least not in the calculations, it ended up changing our processing code. When the results were examined, the overall accuracy appeared unchanged, but a small number of samples fell on the wrong side of the decision boundary. This was not reflected in a significant change to the accuracy metric, yet the system’s behavior felt slightly different. The model was functioning. The system had become sensitive. Those are not equivalent conditions. How Small Shifts Amplify in Production How Small Shifts Amplify in Production In high-throughput environments, small statistical effects tend not to remain small. A few percent translates into thousands of events per day. And when those events start to appear in workflows or downstream automation, their impact grows. There is a model designed to auto-approve certain types of transactions. It looks good initially after deployment and does not trigger any escalations within the defined alert thresholds. But then, a few days later, a small pattern of what we'd call “marginal” approvals shows up in the data. Although the overall service metrics remain strong, the increasing number of manual reviews required to handle these transactions, combined with heavier queue workloads over time, results in backlogs and longer batch processing times during peak traffic. It may take some time before the KPIs are updated to reflect the problem, even as additional rules are added, and internal tolerances are adjusted to correct it. None of these items is captured in the KPIs. From a distance, everything appears stable. Upon closer inspection, it becomes apparent that the stability has to be maintained by manually controlling each of the variables. Large production systems are designed for small changes. Average accuracy reflects the distribution of errors that will be experienced during the operational life of a loaded system. Bill Parcells once said a team is “what its record says it is.” Being right 95 percent of the time is one thing. Being wrong five times in a row is quite another. You might not be able to discern five scattered mistakes. Five in a row are impossible to miss. Drift as an Ongoing Operational Condition Drift as an Ongoing Operational Condition Data drift is rarely dramatic or visible. Models do not fail abruptly; instead, it unfolds gradually over time. New user behaviors emerge, seasonal trends return, and environmental conditions shift. And before you know it—small changes add up. Tracking aggregate accuracy is useful, but the real signal of distress often comes from elsewhere. It takes more adjustments to keep everything within expected ranges. The effort required to keep outputs within bounds becomes noticeably larger. And while the number of anomaly alerts is still small, they’re becoming more commonplace and less shocking. By the time accuracy visibly declines, the time, money, and effort required to recalibrate the system can outweigh the value it provides, effectively forcing a decision about whether it should continue operating. Drift is always occurring; the real questions are how quickly deviations from normal can be detected, and how long the system can operate before required maintenance becomes safety or business critical. It’s not all about accuracy. Timing and response discipline matter just as much. Stability, Recoverability, and Auditability Stability, Recoverability, and Auditability Most definitions of production performance focus on forecast accuracy. But there’s more to it. One important dimension is stability. By stability, we mean that a production system can deal with changing conditions without falling apart. This includes handling varying workloads from increased user activity, as well as unusual events such as power failures or unpredictable user behavior. It should also cope with minor hardware failures without having a major impact on overall functionality. A stable system can handle high loads without losing track of important logs and messages. When a failure occurs, the question becomes one of recoverability. This concerns whether a problematic change can be easily rolled back. It also concerns the versioning of artifacts across services and systems. Is this fully automated or a set of manual steps that need to be performed during stressful events, where time is of the essence? Incidents are the best way to answer these types of questions. Auditability increases the time horizon of the consequences of our current decisions. Even if a model is generally correct, users may only discover months later that certain model-driven decisions require explanation. It is not sufficient for the model to be generally correct. Rather, we need to ensure that we can trace every individual decision that was driven by the model. We need to know which model version was used, which decision was taken during the data-preprocessing pipeline, which feature engineering was selected, and which state the system was in when a prediction was made. These dimensions rarely affect benchmark results and are seldom measured during performance reviews. They are very important when determining if a system is ready for production as a highly performing, highly available, productive production system. The absence of visible problems does not necessarily imply resilience. Rethinking Production Readiness Rethinking Production Readiness Even if accuracy is a goal, a poorly trained model remains poorly trained, regardless of how reliable the underlying platform may be. Reducing reliability to model accuracy alone turns the discussion into a surface-level one. From a production readiness perspective, the classification score or the MAE/RMSE in regression tasks is not the only relevant factor. Other important factors include detection latency, rollback behavior, anomaly visibility, and trace completeness. In production, these factors can negate gains that were observed on the evaluation set. As a system grows and matures, the questions that you are asking start to change. Rather than asking “how can this metric be improved?”, the question becomes “what happens when small changes are introduced across different inputs and conditions?” What was once a question of “will it give the right answer for this input?” becomes “how will it behave when those inputs contain noise?” What began as an edge case, a quick hack, or a minor adjustment eventually becomes part of a collection of small changes that together alter overall behavior. These are difficult questions to answer, as the effort required to shift a metric by a small amount can vary significantly. Accuracy can initiate confidence. Durability sustains it. The most reliable production AI systems are not those that achieve the highest benchmark score at a single point in time. They are the systems that continue to function as intended under changing circumstances and over long periods of time. In the long run, reliability matters far more than achieving a high score. The views expressed in this article are solely those of the author and do not represent those of any employer or affiliated organization. The views expressed in this article are solely those of the author and do not represent those of any employer or affiliated organization.

Accuracy Is Not Enough: Rethinking KPIs for Production AI

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

AI Models Keep Breaking in Production; Strong Documentation Can Fix It

The Future of Chip Making: Using AI to Minimize Testing and Maximize Throughput

What Six Months of Real Users Taught Me About Running AI Systems in Production

Metrics: Swallow the Red Pill

AI Metrics Made Simple: Precision, Recall, F-Score, and ROC-AUC

As Coding Agents Go Mainstream, Teams Want Proof—and Metrics

AI Models Keep Breaking in Production; Strong Documentation Can Fix It

The Future of Chip Making: Using AI to Minimize Testing and Maximize Throughput

What Six Months of Real Users Taught Me About Running AI Systems in Production

Metrics: Swallow the Red Pill

AI Metrics Made Simple: Precision, Recall, F-Score, and ROC-AUC

As Coding Agents Go Mainstream, Teams Want Proof—and Metrics

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps