A 2025 MIT report shows that, even after massive enterprise spending of billions of dollars on GenAI pilots, 95% of organizations are seeing no business returns. AI-related incidents and failures are one of the main things breaking consumer trust and hindering wider adoption of AI. Stanford Institute for Human-Centered Artificial Intelligence released the 2025 AI Index report, which shows that 233 AI-related failures were reported in 2024 alone across multiple organizations. 2025 MIT report 2025 AI Index report Imagine deploying a machine learning model that predicts equipment failures in a manufacturing company. During training and the first day of inferencing in production, the model is 95% accurate. Six months later, the accuracy degrades to 70%, and no one is aware. The model keeps making bad predictions, and the business continues trusting it, which finally results in massive failures and disruption of the business. Now, imagine the damage if the same were to happen in the healthcare industry, where actual lives are at stake. These are not hypothetical situations, but the reality of trust-breaking failures in AI. These are the exact problems AI observability can solve. What is AI Observability? AI Observability is the concept of continuously monitoring, understanding, and explaining the behavior of AI systems. It covers three key areas: traditional software observability (i.e., metrics, logs, and traces), data engineering (i.e., data lineage and quality), and finally AI-specific concerns like drift, fairness, and explainability. Think of it as monitoring, debugging, and governance of AI, all encapsulated into one. Traditional observability tools alone are simply not equipped to handle the several layers of complexity added by the AI systems. Traditional software is deterministic, i.e., the same input gives you the same output. If there is an issue, you would see exceptions, stack traces, etc. However, AI models are probabilistic in nature, i.e., the same input can give you different outputs as the data or context changes. They need a different kind of instrumentation to ensure the models don't silently degrade over time. Three ways AI models degrade over time Data Drift Data Drift Data Drift Data drift happens when the statistical distribution of input features deviates from what the model was originally trained on. A banking fraud detection model trained on pre-pandemic buying patterns might behave differently when the buying habits have permanently changed. In this case, the model hasn’t changed or isn’t broken, but it is just operating outside the scope of its training distribution. Concept Drift Concept Drift Concept Drift Concept drift is more damaging as it is subtler and more difficult to identify. The input distribution might be the same, but the relationship between the model inputs and outputs changes. A credit scoring model might lose its predictive power in scoring applicant profiles as the economic conditions change, and therefore the default risk profile changes fundamentally. Label Drift Label Drift Label Drift If the model outputs are used for deriving insights and making decisions, but at the same time are also generating data for the next round of training, you can run into a feedback loop where the model reinforces its own biases. This is very dangerous in recommendations and content moderation. As you can see, none of the above failure modes produce traditional error signals like bad infrastructure metrics or an increase in errors/exceptions, or an increase in API response times, etc. But the model is still silently wrong. Real World Impact of AI Model Degradation The risks of deploying AI without observability are not hypothetical. The Epic Sepsis Model (ESM) is one of the most documented examples of undetected clinical AI degradation. ESM is a proprietary algorithm embedded in Epic’s EHR and deployed across hundreds of hospitals in the US. ESM ESM was trained on data between 2013 and 2015 from approximately 405,000 encounters. During development and internal testing at Epic, the model reported an AUC of 0.76 - 0.83. But when independently validated at Michigan Medicine in 2021, it achieved an AUC of just 0.63. This means it missed two-thirds of sepsis patients while generating false alerts on 18% of all hospitalizations. A follow-up study in NEJM AI showed that, in certain conditions, the AUC further dropped to 0.47. ESM was trained follow-up study Here are the failure modes that could have been caught if observability were in place: Here are the failure modes that could have been caught if observability were in place: Feature Drift - The model had no monitoring in place for changes in medication codes, lab distributions, and ICD coding practices that had been updated since 2013
Performance Proxy Degradation - There was no dashboard tracking the patients flagged as low-risk by the ESM, who were experiencing rising deterioration rates
Subgroup fairness - No alerts on performance differences across age cohorts or patient demographics over time
Lineage/Schema drift - Epic’s proprietary, opaque architecture meant upstream EHR schema changes had no visibility Feature Drift - The model had no monitoring in place for changes in medication codes, lab distributions, and ICD coding practices that had been updated since 2013 Feature Drift Performance Proxy Degradation - There was no dashboard tracking the patients flagged as low-risk by the ESM, who were experiencing rising deterioration rates Performance Proxy Degradation Subgroup fairness - No alerts on performance differences across age cohorts or patient demographics over time Subgroup fairness Lineage/Schema drift - Epic’s proprietary, opaque architecture meant upstream EHR schema changes had no visibility Lineage/Schema drift Epic eventually released a V2 of the model with gradient boosting and local retraining capability. A proper observability stack would have surfaced these signals, which were hiding in plain sight. released a V2 Five Pillars of AI Observability 1. Data Quality Monitoring 1. Data Quality Monitoring Data Preprocessing is the first step in the Machine Learning lifecycle. Before worrying about model performance, one needs to know whether the data that is being used for training the model is trustworthy. This means monitoring for: Schema Violations i.e. missing fields, type changes etc
Statistical distribution changes i.e. mean, variance, percentile shifts in features
MIssing value rates i.e. sudden spikes in nulls etc
Referential Integrity i.e. broken foreign keys and orphaned records etc Schema Violations i.e. missing fields, type changes etc Statistical distribution changes i.e. mean, variance, percentile shifts in features MIssing value rates i.e. sudden spikes in nulls etc Referential Integrity i.e. broken foreign keys and orphaned records etc 2. Model Performance Monitoring 2. Model Performance Monitoring How do we know if the model is still doing what it was trained to do? The challenge in many real world systems is, ground truth feedback arrives late or not at all. You cannot always know immediately whether the model prediction was right.Two strategies address this. Proxy Metrics - These are upstream signals correlated with model quality. For example, click-through rates for recommendation models, dispute rates for fraud models and return rates for demand forecasting etc
Delayed Evaluation - This batches ground truth as it becomes available and continuously compares performance metrics against historical predictions and recomputes Proxy Metrics - These are upstream signals correlated with model quality. For example, click-through rates for recommendation models, dispute rates for fraud models and return rates for demand forecasting etc Delayed Evaluation - This batches ground truth as it becomes available and continuously compares performance metrics against historical predictions and recomputes 3. Explainability and Interpretability 3. Explainability and Interpretability Knowing something went wrong is not enough. Understanding why it went wrong is true observability. Modern explainability techniques are as below SHAP - Shapley Additive Explanations, assigns each feature a contribution value for individual predictions
LIME - Local Interpretable Model-agnostic Explanations, approximates a model locally with a simple interpretable one
Attention - For transformer based model, Attention Visualization shows which input tokens the model focused on
IG - Integrated Gradients, traces prediction back through a neural network to attribute importance to input features SHAP - Shapley Additive Explanations, assigns each feature a contribution value for individual predictions LIME - Local Interpretable Model-agnostic Explanations, approximates a model locally with a simple interpretable one Attention - For transformer based model, Attention Visualization shows which input tokens the model focused on IG - Integrated Gradients, traces prediction back through a neural network to attribute importance to input features In highly regulated industries like healthcare and finance, explainability is a regulatory requirement. 4. Fairness and Bias Monitoring 4. Fairness and Bias Monitoring A model can be highly accurate on average but failing specific demographic subgroups systematically. For example, in healthcare, a model trained predominantly on data from one population can be inaccurate for under-represented groups, which could have dangerous consequences, even fatal sometimes. Common fairness metrics include demographic parity (equal positive prediction rates across groups), equalized odds (equal true positive and false positive rates) and individual fairness (similar individuals treated similarly) 5. Lineage and Provenance Tracking 5. Lineage and Provenance Tracking Not if, but when a model makes an unexpected prediction, you need the ability to trace back the entire chain i.e. which model version was used, what was the training data, what were the pre processing steps, which feature engineering pipeline provided the inputs etc. Without lineage, tracking and debugging capabilities, solving a bad prediction is impossible. You can't fix what you can't see. Architecture of an AI Observability Stack A production-grade AI observability stack typically has four layers: A production-grade AI observability stack typically has four layers: Layer 1: Instrumentation Layer 1: Instrumentation Layer 1: Instrumentation All prediction requests should emit structured telemetry like input features, model version, prediction output, confidence score, latency and any contextual metadata. This is the raw data that feeds all the downstream layers Layer 2: Collection and Storage Layer 2: Collection and Storage Layer 2: Collection and Storage Telemetry events flow into a streaming pipeline (Kafka is the typical standard) and are stored in a timeseries friendly format. A queryable lakehouse powered by Apache Iceberg is ideal for storing heterogeneous ML and telemetry due to Iceberg’s features like ACID transactions and schema evolution support. Layer 3: Analysis and Detection Layer 3: Analysis and Detection Layer 3: Analysis and Detection This layer runs the statistical tests, trains anomaly detection models on the telemetry itself, and computes drift metrics. Population Stability Index (PSI) and Kullback-Leibler divergence are commonly used for drift detection. Evidently AI and WhyLabs provide pre-built drift detection pipelines. Layer 4: Alerting and Action Layer 4: Alerting and Action Layer 4: Alerting and Action This is the final layer which translates detections into actions. PagerDuty alerts for severe drift and automatic rollbacks can be set up when performance drops below SLAs. Retraining triggers can be set up to kick off CI/CD pipelines to retrain and redeploy models. Publishing the insights on dashboards helps for human review. LLM Observability: Beyond classic ML Observability for Large Language Models (LLMs) further introduces additional complexity because LLM outputs are open-ended text. You cannot really compute accuracy against a ground truth label the same way as classic ML. Different concerns emerge, like below: Hallucination - Detecting hallucinations means identifying when the model confidently states something false
Safety - Toxicity and safety monitoring i.e. detecting harmful, biased or policy violating outputs
Security - Prompt injection detection i.e. finding malicious inputs designed to hijack model behavior
Cost - LLM inference is expensive. Observability must track economics like latency and cost per token
Context- Context window utilization i.e. monitoring how much context is being consumed and whether retrieval is effective Hallucination - Detecting hallucinations means identifying when the model confidently states something false Hallucination Safety - Toxicity and safety monitoring i.e. detecting harmful, biased or policy violating outputs Safety Security - Prompt injection detection i.e. finding malicious inputs designed to hijack model behavior Security Cost - LLM inference is expensive. Observability must track economics like latency and cost per token Cost Context- Context window utilization i.e. monitoring how much context is being consumed and whether retrieval is effective Context Conclusion AI Observability is not a feature. It is a foundational layer for AI systems to avoid trust-breaking failures. As organizations move models that make critical decisions from pilots to production, the cost of a silent model failing grows from embarrassing to catastrophic. The good news is that the AI observability tooling ecosystem has matured significantly. What used to take months or years to build from scratch can now be built with open source components in weeks. The patterns are better understood than before. The cost of not implementing them is far higher in terms of degraded model performance, regulatory risk and eroded trust. The organizations that will succeed with AI are those that treat their models not as one-time deliverables, but as mission-critical systems that require continuous care, measurement, and understanding. AI observability is how you build that understanding and is fundamental for building trust in AI systems. References [1] MIT Technology Review, MIT Report Finds Most AI Business Investments Fail, Reveals 'GenAI Divide' (2025), Virtualization Review [2] ComplexDiscovery, AI Index Report 2025: A Wake-Up Call for Cybersecurity and Legal Oversight (2025), ComplexDiscovery [3] J. Harvey et al., A Scoping Review of Reporting Gaps in FDA-Approved AI Medical Devices (2024), npj Digital Medicine [4] E. Lyons et al., External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients (2021), JAMA Internal Medicine

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

WhyLabs

PagerDuty

Building Reliable AI Systems with AI Observability

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Noonification: Charting a Path for Global AI Regulation (11/14/2023)

AGI 2029: Walking the Fine Line Between Utopia and Existential Risks

Biden Charges Federal Government Agencies to Adopt AI

Biden Signs Executive Order to Support Workers Affected by AI

Biden Unveils White House AI Council to Drive Effective AI Policy Implementation

Defining Key Terms in the Presidential Order on AI Governance

The Noonification: Charting a Path for Global AI Regulation (11/14/2023)

AGI 2029: Walking the Fine Line Between Utopia and Existential Risks

Biden Charges Federal Government Agencies to Adopt AI

Biden Signs Executive Order to Support Workers Affected by AI

Biden Unveils White House AI Council to Drive Effective AI Policy Implementation

Defining Key Terms in the Presidential Order on AI Governance

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps