Model drift means that a model's predictions get worse over time, and the causes can include changing data, changing relationships, or data integrity failures.
2
When labels arrive late or are unavailable, monitoring feature and label distributions can provide an early signal that model performance may change.
3
Finding the drifting features, their importance to the model, and the affected traffic helps teams trace a performance problem to its cause.
Summary
Amy Hodler explains model drift as predictions getting worse over time. She separates the way drift appears, such as abrupt, gradual, periodic, or isolated changes, from the causes behind it. Concept drift changes the relationship between inputs and outcomes. Feature drift changes inputs, while label drift changes the distribution of outcomes. Data integrity failures can look like drift, including swapped fields, missing values, schema changes, or a pipeline that sends cents where the model expects dollars. Hodler recommends performance monitoring when ground-truth labels are available. Without timely labels, distribution metrics and unsupervised methods can give earlier warnings. She discusses PSI, KL divergence, Jensen-Shannon divergence, and the Kolmogorov-Smirnov test, while stressing that thresholds depend on the use case. Investigation should connect overall drift with feature importance and feature-level changes. Possible responses include retraining, relabeling, changing model weights, adjusting business logic, or using different models on a schedule.
Hodler defines model drift in practical terms: predictions get worse over time. She says the literature uses model drift, model decay, and prediction drift in overlapping ways, and different authors may be describing the experience, cause, or measurement method. Several types can happen together, with one causing another. Her advice is to avoid spending too much time arguing over labels. Teams need to detect a change, investigate its root cause, and decide whether action is needed. A model can face real-world change even when the terminology used to describe it differs across papers or tools.
Concept drift changes the relationship between data and the outcome
Concept drift occurs when the relationship between inputs and the thing being predicted changes. Hodler illustrates this with buying behavior during the pandemic. The observed data points may look similar, but the behavior behind the decision has changed, so the old decision boundary no longer describes reality. This can be difficult to detect and prove, especially when production ground-truth labels arrive late. In a lending example, a change in economic conditions can alter the creditworthiness threshold itself. When the decision relationship changes, retraining with newer information may be needed.
Feature drift and label drift describe different changes in data
Hodler prefers the more specific terms feature drift and label drift when discussing data drift. Label drift means that the probability of an outcome changes, such as a larger share of loan applicants becoming creditworthy. Accuracy may stay stable, yet the business may need to respond if it does not have enough cash to lend to the larger eligible group. Feature drift means that inputs change, such as applicant income, region, or income-to-debt patterns. The model may remain accurate, or its accuracy may fall if it receives data unlike its training data. Some data change is expected, so teams need to judge its effect and business consequences.
A pipeline or source-data error can create what looks like model drift. Hodler gives examples of debt-to-income and age fields being accidentally swapped, blank fields producing a growing set of null values, and a data pipeline sending transaction amounts in cents after earlier data used dollars. In the currency example, the model continued to run while its performance gradually worsened as more incorrect values entered the system. Schema changes and new product fields can create similar problems. Monitoring missing values, outliers, and schema mismatches is an early troubleshooting step because it can remove simple data failures from the investigation.
Labels determine which monitoring methods are available
When reliable ground-truth labels are available, Hodler recommends monitoring performance metrics such as AUC, false-positive rates, false-negative rates, accuracy, precision, and recall. A substantial drop is a direct warning that something is wrong. Labels often arrive late in production, especially in streaming cases such as credit-card fraud, and they can be expensive or unreliable. Without them, feature and data-distribution monitoring can provide an earlier signal of later label or concept changes. Data integrity checks add another early layer by detecting missing values and outliers before they become harder to diagnose.
Distribution metrics need thresholds that match the use case
For distribution shift, Hodler discusses population stability index, KL divergence, Jensen-Shannon divergence, and the Kolmogorov-Smirnov test. PSI compares a current scoring distribution with the distribution in training data. KL divergence compares one probability distribution with another. Fiddler uses Jensen-Shannon divergence because it is symmetric and always has a finite value. The Kolmogorov-Smirnov test can help with distributions that are not normal. Hodler says thresholds depend on the use case and the team's tolerance for false positives and false negatives. Some drift is expected, so an alert should not automatically trigger a response to every change.
Feature-level investigation connects drift alerts to root cause
After a continuous monitoring system detects drift against a baseline, Hodler recommends drilling into individual features. Teams should compare how much a feature matters to the prediction with how much its distribution has changed. A feature can drift substantially without affecting accuracy if it has little influence on the model. Conversely, a drifting feature that carries significant predictive weight deserves closer attention. Traffic patterns matter too. The combination of feature importance, feature drift, and the affected traffic can point toward the source of the problem. Hodler connects this work with explainability because an alert alone does not explain what to change.
Teams can respond through retraining or targeted model changes
Hodler describes several responses after a team understands the problem. Retraining with new data, or relabeling older data, can address a changed relationship. Teams can also change model behavior, adjust weights, alter business logic, use a second model, or schedule different models for different periods. Her fraud example involves false positives rising during a particular weekend pattern. The general model may still work, but the team could give weekend information more weight, collect different information, or use a model that is scheduled for that period. The right response depends on whether the issue is concept change, feature change, or data failure.
"It doesn't matter if it's the data change or the reality, the concept change or, you know, what actually caused it, as long as you're tracking where there are issues, you're able to analyze to root cause and then of course you're taking care of it."Amy Hodler20:19
Who should watch
You have a production model whose accuracy is declining, but you do not yet know whether the cause is changing behavior, changing inputs, or a broken data pipeline.
Your labels arrive too late to support useful alerts, so you need distribution and integrity checks that can warn about problems earlier.
You are building a monitoring practice and need a way to connect an overall drift alert with the features and traffic that may explain it.
