Model monitoring fails when teams treat it as a technical problem and ignore ownership, service-level agreements, escalation paths, and stakeholder expectations.
2
Monitoring at scale creates large amounts of data, so teams need anomaly detection and context instead of fixed thresholds applied across every model and population.
3
Automatic retraining is only one response to a model issue, and it can make performance worse when the training window contains bad or unusual data.
Summary
Oren Razon argues that model monitoring becomes difficult when machine learning moves from experimentation into business operations. Technical monitoring of drift, data quality, inputs, outputs, labels, and performance is only part of the work. Teams also need to decide what counts as an issue, who owns each type of issue, where alerts should go, and what action should follow. Razon warns against treating retraining as a universal reset button. Depending on the cause, the right response might be fixing an upstream data source, splitting a subpopulation into a separate model, recalibrating outputs, refitting preprocessing values, or changing a classification threshold. He recommends using production data to guide later research and development. Monitoring should show the full context of a change and fit into the tools and processes that each stakeholder already uses.
Model monitoring has organizational work around the technical metrics
Razon says model monitoring is now widely accepted as part of an MLOps infrastructure, but teams miss its purpose when they focus only on computing drift, metrics, and integrations. Models drive business decisions, so an issue needs an owner, an SLA process, and a way to resolve it. Data scientists, ML engineers, business teams, and other people depend on the model. The team has to decide who should know about an issue, how it should be handled, and what process applies. These decisions should be made while implementing observability, rather than after the system is already producing large amounts of information.
Monitoring scales through combinations of models, populations, features, and metrics
The number of things to monitor grows quickly as a model operation expands. Razon gives an example with 10 models, 10 countries or customers for each model, dozens or 100 features, and five basic metrics such as the mean, standard deviation, and drift. He says this can produce more than 50,000 data points every minute. Without processes for deciding what matters, the team is flooded with noise and may end up having someone watch dashboards all day. Scale therefore changes the operating problem, even when the individual monitoring checks are simple.
Anomalies matter more than universal metric thresholds
Traditional IT monitoring can use a clear rule such as treating CPU above 80 percent as bad. Machine learning data is stochastic, with changing distributions, noise, and seasonality. A KL divergence score of 2.34 does not have a universal meaning without context. Razon recommends detecting abnormal behavior relative to a particular metric, population, and use case. In his example, features have different drift scales, yet each can show an anomaly because it has moved away from its normal behavior. A fixed rule such as drift above 10 is therefore unsuitable across all models.
Retraining can make a model worse when its training window contains an incident
Razon challenges the assumption that every monitoring issue should trigger retraining. He describes a weekly automatic retraining setup that uses three weeks of data, with a one-week delay while labels mature. If one of those weeks contains a data quality problem, the automated process can include that problem in the next training set. The model may then learn misleading patterns and lose performance. The team needs to inspect the context of the issue before choosing a response.
Teams have several responses besides rebuilding the entire model
Possible responses depend on what changed. A data integrity issue may require fixing an upstream source or pipeline. If only one subpopulation is drifting, the team might create a separate model for that group instead of retraining the whole system. A regression model for house prices might need recalibration when market prices rise while the underlying patterns remain stable. Other options include refitting preprocessing values such as the mean, standard deviation, or Z score, or changing a binary classification threshold. Razon presents retraining as one item in a wider response strategy.
The right action requires context across inputs, outputs, labels, and business performance
A shift in model outputs does not explain its own cause. Razon uses a binary classifier whose output changes from 30 percent male and 70 percent female to 60 percent male and 40 percent female. That change could come from the model, its inputs, or a data quality problem. The team should compare the output anomaly with input behavior, labels, and current business performance. A correlated data quality event may explain the output shift, although correlation does not prove causality. Full observability gives the team more information before it chooses an intervention.
Different stakeholders need different issues sent through their existing workflows
Razon divides concerns among data scientists, ML engineers, and business stakeholders. Data scientists may care about drift, weak spots, and whether the model remains relevant after retraining. ML engineers care about infrastructure, orchestration, and data source problems. Business teams may need to know when a fraud model starts denying more transactions. Each issue needs a clear owner, and one issue may have more than one owner. Monitoring also has to fit existing work habits, such as BI dashboards, email, Slack, PagerDuty, New Relic, DataDog, or Prometheus, instead of forcing every person into a separate platform.
Production data should guide the next model development cycle
Razon describes model development as an iterative cycle rather than a straight line from business definition to research, engineering, and monitoring. Once monitoring is in production, teams can use real production distributions to choose relevant training history, find weak subpopulations, and understand current performance. They can also use production monitoring during a safe rollout to check that a new model is valid. This changes the development mindset. Instead of selecting six months of data because it seems representative, a data scientist can inspect which history and populations resemble current production behavior.
"Instead of starting from the lab, starting from production, looking on production, understanding what's going on, and based on that, developing your next model iteration."Oren Razon44:44
Who should watch
Your team has deployed several models and is receiving more monitoring signals than it can act on.
You are deciding whether drift should trigger automatic retraining and need a wider set of possible responses.
Model alerts reach a data scientist by default, even though business and ML engineering teams own different parts of the process.
