Amit Paka argues that machine learning models need monitoring across their full lifecycle because their performance can decay as production data changes.
2
The proposed EU AI regulation groups systems by risk and places requirements on high-risk applications for data quality, documentation, transparency, human oversight, robustness, accuracy, and security.
3
Model performance management connects explainability, validation, monitoring, root-cause analysis, and feedback from production data into an iterative operating process.
Summary
Amit Paka explains why monitoring a deployed model requires more than tracking a few operational metrics. His experience with recommendation and propensity models at Samsung showed him how difficult it was to manage versions, run useful A/B tests, and understand why a model selected a particular audience. He connects that problem to Fiddler AI's focus on explainability and model performance management. The conversation covers the proposed EU AI regulation, including its four risk categories and the additional requirements proposed for high-risk systems. Paka describes model performance management as visibility across training, validation, deployment, monitoring, and analysis. Teams should inspect data quality and bias before training, examine performance across relevant slices, monitor drift and data integrity after deployment, and use production data to improve later training sets. He also explains how control theory applies when teams monitor a desired threshold and intervene when production behavior moves beyond it.
Model decisions are hard to trust when teams cannot explain their outputs
Amit Paka traces Fiddler AI's origins to problems he saw at Samsung. The team managed recommendation models without the versioning practices engineers take for granted, and its A/B testing compared one model run for a week with another model run the following week. In one campaign, a propensity model selected 300,000 users from a database of 21 million. The campaign did not convert successfully, and the team could not tell whether it had targeted the wrong audience because the explanation was simply that the model had chosen them. Paka says this lack of transparency kept appearing in practice. He connected it with similar concerns about bias in Facebook's news feed, which led to the company's mission of helping teams build trust with AI.
The proposed EU AI regulation separates systems by the risk they create
Paka describes the proposed EU regulation as a detailed framework built around human-centred and trustworthy AI. It divides applications into four groups. Unacceptable-risk applications, such as behaviour manipulation, would be banned. High-risk applications, including self-driving cars, credit scoring, recruitment, and robotic surgery, would face new oversight. Limited-risk applications, such as chatbots, would have transparency requirements, including telling people when they are interacting with AI. Minimal-risk applications, such as spam filters, would have no proposed intervention. Paka says teams should first determine whether a use case falls into the high-risk category, since that determines whether requirements around data, documentation, transparency, human control, and operational monitoring apply.
High-risk systems need operational evidence after deployment
For high-risk applications, Paka lists requirements for high-quality data sets, documentation, record keeping, transparency, fairness and bias analysis, and human oversight. He distinguishes transparency from control. People need to understand how a model was built and how it operates, while human oversight also requires the ability to intervene. Operational visibility is needed to check whether a model still performs at the level established during training. A self-driving system might encounter live data that differs sharply from its training environment, creating a safety problem. Paka groups the needed operational checks under robustness, accuracy, and security. The proposed regulation would also require enough records to review what happened when a system fails.
Explainability matters most where models affect regulated decisions or expensive choices
Paka says the urgent demand for explainability comes from companies that already face practical or regulatory pressure. In US banking, SR 117 requires visibility into how quantitative models are trained and behave, as well as evidence that they continue to work after deployment. He also cites banking, healthcare, and recruitment as areas where transparency is directly relevant. Other businesses need explanations because an opaque decision can create a large financial loss. In oil and natural gas, an AI recommendation about where to drill can affect a decision worth 10 to 20 million dollars. Paka also points to public concern about bias in facial recognition, where poor performance on darker skin tones led companies to leave that market. Explainability can help with compliance, customer acceptance, debugging, and product development.
Model performance management covers the full model lifecycle
Paka defines model performance management as visibility across the iterative model lifecycle, with the goal of understanding and improving performance. During training, teams can inspect data quality and bias and use production feedback to create a more representative training set. During validation, they can examine slices such as loans to men and women, performance in different states, or other business-relevant groups. During deployment, they can record traffic and compare a champion model with challengers. Monitoring then checks drift, data integrity, and outliers. When a problem appears, analysis should help find its cause, using the same slices examined during validation. Paka describes this as a loop in which models converge and diverge from their intended performance as the data changes.
Monitoring must use slices and leading signals when labels arrive late
Paka argues that broad monitoring is insufficient when a team cares about a specific fairness threshold, region, or business group. The monitoring system should examine the production traffic through the same slices used to validate the model. Accuracy can be monitored when live ground-truth labels arrive, but many systems do not receive those labels in real time. In that case, data drift can provide a leading signal. Data integrity problems and meaningful changes in the distribution of live inputs can indicate that the model is facing a different reality from the one represented in its training data. Teams can then feed that production representation into a later training set instead of continuing to rely on a hypothetical data distribution.
Control theory adds intervention to model monitoring
Paka uses a car to explain the control-theory analogy. A sensor can measure speed, but a control system also changes the car's behaviour when the measured speed differs from the desired speed. For a model, the target may be an acceptable level of bias rather than the highest possible accuracy. A model can meet that threshold during training and breach it after deployment as the data shifts. A control-oriented process detects the breach and feeds the new data representation back into training so a new model can meet the threshold in the changed environment. Paka says this requires two stages: first, teams need operational insight into how the model behaves; then they need a way to act when its behaviour moves away from the desired metrics.
Ownership of deployed models is still being divided across roles
Paka says model performance work is shared across data scientists, machine learning engineers, and, in some organisations, DevOps teams that previously monitored web applications. Data scientists want to understand feature importance, low-performing regions, protected-class behaviour, and other properties of the models they build. Machine learning engineers or operations teams may take responsibility for deployed models, while decision-makers, customer-support staff, and other stakeholders need understandable explanations of model outputs. Paka is direct that these job boundaries are still being formed. Organisations may have several groups involved without a clear MLOps team or an agreed owner. He also gives each group a different reason to adopt monitoring, from protecting business value and reducing manual model babysitting to debugging models and explaining them to regulators.
