Fiddler combines model monitoring with explainability, analytics, and fairness checks so teams can investigate why a model's behavior changes.
2
Teams register a model with a baseline data set, model metadata, and inference logs, then use Fiddler to inspect drift, data integrity, service metrics, and performance.
3
Fiddler's analysis tools can isolate an underperforming cohort, compare it with the rest of the data, and explain individual predictions with feature impacts and counterfactuals.
Summary
Danny Brock presents Fiddler as a Model Performance Management platform that works alongside an existing machine learning stack. He explains how teams can monitor data drift, prediction drift, data integrity, model performance, and serving metrics, then investigate problems with cohort analysis and explainability. His demo uses a customer churn model. A rise in prediction drift is traced to the number of products feature, whose distribution has moved from mostly ones and twos to more threes and fours. Further slicing shows that the problem is concentrated among customers from Hawaii, where accuracy falls to 0.26 compared with the rest of the data. Brock then shows point-level explanations and a counterfactual change to the number of products. The hands-on section walks through Fiddler's Python client, a Google Colab quick start, baseline data upload, model registration, event publishing, and asynchronous ground-truth updates. He also explains deployment options and the requirement to send data to Fiddler's infrastructure unless the platform is deployed on-premises.
Fiddler combines monitoring with tools for finding causes
Danny Brock describes Fiddler as a Model Performance Management platform that centralizes the models an organization wants to track. It monitors how models behave over time, including input drift, output drift, and accuracy when labels are available. Monitoring is paired with explainability and analytics, so a performance problem can be traced to the cohorts behind it. Fiddler also includes fairness checks. Brock says these capabilities are meant to help model developers, validators, business stakeholders, auditors, and regulators understand model behavior.
The platform covers model development through production
Brock places Fiddler across the model life cycle. During development, teams can inspect training data, bias, and fairness issues. Model validators can examine behavior and sign off before deployment. After release, teams can continue checking whether performance remains similar to the day the model went live. When an issue appears, they can investigate its cause and work toward remediation. He connects this workflow to reducing the time spent on one-off notebook investigations, lowering the cost of building monitoring in-house, and making it easier to scale the number of models under observation.
Fiddler needs a baseline, model metadata, and inference events
To use Fiddler, a team supplies three kinds of information. The baseline reference data set can be training data or another data set that describes the expected production distribution. Model information includes inputs, outputs, and the task being performed. The team also publishes inference events from its production logs. Fiddler uses these inputs to calculate performance information and provide a central view of models. Brock says the platform can run in Fiddler Cloud, on-premises, or in a private cloud, alongside different model frameworks and serving environments.
Drift and integrity monitoring reveal changes in model inputs and outputs
In the demo, Fiddler monitors data drift, model performance, data integrity, and service metrics. Integrity checks can identify null inputs, type mismatches, and values outside an expected range. Service metrics can include traffic counts, inference latency, and serving errors when those values are published. Alerts can be configured for these measurements and delivered through Slack or email. Brock shows prediction drift calculated with Jensen-Shannon Divergence and says users can switch to PSI. Performance monitoring requires ground-truth labels, while drift can provide an earlier view when labels are unavailable.
Cohort analysis traces prediction drift to a specific feature
Brock investigates a prediction-drift alert by comparing feature distributions and feature impact. The number of products accounts for 67 percent of the observed prediction drift. In the training data, that feature is mostly one or two, while the later data contains more threes and fours. This indicates that the model is seeing part of the feature space it did not encounter often during training. Fiddler combines the amount of feature drift with the feature's impact on the model to help prioritize the investigation.
Slicing the data exposes an underperforming geography
Fiddler's Analyze view lets users query inference logs and add metrics to the resulting slice. Brock starts with all inferences from a day, where the example slice has accuracy of 0.66 and many false negatives. Feature correlation suggests that geography may be related to the drift in number of products. Filtering for Hawaii reduces accuracy to 0.26 and makes false negatives much more common. The complementary group, customers outside Hawaii, performs much better. This lets the investigator move from a broad alert to a specific cohort that may need attention.
Point explanations and counterfactuals show why one prediction was made
For individual inferences, Fiddler provides local explanations using methods such as KernelSHAP, Fiddler's variation of KernelSHAP, and other explainers for some model types. Brock sorts for cases where the model is most wrong, then examines a tornado plot showing which inputs influenced a low prediction. The number of products is the strongest factor in the example. Changing that value from three to one in a counterfactual analysis makes the model's estimated likelihood of churn rise. Brock presents this as an explanation that people outside the data science team can inspect.
The hands-on workflow uses a Python client and published events
The workshop uses Fiddler documentation and private Google Colab notebooks. Participants install the Python client, set the environment URL, organization ID, and authentication token, then create a project. The notebook uploads a supplied baseline CSV, profiles its columns, registers the model's task and fields, and publishes inference events. Brock notes that a model artifact can be uploaded for full-fidelity explainability, or Fiddler can create a surrogate model. Ground truth can arrive later: a data frame containing the same inference IDs and updated labels can be published asynchronously.
"Fiddler needs really three things: we need a baseline reference data set, information about your model, and then last but not least we need you to publish in your event logs or your production logs your inferences."Danny Brock10:02
Who should watch
You are responsible for monitoring production models and need a way to move from a drift alert to the cohort causing it.
Your team wants model explanations that business stakeholders or validators can inspect without opening a notebook.
You are evaluating a hands-on monitoring workflow and want to understand what data, metadata, deployment choices, and ground-truth updates Fiddler requires.
