Explainability is a foundation for responsible AI because teams need to understand how models make decisions before they can validate, monitor, and debug them.
2
Explainability helps teams find weak data segments, false positives, and possible bias, then use those findings to improve features, training examples, or model design.
3
Explanations show how a model behaves through its learned patterns, but they do not establish causation or explain events that the model did not capture in its data.
Summary
Krishna Gade explains why machine learning models create an operational problem that ordinary software does not. Models are black boxes whose behavior can change with incoming data, so teams cannot responsibly deploy them without ways to validate, monitor, debug, and explain their decisions. He connects this need to high-stakes uses such as lending, clinical diagnosis, and recruiting, while showing that less obvious systems can also create unfair outcomes. Explainability gives teams feature attributions and data slices that reveal where a model performs poorly, produces false positives, or may treat groups differently. Those findings can guide new training examples, feature changes, or different model architectures. Gade distinguishes interpretable models, such as some decision trees and linear models, from post-hoc explanations for black-box models. He is also clear about the limits: explanations describe the model's learned patterns, not real-world causes. Building a general platform remains difficult because models have different formats, APIs, and levels of complexity.
Machine learning models need operational visibility because they are black boxes that change with data
Gade describes the model as a new kind of software artifact. Unlike ordinary code, it cannot be opened and read line by line to understand every rule. Even a simple logistic regression model or decision tree can be difficult to inspect in the way engineers inspect source code. Model behavior also changes with the data received in production. A system trained on one dataset may encounter different data after deployment, so its predictions can surprise the people operating it. Gade says these properties make models difficult to operationalize. Teams need to understand, test, monitor, and debug them before they can claim that their systems make responsible decisions.
Responsible AI is a process that spans training data, validation, monitoring, and debugging
Gade defines responsible AI as a process framework for the full model lifecycle. Teams should assess bias in the training data, validate the model, monitor it after deployment, and have ways to debug it when people ask why a decision was made. This matters especially for lending, mortgages, jobs, and clinical diagnosis, where a prediction can affect someone's life. Explainability is central because a team cannot properly understand or investigate a system if it cannot explain its predictions. For Gade, the explanation is part of the operational foundation needed to make responsible decisions, rather than an extra feature added after deployment.
Responsible AI concerns can appear in ordinary services that seem far removed from high-stakes decisions
Gade says teams often focus on obvious cases such as credit underwriting, recruiting, and clinical diagnosis, but less obvious services can also produce unfair outcomes. He cites a UK logistics company that was sued after its delivery-order system treated some people differently. Similar questions arise for services such as DoorDash, Uber, and housing marketplaces. Their algorithms may affect employment, the assignment of orders, or access to housing. Even when a system does not directly make a formal decision about a person's loan or diagnosis, its patterns can still affect people. Gade argues that teams should examine those effects instead of assuming that only traditional high-stakes applications need responsible AI practices.
Explanations can turn model errors into concrete changes to data and model design
Gade describes a fraud model that produces false positives. A team can examine the explanations for those cases, group common attributions, and look for patterns such as transactions from a particular region being classified as fraud. The pattern may point to missing or poorly represented training examples, a problematic feature, or a model that is not suitable for that segment. Engineers can then add positive and negative examples, change the features, try another architecture, or use an ensemble of models. Explainability reduces the need for a person to search through large volumes of data manually. It also supports model improvement, since the same insights used to answer stakeholder questions can reveal performance problems.
Explainability can help before training, but it only provides a view through the model
Gade says explainability can be used during feature selection, before a model is fully built. Teams can inspect feature importance, feature correlations, and partial dependence plots to decide which inputs deserve more investigation or might be removed. He gives a firm limit to this approach: an explanation is a lens into what the model learned, not a causal account of the world. If an external event causes a store's sales to fall, such as people stopping visits during the coronavirus pandemic, the explanation cannot establish that cause unless the model captured it. The team is still looking at data through the model's learned patterns.
Shapley-based methods provide theory, while interactive testing helps people judge an explanation
Gade explains that attribution methods based on Shapley values borrow a game-theory idea. Features are treated like players whose combined contribution produces a prediction, and the prediction is divided into feature attributions. Shapley methods have axiomatic properties, such as giving equivalent features equivalent values and making all attributions add up to the prediction. Gade says these properties have made Shapley-based techniques a strong standard for explainability, with related methods including integrated gradients. Theory does not settle whether a person should trust an explanation, though. Fiddler supports what-if analysis, where a user changes a feature value and observes how the prediction or explanation changes. Gade expects better metrics for explanation quality over time.
Interpretability describes the model itself, while explainability usually describes a post-hoc analysis
Gade separates two terms that are often used together. An interpretable model exposes its own logic, as with a decision tree whose prediction path can be followed or a linear model whose coefficients can be inspected. Explainability usually takes a model that is not directly interpretable, such as a neural network, and produces an explanation for a prediction. Methods such as Shapley values, LIME, and integrated gradients can be applied after the model has been built. This distinction matters because a team may choose a model for its predictive performance and then need a separate method to understand it. The explanation is an analysis of the model, rather than a readable version of its source code.
Complex and changing models make general-purpose explainability difficult to build
Gade identifies model complexity and platform diversity as practical barriers. Hybrid models may combine tabular inputs with sequences, while ensembles and reinforcement-learning systems add more layers or change continuously. Explanations may be needed at different levels of such a model, and methods that need internal information, such as gradients, require access to the model artifact. A general-purpose platform also has to work across TensorFlow, PyTorch, scikit-learn, XGBoost, MLflow, Spark, and other formats. Gade says container technology helps by providing an abstraction with a standard API for inputs and outputs. That approach can support common explanation methods, but some methods still need deeper access to the model and its internals.
"Ultimately explainability is to find bugs or performance or problems within the model, or explain the model and also find issues within the model so you can actually make it better."Krishna Gade50:17
Who should watch
You are deploying models that affect lending, recruiting, healthcare, delivery work, housing, or another part of people's lives and need a way to inspect their decisions.
Your model produces false positives or performs unevenly across regions or user groups, and you need evidence that can guide data and model changes.
You are building model infrastructure across several frameworks and want to understand what a general-purpose explainability system can and cannot support.
