# Trustworthy Machine Learning

Kush Varshney, IBM Research | MLOps Coffee Sessions | Episode 124 | 52:34
Hosted by Krishnaram Kenthapadi

Source: https://www.youtube.com/watch?v=ASSrHdTQhJo
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/trustworthy-machine-learning
Published: 2022-09-21
Tags: drift, governance, guardrails, monitoring

## TL;DR
- Trustworthy machine learning applies across the full lifecycle, from problem specification through deployment and monitoring.
- Trust depends on more than model accuracy. Systems also need appropriate fairness, explainability, robustness, uncertainty calibration, and stakeholder input.
- Kush Varshney argues that problem specification and post-deployment evaluation need more human attention than the modeling stage itself.

## Summary
Kush Varshney explains trustworthy machine learning as a way of working across the entire ML lifecycle. It begins with deciding whether a problem should be addressed, identifying affected communities, and choosing metrics with input from policy, legal, product, business, and domain stakeholders. Trust involves competence, reliability, communication, and goals beyond the system's own operation. Model users also need calibrated trust, since neural networks can express high confidence when they are wrong. The discussion covers competing definitions of fairness, the risk of scaling bias through a single model, and the need to monitor deployed systems for fairness, drift, robustness, and other properties. Varshney describes IBM's open-source AI Fairness 360, AI Explainability 360, Adversarial Robustness 360, and Uncertainty Quantification 360 toolkits. He also discusses why large language models make trustworthy ML harder because their possible harms are still difficult to categorize.

## Key ideas
### Trustworthy ML includes the people affected by a model
[02:36](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=156s)
Kush Varshney says trustworthy ML is for more than model developers. Engineers need enough understanding to build systems that others can trust, while doctors, lending officers, patients, loan applicants, regulators, and other affected people have different needs. He distinguishes interpersonal trust from institutional trust, then describes trust through competence, reliability, communication, and concern for goals beyond the system's own interests. A system should also be worthy of trust, since people can trust something more than they should.

### Trust needs to be calibrated to a system's limits
[14:40](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=880s)
Krishnaram Kenthapadi points out that ML systems can behave in unexpected ways, unlike machines whose behavior is more deterministic. Kush Varshney says users need to know when a model should be trusted, when they should use their own judgment, and what its failure modes are. Explainability can help people understand decisions. Uncertainty quantification can show whether a system is confident or guessing. Varshney warns that neural networks are often poorly calibrated, so a model may say it is 99 percent sure when it should not.

### Fairness has competing definitions and value choices
[16:59](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=1019s)
Varshney describes fairness as a contested idea with philosophical and sociological roots. Disparate impact ratio and statistical parity difference focus on equal selection rates across groups, such as men and women having the same chance of receiving a loan. Average odds difference focuses on equal underlying accuracy instead. These measures reflect different assumptions about whether the inputs already contain social bias. Because the definitions can conflict, teams need to decide which values matter with their policy, compliance, leadership, and other organizational stakeholders.

### Bias can be amplified when one model operates at scale
[22:08](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=1328s)
Krishnaram Kenthapadi connects trustworthy ML to the way a biased model can spread decisions across a country or several countries. Varshney agrees that scale is a major concern. Different human decision makers may have different biases, while one model can apply the same bias everywhere. He compares this with ensemble models, where averaging multiple models can reduce some biases. The example shows why a model's reach changes the consequences of errors and unfair decisions.

### Trustworthiness starts before model training
[28:44](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=1724s)
Varshney places possible harms at every stage of the lifecycle. Teams should first ask whether they should work on a problem, then decide which metrics matter, collect and prepare data, build and evaluate the model, and monitor it after deployment. He gives a healthcare example in which combining different types of healthcare costs into one variable created more racial bias than keeping inpatient, outpatient, and emergency costs separate. A routine modeling choice, such as combining variables with many zero values, can therefore affect fairness.

### Monitoring must cover more than accuracy
[31:09](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=1869s)
For modeling, Varshney recommends mitigation methods such as the pre-processing, in-processing, and post-processing techniques in AI Fairness 360. During evaluation, teams should include people with lived experience of marginalization because they may recognize harms that others miss. After deployment, monitoring should continue rather than allowing the system to run unattended. Teams should compute the fairness and drift metrics they identified earlier. Krishnaram Kenthapadi adds that model monitoring is a practical industry problem, while academic work often concentrates on data preparation and validation.

### The least automated parts of ML need the most human judgment
[35:06](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=2106s)
Varshney compares an ML lifecycle with flying an airplane. The modeling stage resembles cruising and may become increasingly automated. Problem specification resembles takeoff, while evaluation and related post-deployment work resemble landing. Those stages need human judgment and are difficult to automate. One open problem is estimating model drift when ground-truth labels arrive late or are unavailable. This makes monitoring an active research area as well as an operational task.

### Large language models expand the range of possible harms
[40:32](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=2432s)
Varshney says trustworthy ML is harder to define for large language and foundation models than for tabular data. Possible problems include occupational gender bias, unequal sentiment, toxicity, and a lack of common-sense reasoning. He says researchers have not yet fully categorized or collected the harmful behaviors these models can produce. Their broad capabilities also make them easy to repurpose for uses that were not part of the original goal, which creates another need to understand their behavior.

### Open-source toolkits turn trustworthy ML research into practice
[46:44](https://www.youtube.com/watch?v=ASSrHdTQhJo&t=2804s)
Varshney describes a set of IBM Research projects intended for practicing data scientists. AI Fairness 360 and Adversarial Robustness 360 came first, followed by AI Explainability 360 and Uncertainty Quantification 360. The projects include algorithms, Python packages, reference material, interactive demos, and glossaries. AI Fairness 360 also has an R version. The fairness, adversarial robustness, and explainability projects were donated to the Linux Foundation, while the team was working toward doing the same with others.

## Notable quotes
- Kush Varshney: "Trustworthy ML is a way of thinking and something to be worked on and operationalized throughout the entire machine learning development lifecycle." (28:44)
- Kush Varshney: "We want to get to a point not only where these systems are trusted but they're worthy to be trusted." (14:22)
- Kush Varshney: "A single machine learning model that's applied at a very large scale, it has exactly one bias and that's gonna apply across." (23:34)
- Kush Varshney: "In real world practice, it's the beginning and the end that are the important part." (34:43)
- Kush Varshney: "I think we haven't even gotten to a point where we can even categorize or collect all of those harmful behaviors from large language models." (43:27)

## Tools & references mentioned
- IBM Research
- Fiddler
- Trustworthy Machine Learning
- AI Fairness 360
- Adversarial Robustness 360
- AI Explainability 360
- Uncertainty Quantification 360
- Linux Foundation
- Kathy O'Neill
- Weapons of Math Destruction
- David Donoho
- European AI Act
- GPT-3
- DALL-E
- Chloroquine

## Who should watch
- You are building or operating ML systems whose decisions affect people and need a way to include fairness, explainability, and other trust properties in the lifecycle.
- Your team monitors production models but struggles with delayed ground truth, drift, or deciding which metrics belong in production monitoring.
- You want practical context for IBM's open-source trustworthy AI toolkits and a clearer account of why large language models create additional trust problems.

## Related talks

- [Reliable Machine Learning](https://mlopstalks.com/talks/reliable-machine-learning) (Niall Murphy, Stanza Systems & Todd Underwood, Google, 1:02:25)
- [Model Monitoring in Practice: Top Trends](https://mlopstalks.com/talks/model-monitoring-in-practice-top-trends) (Krishnaram Kenthapadi, Fiddler AI, 51:34)
- [Maturing Machine Learning in Enterprise](https://mlopstalks.com/talks/maturing-machine-learning-in-enterprise) (Kyle Gallatin, Etsy, 47:09)
- [GenAI: An Unreliable Information Store](https://mlopstalks.com/talks/genai-an-unreliable-information-store) (Noble Ackerson, Venta, 27:55)
- [High Stakes ML: Active Failures, Latent Factors](https://mlopstalks.com/talks/high-stakes-ml-active-failures-latent-factors) (Flavio Clesio, MyHammer AG, 55:27)