# Scaling Human-in-the-Loop Machine Learning

Robert Munro | MLOps Meetup | Episode 15 | 55:04
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=LwbbGsuNpao
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/scaling-human-in-the-loop-machine-learning
Published: 2020-06-05
Tags: continual-learning, data-quality, human-in-the-loop

## TL;DR
- Most deployed machine learning systems improve through more human data, but teams often rebuild the same annotation and active learning methods because training rarely covers data preparation and uncertainty.
- Active learning should select data using both model uncertainty and diversity, since random or narrowly defined sampling can reinforce bias and miss examples the model has not seen.
- A production human-in-the-loop system must coordinate the model, annotation workforce, quality control, retraining cadence, infrastructure, and the way automation is presented to users.

## Summary
Robert Munro explains that human-in-the-loop machine learning is already common because supervised systems improve with additional human data. The hard work includes choosing what to label, organizing the workforce, measuring annotation quality, updating models, and helping people act on model outputs. He describes active learning methods based on uncertainty, diversity, entropy, ensembles, and dropout, and says teams should test several methods rather than rely on one heuristic. Quality control can use known reference labels, agreement between annotators, or both, while allowing disagreement when multiple answers are valid. Production systems also need retraining strategies that fit the required update speed. Munro gives examples from health information and misinformation services on WhatsApp, where machine learning first helps human operators find and select responses before any chatbot automation. He is direct about the limits of fully automated data preparation and prediction, especially in health and disaster response.

## Key ideas
### Human-in-the-loop systems are common, but teams often overlook their human components
[07:31](https://www.youtube.com/watch?v=LwbbGsuNpao&t=451s)
Munro says that most deployed machine learning systems get better with more human data, so supervised and lightly supervised systems are already human-in-the-loop systems in practice. The level of sophistication differs. Teams commonly know how to build algorithms, but they are less often taught how to prepare data, interpret uncertainty, or optimize the human side of supervised learning. Annotation work can involve subject matter experts, crowd workers on platforms such as Mechanical Turk, or specialist outsourced workers who label one type of data all day. Munro says data labeling is an overlooked workforce in the machine learning community.

### Active learning should combine uncertainty with a search for missing data
[11:56](https://www.youtube.com/watch?v=LwbbGsuNpao&t=716s)
Active learning chooses which unlabeled items a person should review next. Munro uses news topics as an example: random selection can produce mostly political articles and fail to provide a diverse training set. Uncertainty can be measured through the gap between predictions, ratios, entropy, ensemble predictions, or dropout-based variation. Diversity sampling asks what the model is missing, including examples it classifies confidently but gets wrong because the data has shifted or contains something new. Munro says these methods are relatively simple to implement, but teams often use only one because the alternatives are not covered in many courses or books.

### The annotation workforce and quality process should influence model design
[17:05](https://www.youtube.com/watch?v=LwbbGsuNpao&t=1025s)
Munro describes state-of-the-art human-in-the-loop systems as optimizing the whole cycle at once. The active learning strategy should account for whether labels come from crowd workers, specialists, or subject matter experts. Model interpretability and confidence also matter, even if a simpler model gives lower overall accuracy. Human labels are noisy, so models can incorporate uncertainty in those labels. For quality control, teams can insert items with known answers among new tasks and measure an annotator against that reference data. They can also compare agreement between people, while recognizing that disagreement can be valid when a task allows several good answers.

### Human analysis of data remains necessary because the data is inherently messy
[21:10](https://www.youtube.com/watch?v=LwbbGsuNpao&t=1270s)
Munro does not expect data annotation and analysis to disappear. Human-generated text and speech are noisy, and computer vision data is shaped by choices about cameras, resolution, sampling, and collection conditions. Before choosing a model or annotation strategy, a data scientist can learn what is easy, what is difficult, and whether the problem is better treated as information retrieval or classification. Better tools can make this work more efficient, but Munro says experienced data scientists will continue to spend substantial time looking at data before designing the model.

### Confidence lets a system hand uncertain cases back to people
[23:16](https://www.youtube.com/watch?v=LwbbGsuNpao&t=1396s)
A model that cannot indicate when it is wrong limits the use cases where it can be deployed. Munro gives voice devices as an example: saying that a command is not understood is better than taking the wrong action. Autonomous vehicles and other systems also need a way to hand control to a person. Confidence information helps teams find the parts of a dataset where the model is uncertain and collect more suitable labels. Munro contrasts this with academic benchmarks that focus on accuracy and may not capture how well a system knows its own limits.

### Retraining speed changes the entire production system
[28:35](https://www.youtube.com/watch?v=LwbbGsuNpao&t=1715s)
A static model that takes weeks to train may be unsuitable when incoming data changes quickly. Munro uses news classification to explain that a model can become outdated when new topics appear. Possible designs include a simple model that updates quickly, a more complicated model with only some parameters retrained, or an ensemble that combines a slowly retrained model with a current one. The update cadence affects annotation as well. Near-real-time model updates require near-real-time annotation, while batch updates allow batch labeling. The choice also affects the workforce, quality controls, infrastructure, and monitoring.

### Machine learning can first improve human work before it replaces it
[35:55](https://www.youtube.com/watch?v=LwbbGsuNpao&t=2155s)
Munro describes a World Health Organization WhatsApp service that handles health information in many countries and languages. Rather than immediately deploying a chatbot where errors could be harmful, the team uses customer service representatives and studies the incoming data. A system can suggest five to ten possible responses, allowing a representative to choose a relevant and diverse answer faster than typing one from scratch. Those human choices create training data that could later support more automation. He describes a similar misinformation service for journalists, where a human verifies suggested matches before further automation is attempted.

### Domain expertise matters more than confidence when building high-stakes models
[42:05](https://www.youtube.com/watch?v=LwbbGsuNpao&t=2525s)
Munro warns against people building prediction models in areas where they lack training, using COVID-19 epidemiological models and health misinformation as examples. He says he would work on data preparation, normalization, validation, and annotation for such a system, but would not attempt the epidemiological prediction model himself. Incorrect claims about face coverings also showed how technical errors can become political messaging. In health and disaster response, the cost of an incorrect answer makes human review and appropriate expertise part of the system design.

## Notable quotes
- Robert Munro: "Most of the time when you look at machine learning in industry it is some form of human-in-the-loop machine learning." (07:31)
- Robert Munro: "Active learning is the process of identifying what data is currently not labeled and needs a human label." (11:56)
- Robert Munro: "If you can't trust your machine learning model, you know when it's right or when it's wrong, actually really limits the amount of use cases that you can deploy it for." (23:16)
- Robert Munro: "The solution might be having a model that is simple to begin with and you're relying on data more than the algorithm itself to do the heavy lifting." (29:23)
- Robert Munro: "This enables you to improve both the efficiency and the accuracy of the humans using machine learning without dropping that human-level accuracy." (38:07)

## Tools & references mentioned
- Human-in-the-Loop Machine Learning
- Mechanical Turk
- World Health Organization
- WhatsApp
- Udacity
- PyTorch
- TensorFlow
- Keras
- Stanford Natural Language Processing Group
- AWS
- Figure Eight
- Kubernetes
- Sierra Leone
- Liberia
- Swahili
- Red Cross
- Translators Without Borders
- David
- BenevolentAI

## Who should watch
- You are building a supervised learning product and need to decide what data people should label next.
- Your team is moving from an offline model to a system that needs fresh labels and regular retraining.
- You work with health, disaster response, misinformation, or another setting where uncertain predictions need human review.

## Editor's note

Robert Munro says retraining speed changes the entire production system: near-real-time updates require near-real-time annotation, while batch updates allow batch labeling. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a model or artifact to what produced it and cache unchanged steps instead of recomputing them.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [Designing Human in the Loop Experiences for LLMs](https://mlopstalks.com/talks/designing-human-in-the-loop-experiences-for-llms) (Alberto Rizzoli, V7, 11:40)
- [Continuous Delivery and Automation Pipelines in Machine Learning, Part 1](https://mlopstalks.com/talks/continuous-delivery-and-automation-pipelines-in-machine-learning-part-1) (, 59:05)
- [Continuous Evaluation & Model Experimentation](https://mlopstalks.com/talks/continuous-evaluation-model-experimentation) (Danny Ma, Sydney Data Science, 1:00:46)
- [Product Management in Machine Learning](https://mlopstalks.com/talks/product-management-in-machine-learning) (Laszlo Sragner, Hypergolic, 57:52)