Most deployed machine learning systems improve through more human data, but teams often rebuild the same annotation and active learning methods because training rarely covers data preparation and uncertainty.
2
Active learning should select data using both model uncertainty and diversity, since random or narrowly defined sampling can reinforce bias and miss examples the model has not seen.
3
A production human-in-the-loop system must coordinate the model, annotation workforce, quality control, retraining cadence, infrastructure, and the way automation is presented to users.
Summary
Robert Munro explains that human-in-the-loop machine learning is already common because supervised systems improve with additional human data. The hard work includes choosing what to label, organizing the workforce, measuring annotation quality, updating models, and helping people act on model outputs. He describes active learning methods based on uncertainty, diversity, entropy, ensembles, and dropout, and says teams should test several methods rather than rely on one heuristic. Quality control can use known reference labels, agreement between annotators, or both, while allowing disagreement when multiple answers are valid. Production systems also need retraining strategies that fit the required update speed. Munro gives examples from health information and misinformation services on WhatsApp, where machine learning first helps human operators find and select responses before any chatbot automation. He is direct about the limits of fully automated data preparation and prediction, especially in health and disaster response.
Human-in-the-loop systems are common, but teams often overlook their human components
Munro says that most deployed machine learning systems get better with more human data, so supervised and lightly supervised systems are already human-in-the-loop systems in practice. The level of sophistication differs. Teams commonly know how to build algorithms, but they are less often taught how to prepare data, interpret uncertainty, or optimize the human side of supervised learning. Annotation work can involve subject matter experts, crowd workers on platforms such as Mechanical Turk, or specialist outsourced workers who label one type of data all day. Munro says data labeling is an overlooked workforce in the machine learning community.
Active learning should combine uncertainty with a search for missing data
Active learning chooses which unlabeled items a person should review next. Munro uses news topics as an example: random selection can produce mostly political articles and fail to provide a diverse training set. Uncertainty can be measured through the gap between predictions, ratios, entropy, ensemble predictions, or dropout-based variation. Diversity sampling asks what the model is missing, including examples it classifies confidently but gets wrong because the data has shifted or contains something new. Munro says these methods are relatively simple to implement, but teams often use only one because the alternatives are not covered in many courses or books.
The annotation workforce and quality process should influence model design
Munro describes state-of-the-art human-in-the-loop systems as optimizing the whole cycle at once. The active learning strategy should account for whether labels come from crowd workers, specialists, or subject matter experts. Model interpretability and confidence also matter, even if a simpler model gives lower overall accuracy. Human labels are noisy, so models can incorporate uncertainty in those labels. For quality control, teams can insert items with known answers among new tasks and measure an annotator against that reference data. They can also compare agreement between people, while recognizing that disagreement can be valid when a task allows several good answers.
Human analysis of data remains necessary because the data is inherently messy
Munro does not expect data annotation and analysis to disappear. Human-generated text and speech are noisy, and computer vision data is shaped by choices about cameras, resolution, sampling, and collection conditions. Before choosing a model or annotation strategy, a data scientist can learn what is easy, what is difficult, and whether the problem is better treated as information retrieval or classification. Better tools can make this work more efficient, but Munro says experienced data scientists will continue to spend substantial time looking at data before designing the model.
Confidence lets a system hand uncertain cases back to people
A model that cannot indicate when it is wrong limits the use cases where it can be deployed. Munro gives voice devices as an example: saying that a command is not understood is better than taking the wrong action. Autonomous vehicles and other systems also need a way to hand control to a person. Confidence information helps teams find the parts of a dataset where the model is uncertain and collect more suitable labels. Munro contrasts this with academic benchmarks that focus on accuracy and may not capture how well a system knows its own limits.
Retraining speed changes the entire production system
A static model that takes weeks to train may be unsuitable when incoming data changes quickly. Munro uses news classification to explain that a model can become outdated when new topics appear. Possible designs include a simple model that updates quickly, a more complicated model with only some parameters retrained, or an ensemble that combines a slowly retrained model with a current one. The update cadence affects annotation as well. Near-real-time model updates require near-real-time annotation, while batch updates allow batch labeling. The choice also affects the workforce, quality controls, infrastructure, and monitoring.
Machine learning can first improve human work before it replaces it
Munro describes a World Health Organization WhatsApp service that handles health information in many countries and languages. Rather than immediately deploying a chatbot where errors could be harmful, the team uses customer service representatives and studies the incoming data. A system can suggest five to ten possible responses, allowing a representative to choose a relevant and diverse answer faster than typing one from scratch. Those human choices create training data that could later support more automation. He describes a similar misinformation service for journalists, where a human verifies suggested matches before further automation is attempted.
Domain expertise matters more than confidence when building high-stakes models
Munro warns against people building prediction models in areas where they lack training, using COVID-19 epidemiological models and health misinformation as examples. He says he would work on data preparation, normalization, validation, and annotation for such a system, but would not attempt the epidemiological prediction model himself. Incorrect claims about face coverings also showed how technical errors can become political messaging. In health and disaster response, the cost of an incorrect answer makes human review and appropriate expertise part of the system design.
"If you can't trust your machine learning model, you know when it's right or when it's wrong, actually really limits the amount of use cases that you can deploy it for."Robert Munro23:16
Who should watch
You are building a supervised learning product and need to decide what data people should label next.
Your team is moving from an offline model to a system that needs fresh labels and regular retraining.
You work with health, disaster response, misinformation, or another setting where uncertain predictions need human review.
