# Automating Data Annotation with LLMs

Nikolai Liubimov, Michael Malyuk & Chris Hoge, HumanSignal | LLMs in Production 2023 | 1:03:04

Source: https://www.youtube.com/watch?v=mTVNE0Sw5vI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/automating-data-annotation-with-llms
Published: 2023-10-20
Tags: human-in-the-loop, prompt-engineering, structured-outputs

## TL;DR
- LLMs can create initial labels, but human review is needed to turn predictions into ground truth data.
- Prompt refinement improves labeling by using reviewed errors and examples from the ground truth set.
- Constrained generation, active learning, few-shot prompting, chain-of-thought reasoning, and self-consistency can improve automated labels, with added cost or complexity.

## Summary
The workshop presents a human-in-the-loop workflow for using LLMs to label data. Chris Hoge explains why human annotations remain the reference point while large models need far more data than people can label. Michael Malyuk then shows a progression from naive zero-shot prompts to constrained generation, confidence scores, active learning, and prompt optimization. The demo uses product reviews classified as subjective opinions or objective facts. Initial model outputs contain unwanted free-form text, so the workflow restricts outputs to the allowed classes and sends uncertain examples to Label Studio for review. Reviewed labels form a ground truth set that can expose errors and guide better prompts. The speakers also cover few-shot examples, chain-of-thought rationales, self-consistency, and an automated loop that rewrites instructions based on observed mistakes. They are candid that domain knowledge, annotation disagreement, context limits, privacy, and API costs affect whether the approach works.

## Key ideas
### Human labels remain the reference point for automated annotation
[05:22](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=322s)
Chris Hoge argues that data reviewed by human annotators remains the ground truth for developing and training models. Human labeling takes time and attention, and some tasks require several annotators because judgments can differ. Sentiment and harmful-speech classification are examples where one person's decision may not match another's. Comparing annotations helps establish a label for the dataset. At the same time, foundation models consume far more data than people can label. The proposed compromise is to use automation for much of the dataset while reserving human review for the examples that matter most. The same human feedback loop also helps improve LLMs through reinforcement learning from human feedback.

### A naive prompt can produce labels, but its output needs supervision
[10:22](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=622s)
Michael Malyuk describes the simplest workflow: ask an LLM to classify each item and return a class. This can work for a basic task, but the label quality depends heavily on the prompt. The model may produce outputs that are close to the intended answer without matching the allowed labels. A human annotator therefore checks each prediction in a labeling tool. Once verified, the result can be treated as ground truth for the task. The ground truth set then provides a way to compare later predictions with reviewed labels. Malyuk says this process can be repeated to find where the prompt fails and to decide what needs to change.

### Ground truth examples let teams refine prompts from observed errors
[13:13](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=793s)
The next iteration compares new predictions against the reviewed dataset. The annotator looks at mistakes and uses them to adjust the prompt, then runs the process again. Malyuk describes this as useful but time consuming because several prompt iterations may be needed before the predictions match the ground truth set well. The demo automates this refinement. A system examines mismatches between predictions and reviewed labels, then proposes a better instruction. The dependency is a ground truth dataset for validation. Without reviewed examples, the system has no reliable basis for deciding whether a change improved the labeler.

### Constrained generation keeps model outputs inside the label set
[27:40](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=1660s)
In the live demo, a free-form GPT-3.5 output sometimes includes prefixes, extra text, or labels that cannot be used directly. Malyuk calls these outputs hallucinations in the context of the labeling task. Instead of fine-tuning immediately, he uses the guidance library to define a prompt template with an allowed choice between objective and subjective. The model is asked to select from those classes rather than write an unrestricted completion. This produces labels that are easier to consume and also exposes probabilities for the selected tokens. The approach avoids having to clean up arbitrary generated text after each prediction.

### Confidence scores support active learning and human review
[29:50](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=1790s)
The constrained labeler also returns a confidence measure from the model's token probabilities. Malyuk uses these scores to order examples by uncertainty. In an active learning workflow, annotators review the items where the model is least certain instead of labeling every example equally. He mentions log probabilities, entropy, and other uncertainty measures as possible ways to rank items. The goal is to focus human effort on cases that need judgment while leaving highly certain examples to automation. In Label Studio, the demo imports the predictions and scores, sorts the examples, and lets the annotator correct the uncertain cases.

### Prompt quality can improve through examples and explicit reasoning
[48:43](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=2923s)
Malyuk presents few-shot prompting as one way to improve accuracy. The prompt can include examples from the ground truth set, previous model errors, or semantically similar examples selected for the current item. He also describes chain-of-thought prompting, where the model is asked to produce a rationale before its final class. The rationale can help the model make a decision and give annotators more information to inspect. The workflow can even ask an annotator to correct the reasoning. These additions may improve the label, but they add more generated text and require care when the reasoning itself is wrong.

### Self-consistency trades extra generations for more stable labels
[51:53](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=3113s)
The speakers extend the workflow with self-consistency. Instead of asking for one answer, the system asks the model to reason in several ways and produce multiple outputs. Those outputs can then be combined, for example by majority vote. Malyuk presents this as a way to reduce disagreement and make labels less dependent on one reasoning path. The method has a direct cost because it generates more tokens, which matters when using a paid LLM API. It also does not remove the need for human review. The resulting labels still need to be checked against the task's definitions and the ground truth process.

### Automated instruction search can refine a labeler without rewriting prompts by hand
[53:50](https://www.youtube.com/watch?v=mTVNE0Sw5vI&t=3230s)
The final demo replaces manual prompt editing with an iterative instruction optimizer. It starts with an instruction set, evaluates each instruction against the ground truth labels, selects a strong candidate, and asks a model to adapt it using sampled errors. The revised instruction goes back into the set, and the cycle repeats. Malyuk compares this with optimization in natural-language instruction space rather than weight-based gradient descent. A more powerful model can generate revised instructions without handling the private data itself, since the adaptation step can focus on the instruction and error information. The method still depends on reliable reviewed labels and can produce instructions whose useful details are not obvious to the operator.

## Notable quotes
- Chris Hoge: "Data reviewed by human labelers remains the gold standard." (05:22)
- Nikolai Liubimov: "The final goal is actually consume the labels." (27:04)
- Nikolai Liubimov: "You don't actually need to label all the data, but you only focus your labeling efforts into the most uncertain examples provided by the model." (30:00)
- Michael Malyuk: "The dependency that you need to have and satisfy to get you there is a ground truth data set that is used for validation." (16:15)
- Nikolai Liubimov: "We can actually modify our instructions on the fly automatically without even human intervention." (58:02)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- HumanSignal
- Label Studio
- OpenAI
- GPT-3.5
- GPT-3.5 Turbo Instruct
- Llama 2
- Amazon product reviews
- RLHF
- guidance
- retrieval augmented generation
- few-shot prompting
- chain of thought
- self-consistency
- active learning

## Who should watch
- You are building a labeling pipeline and need a practical way to combine LLM predictions with human review.
- Your team has a large set of unlabeled text and wants to direct annotators toward uncertain or error-prone examples.
- You are evaluating prompt optimization and need to understand its dependence on ground truth data, model confidence, cost, and domain knowledge.

## Related talks

- [Designing Human in the Loop Experiences for LLMs](https://mlopstalks.com/talks/designing-human-in-the-loop-experiences-for-llms) (Alberto Rizzoli, V7, 11:40)
- [Incorporating LLMs in High-stake Use Cases](https://mlopstalks.com/talks/incorporating-llms-in-high-stake-use-cases) (Yada Pruksachatkun, Moonhub, 11:01)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo, 33:56)
- [Building and Curating Datasets for RLHF and LLM Fine-tuning](https://mlopstalks.com/talks/building-and-curating-datasets-for-rlhf-and-llm-fine-tuning) (Daniel Vila Suero, Argilla, 58:51)