# Data Labeling Best Practices

Charles Brecque, TextMine | AI in Production 2024 | 12:59

Source: https://www.youtube.com/watch?v=9hf_nFMiLCo
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/data-labeling-best-practices
Published: 2024-05-01
Tags: data-quality, fine-tuning, human-in-the-loop, prompt-engineering

## TL;DR
- Data labeling is useful when a foundation model lacks the business-specific knowledge needed for an accurate task.
- Labeling prompts must make sense to both the model and the people answering them, or the team can teach the model incorrect behavior.
- Feedback loops should capture when the model does not know an answer and correct misunderstandings in its outputs.

## Summary
Charles Brecque describes the data labeling process TextMine developed while fine-tuning models for extracting information from business documents. He explains when labeling is useful, especially for proprietary data, open-source model fine-tuning, and focused extraction or ETL tasks. Before hiring labelers, teams should define the task, decide whether it needs reasoning, split complex work into smaller tasks, design clear prompts, and create useful data chunks. Labelers need domain knowledge, consistency, and attention to detail. Brecque recommends trialing people before longer contracts, creating a communication channel, and signing confidentiality agreements. He also shows why feedback loops matter. Labelers should mark unknown answers instead of guessing, and should correct cases where a model confuses a country with a city or produces unnatural entity names. TextMine connects its live model with labeling so corrections can improve performance over time.

## Key ideas
### Fine-tuning needs labeled business data when general models lack task-specific knowledge
[02:36](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=156s)
Brecque says foundation models have been trained on vast amounts of data and can perform impressive general tasks, but they often lack the knowledge needed for a specific business use case. Data labeling becomes useful when a team is fine-tuning an open-source model for its own requirements. Proprietary data can also be valuable, especially for a software platform with many users and years of accumulated examples. He extends this beyond chat applications to extraction, ETL, and other focused tasks where fine-tuning can improve performance. The need for labeling comes from the gap between broad training data and the details required by a particular product.

### Teams should test whether a task needs reasoning before choosing an LLM workflow
[04:18](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=258s)
The first design question is what the system must solve. Brecque says teams should ask whether the task requires reasoning, because LLMs do not reason by default. If an LLM is still the right choice, a complex problem can be split into smaller tasks and handled through chained prompts or models. This decision affects the labeling setup because labelers need to answer well-defined questions rather than work against an unclear, oversized objective. The workflow should be designed around the actual task, with the model's limits considered before prompts, data chunks, or staffing are finalized.

### A labeling prompt must be clear to the labeler as well as useful to the model
[05:05](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=305s)
Prompt design has two audiences. The prompt must guide the LLM toward the intended output, but it also must be understandable to people who may be new to data labeling and unfamiliar with machine learning. Brecque says TextMine did not always get this right. If labelers misunderstand the instructions, their answers can teach the model the wrong behavior. He recommends iterating until the prompt works for both sides. The prompt should provide context, state what the system can do, and give clear instructions about what it must not do. The right wording is found through repeated testing rather than written perfectly on the first day.

### Data chunks should contain enough signal to answer the labeling question
[06:22](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=382s)
After defining the task and prompts, the team needs to decide whether it has suitable data and how to split it into chunks. The chunks must contain enough information for a labeler or model to answer the question. Brecque warns against producing many chunks where the answer is unavailable or not applicable, because those examples do not teach the model much. This is a practical data preparation issue: the unit sent for labeling needs to match the question being asked. Poor slicing can reduce the useful signal even when the underlying documents contain valuable information.

### Domain expertise is useful, but consistency and attention to detail determine labeling quality
[07:03](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=423s)
Brecque says TextMine found LinkedIn effective for sourcing labelers, including students and people with substantial professional experience. Domain expertise helps, but it does not guarantee that someone will be a good labeler. The people doing the work need to be consistent and attentive to detail. He recommends running a trial before entering a proper contract, then giving labelers a community or Slack channel where they can communicate and improve their practice. Because labelers may access sensitive documents, confidentiality agreements are needed. He also says this protection matters even when the data is not sensitive, since companies may not want their data shared with competitors.

### Labelers should explicitly mark unknown answers instead of making the model guess
[08:10](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=490s)
A labeling workflow needs a clear way to represent uncertainty. Brecque says labelers should indicate when the model does not know, rather than supply an answer that is not supported by the document. Otherwise, the fine-tuned model can learn to hallucinate answers when information is absent. He also describes a case where a model suggested United Kingdom as a city because it did not understand that United Kingdom is a country. Feedback from the labeling process can add that distinction. The same process can correct generated entity names, such as turning an awkward answer like 'law of England' into the more natural 'English law' when that is how a person would respond.

### The labeling workflow should follow the model's needs rather than an engineering convenience
[08:31](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=511s)
TextMine encountered a form with first-line and second-line address fields. The software team initially proposed splitting the prompts into separate questions for the two address lines. In practice, Brecque says this confused the model, which returned the same answer for both fields. The example shows why teams should not preserve a data or software structure automatically when it does not fit the model's interpretation of the task. Prompt and schema decisions need to be tested against actual model behavior. A clean engineering decomposition can still produce poor labels if it does not match how the information appears in the document.

### A live connection between the model and labeling can speed up corrections
[11:38](https://www.youtube.com/watch?v=9hf_nFMiLCo&t=698s)
Brecque briefly discusses the tooling TextMine uses for data labeling and says the interface works well for its labelers. His larger point is that the labeling system should connect with the live model where possible. When model outputs and human corrections can move through the same workflow, teams can identify errors and improve performance faster. The feedback is useful for correcting missing world knowledge, inconsistent answers, and misunderstandings of the question. He does not present one required tool stack. The workflow matters more than adopting a particular product, and the right setup depends on the task and the team's data.

## Notable quotes
- "The prompt needs to be aligned both for the data labelers and the LLM." (07:49)
- "If the model doesn't know then the model should say that it doesn't know." (08:10)
- "It's really important to hire or contract people who are consistent and really have an attention to detail." (09:29)
- "You need to think what makes sense for the model." (09:02)

## Tools & references mentioned
- TextMine
- Amber Akhtar
- knowledge graphs
- LinkedIn
- Slack

## Who should watch
- You are fine-tuning an open-source model on company or domain-specific documents and need to decide whether labeling is worth the effort.
- Your labeling team includes domain experts who may be new to machine learning, and prompt misunderstandings are creating inconsistent answers.
- You need a practical workflow for handling unknown answers, protecting sensitive documents, and feeding corrections back into a live model.

## Related talks

- [Labeled Datasets that Correct Themselves Automatically](https://mlopstalks.com/talks/labeled-datasets-that-correct-themselves-automatically) (Curtis Northcutt, Cleanlab, 1:06:03)
- [RLHF Data Collection in Practice](https://mlopstalks.com/talks/rlhf-data-collection-in-practice) (Andrew Mauboussin, Surge AI, 12:10)
- [Data Selection for Data-Centric AI: Data Quality Over Quantity](https://mlopstalks.com/talks/data-selection-for-data-centric-ai-data-quality-over-quantity) (Cody Coleman, 1:11:01)
- [Turn Data Chaos into AI Strategy with Programmatic AI Data Development](https://mlopstalks.com/talks/turn-data-chaos-into-ai-strategy-with-programmatic-ai-data-development) (Elena Boiarskaia, Snorkel AI, 27:14)
- [Automating Data Annotation with LLMs](https://mlopstalks.com/talks/automating-data-annotation-with-llms) (Nikolai Liubimov, Michael Malyuk & Chris Hoge, HumanSignal, 1:03:04)