# Solving the Last Mile Problem of Foundation Models with Data-Centric AI

Alex Ratner, Snorkel AI and University of Washington | LLMs in Production 2023 | 36:47

Source: https://www.youtube.com/watch?v=-oDgV6q6KtI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/solving-the-last-mile-problem-of-foundation-models-with-data-centric-ai
Published: 2023-04-24
Tags: data-quality, distillation, fine-tuning, synthetic-data

## TL;DR
- Foundation models provide useful starting points, but production applications still need application-specific data development.
- Private data and domain knowledge are more durable sources of value than access to closed model APIs.
- Teams can improve foundation models through guided error analysis, programmatic labeling, fine-tuning, and distillation into smaller task-specific models.

## Summary
Alex Ratner argues that foundation models are the base layer of AI applications, rather than complete products. Their broad capabilities work well for generic tasks and human-in-the-loop applications, but many production systems require high accuracy on private, specialized data. The difficult work is the last mile: finding the model's failure modes, labeling corrective examples, and adapting the system to a particular task. Ratner places this work within data-centric AI, where teams spend more effort developing, curating, slicing, sampling, and labeling data than changing model architectures. He presents Snorkel Flow as a way to apply a foundation model to task data, inspect errors, combine heuristics, knowledge bases, prompts, embeddings, and manual labels, then use the resulting data for fine-tuning or distillation. His practical point is that the right development method depends on failure tolerance. A copilot may work with imperfect output, while many enterprise systems cannot ship until they reach much higher accuracy.

## Key ideas
### Foundation models apply beyond language and generation
[05:11](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=311s)
Ratner uses the term foundation model because the underlying approach applies beyond language models. Data with graph structure can support self-supervision or autoregressive methods, including images, multimodal data, databases, and genomics. He also argues that predictive and discriminative applications can use foundation models, even though current discussion often treats generative AI as synonymous with large language models. The foundation metaphor captures the central limitation: a foundation is only a base, and each organization still has to build the application-specific system on top of it.

### Enterprise value will concentrate in private data and the last mile
[07:32](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=452s)
Ratner predicts that much enterprise usage will look more like a customized model than repeated calls to a general closed API. He points to open model work such as Alpaca, Databricks' Dolly, and Berkeley's Koala as evidence that inexpensive API-generated data and careful curation can reproduce much of a general model's behavior. Private data and domain knowledge therefore become more durable sources of differentiation. Even domain-specific models such as BloombergGPT still require further work before they reach the accuracy needed for production applications.

### Accuracy requirements divide easy demonstrations from hard production work
[15:10](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=910s)
Ratner describes a two-dimensional view of applications. One dimension is how different the task data is from the data used to train the foundation model. The other is how much accuracy the application needs. Marketing copy or a copilot can tolerate failures because a person edits the result. A bank, hospital, government agency, or other production system may need accuracy in the 90, 95, or 99 percent range before deployment. The difficult and valuable work sits where data is unusual and failure tolerance is low.

### Foundation models make development more data-centric
[21:00](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=1260s)
Data-centric development shifts the workflow away from treating data as a fixed input and model design as the main activity. Teams instead iterate on labeling, sampling, curation, and augmentation while the model stays relatively standardized. Ratner says foundation models make this shift more pronounced because users generally cannot change the architecture or manually tune vast numbers of parameters. When a model fails on a patient group, satellite-image subset, or legal-document category, the practical interface for correcting it is the data, the prompt, or both.

### Human supervision drove much of the improvement from GPT-3 to GPT-3.5
[22:57](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=1377s)
Ratner uses the GPT-3 to GPT-3.5 transition as an example of data-centric improvement. He separates the human inputs from the reinforcement-learning mechanism used to update the model. Those inputs included rankings, thumbs-up and thumbs-down feedback, and generated responses. In his framing, a large part of the improvement came from added supervision rather than a new model architecture. This supports his broader claim that labeled and curated data can produce major gains on top of a foundation model.

### Programmatic labeling combines many imperfect sources of supervision
[24:04](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=1444s)
Manual annotation is expensive, slow, and difficult to repeat when a task changes, especially when labels require clinicians, lawyers, network technicians, or underwriters. Ratner describes programmatic labeling, also called weak supervision, as a way to combine different sources of information. These can include heuristics, knowledge bases, embedding clusters, manual labels, and prompts. The goal is not to find one perfect prompt or rule. Multiple imperfect signals can be combined and modeled to create training data more efficiently.

### The Snorkel Flow loop starts with errors in the deployed task
[31:07](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=1867s)
Ratner outlines a workflow that begins with any base foundation model, whether closed or open source. The model is applied to the organization's task data, which allows the team to inspect its behavior through guided error analysis. Developers find recurring error modes, create corrective labels with programmatic or manual methods, and repeat the data-centric loop. They can export the resulting data, fine-tune the foundation model, or distill the behavior into a smaller model specialized for the task.

### Distillation can turn a broad foundation into a much smaller production model
[33:44](https://www.youtube.com/watch?v=-oDgV6q6KtI&t=2024s)
Ratner describes a contract-classification case study in which an ensemble of foundation models was followed by data development and distillation. The resulting system had a 41 percentage-point accuracy improvement over the GPT-3 baseline. The final specialized foundation model was 1,400 times smaller. His point is that organizations can begin with a broad model for capability, then use task-specific data to produce a smaller model that is more practical for production.

## Notable quotes
- "Foundation models are foundations and you still need to build the specific house or building for your specific setting and needs on top." (06:28)
- "The vast majority of the effort goes, and the vast majority of the value and differentiation is going to be captured." (08:57)
- "You have to go to the data, whether that's labeling, prompting, or various kinds of data-centric interfaces." (22:37)
- "The model is now fairly standardized and fixed and may not even change, or maybe automatically configured in your process, and most of your data science process is really about iterating on the data." (21:38)
- "You actually now distill this into a smaller model that is specialized for this task." (33:32)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Snorkel AI
- University of Washington
- Stanford
- Christopher Ré
- GPT-3
- GPT-3.5
- GPT-4
- Alpaca
- Llama
- Databricks
- Dolly
- Dolly v2
- Koala
- Berkeley
- BloombergGPT
- Snorkel Flow
- BERT
- CLIP
- RLHF
- weak supervision
- FlashAttention
- GPT-3.5
- GPT-4

## Who should watch
- You are deciding whether a general foundation model is accurate enough for a specialized production task.
- Your team has domain experts who can label examples, but manual annotation is too slow or expensive to repeat.
- You want a practical workflow for finding model errors and turning broad model behavior into a smaller task-specific system.

## Related talks

- [Foundational Models are the Future but...](https://mlopstalks.com/talks/foundational-models-are-the-future-but) (Alex Ratner, Snorkel AI, 52:42)
- [Foundation Models in the Modern Data Stack](https://mlopstalks.com/talks/foundation-models-in-the-modern-data-stack) (Ines Chami, Numbers Station, 13:17)
- [Data Selection for Data-Centric AI: Data Quality Over Quantity](https://mlopstalks.com/talks/data-selection-for-data-centric-ai-data-quality-over-quantity) (Cody Coleman, 1:11:01)
- [Managing Data for Effective GenAI Application](https://mlopstalks.com/talks/managing-data-for-effective-genai-application) (Anu Arora & Anass Bensrhir, QuantumBlack AI by McKinsey, 51:01)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon, 11:11)