Vikram Chatterji argues that as language models become commoditized, high-quality proprietary data becomes the main source of model performance and business differentiation.
2
LLM applications need better control of both fine-tuning data and prompts because hallucinations, incorrect labels, stale data, and prompt changes can reduce output quality.
3
Teams need tools and metrics that let them compare prompts, models, inputs, and outputs across many experiments instead of relying on manual data detective work.
Summary
Vikram Chatterji argues that the current LLM cycle resembles earlier waves of machine learning. AlexNet and Transformers brought major attention, models became easier to reuse, and teams then focused on adapting them with their own data. He expects the same pattern with LLMs. Open and closed models are spreading quickly, so the differentiator will be the data used for fine-tuning, retrieval, and prompts. Chatterji describes data work as the part of ML that takes most of a data scientist's time, including finding failure cases, correcting labels, checking regressions, and choosing what to train on next. For LLM applications, teams also need to test prompt structure, context, retrieval, and instructions. He recommends treating prompt evaluation as an engineering and research problem because prompt and model combinations grow quickly, while widely useful evaluation metrics are still limited. He is direct that production requires repeated testing and fine-tuning after deployment.
LLMs repeat an older cycle in which breakthroughs become reusable building blocks
Chatterji places LLMs in a longer ML history. AlexNet brought image classification into mainstream attention, and the Transformer paper later led to models such as BERT. Once BERT was open source, teams could build on it and create specialized versions. PayPal could start with BERT and fine-tune it on PayPal data, while financial teams could adapt models for stock-news sentiment. Chatterji says the same pattern is now appearing with LLMs. Models are becoming available as shared foundations, which lets companies concentrate on adapting them to their own use cases.
Proprietary, high-quality data becomes the main differentiator as models commoditize
Chatterji says data scientists already understood the shift toward data-centric AI. The important question is no longer simply how much data a team has. Teams need to know which data is useful, whether labels are correct, and how to create or obtain more data when their existing data is insufficient. Proprietary data can become a business advantage because other companies cannot easily reproduce it. He connects this view to Galileo, which covers data work before labeling, after labeling, during training, and during repeated iteration.
Manual data investigation is expensive even though it drives model quality
Chatterji describes ML debugging as data detective work. At Google AI, he recalls spending days with spreadsheets and Python scripts to find where models failed. He cites Greg Brockman's estimate that data scientists spend 80 to 90 percent of their time looking at data, fixing it, and iterating. Customers have described this work as both the most unpleasant part of the job and the part they cannot skip. Chatterji argues that more of this inspection should be automated so teams can find failure points and select the next data to use without repeating the same manual process.
LLM data quality matters because confident hallucinations are hard to trust
Chatterji says LLMs can produce answers with confidence even when the answers are wrong. He compares this problem to the spread of false information, where a user cannot tell what to believe from confidence alone. He points to the factual error in Google's Bard launch demonstration and warnings around other chat products. Better inputs and better prompts can reduce the problem, although they do not remove the need to evaluate outputs. For enterprise applications, teams need responses they can trust before putting a model into production.
Prompt quality depends on structure, context, retrieval, and instructions
Chatterji treats prompts as part of the data workflow. Prompt structure can change results, as shown by Chain-of-Thought prompting, where the model receives an explanation of how an answer was reached instead of only the final answer. Context and retrieval also matter. A vector database may return examples that are close in embedding space, but the team still needs to ask whether those examples are the right context. Instructions and examples need repeated adjustment. These choices make prompt engineering a real part of building an LLM application.
Fine-tuning requires checks for regressions and incorrect ground truth
Fine-tuning with labeled data can be expensive and time-consuming, but Chatterji says it is often needed for a specific use case. Teams must check whether fine-tuning improves the intended behavior while damaging other model performance. Incorrect ground truth remains a problem whether labels come from in-house workers, human review, or reinforcement learning with human feedback. Because a large pretrained model can respond differently after being tuned on a narrower dataset, teams may need repeated cycles before the model is suitable for finance, contact center, or another application.
Production models need fresh data and repeated evaluation
Chatterji says deployment is not the end of the work. Teams need to monitor models, decide when to fine-tune again, and identify which data should be used in the next cycle. His data-centric approach therefore includes more than labeled training data. It also includes prompts, retrieved context, and the outputs produced from those inputs. Model performance can change as the data and application change, so production teams need a continuing evaluation process.
Prompt and model comparisons need better metrics and software
When a team compares many prompts across several models, the number of combinations quickly reaches the hundreds or thousands. Chatterji calls this partly a software engineering problem and partly an ML research problem. Teams need to record which model, input, prompt, and response were used, then make decisions from those comparisons. He says practitioners often fall back on familiar metrics such as BLEU scores, but there is little agreement on metrics that evaluate prompts across use cases. His practical position is that prompting and fine-tuning are both needed today, depending on the application and the cost of an incorrect response.
"80 to 90 percent of what a data scientist does today is basically just staring at the data trying to figure things out, trying to fix it, and then iterate on it over and over again."15:07
Who should watch
You are building an LLM application and need to decide when to use prompting, retrieval, fine-tuning, or a combination of them.
Your team spends large amounts of time inspecting examples, correcting labels, or comparing model outputs by hand.
You need a production process for tracking prompt changes, model changes, regressions, and the data behind each result.
