Kenny Daniel argues that data quality defines model quality, yet researchers and engineers rarely examine large language model datasets closely.
2
Modern models can help filter and inspect huge datasets, but people still need an interface to guide the process and judge domain-specific errors.
3
Better data and evaluations remain useful as models improve, while infrastructure or model-specific work can become obsolete when new models arrive.
Summary
Kenny Daniel argues that data is still the least examined part of modern AI development. Companies discuss model architectures and hardware, while training data, filtering, formatting, and evaluation receive less attention. Large datasets make manual inspection difficult, but Daniel says models can help classify, score, and group data at scale. Human experts still need to guide that process through a usable interface, especially in fields such as medicine, finance, and law. He connects this idea to his work at Hyperparam and to Highpark, an open-source JavaScript Parquet parser that lets people browse remote datasets in a browser without downloading entire files. The conversation also covers Daniel's earlier company, Algorithmia, which focused on serving machine learning models over GPUs before the current AI boom. His view is that data improvements survive changes in model architecture, while many infrastructure and model-specific optimizations do not.
Data defines the differences between leading models
Kenny Daniel says data quality strongly affects language models and models in general, even though the topic receives little public discussion. He thinks the major labs avoid describing their data because it is part of their competitive advantage. The Llama and DeepSeek papers offer limited glimpses into data choices, including proportions of code, mathematics, English, and Chinese content. Daniel does not demand that companies reveal every source. He wants more detail about how they clean, filter, and distribute the data before training.
Large datasets have made basic data inspection difficult
Daniel says datasets on Hugging Face can reach terabyte scale, which makes it hard for a data scientist or ML engineer to become familiar with the contents. Many people respond by sending the data into a model and hoping it works. He points to Llama 3 as an example of model-based filtering. The team reportedly started with a much larger pile and trained on less than 10 percent of it, using a language model to label, classify, and quality-filter the available data.
Model-assisted inspection is becoming affordable at dataset scale
Daniel describes using models to rate quality, classify examples, and test what the model can or cannot predict. He says processing a trillion input tokens with GPT-4 had cost about $5 million, while a year later a cheaper model in batch mode could process the same scale for about $50,000. He presents this as an increasingly accessible way to inspect very large datasets. The point is not to let a model make every decision. It is to use model inference to find useful regions and examples for further examination.
Looking at examples in the context of model errors reveals data problems
The discussion turns to PDFs whose tables are extracted without the surrounding references, leaving text such as 'see table 3.2' disconnected from the actual table. Daniel says these problems become visible only when engineers inspect the data and its relationship to model behavior. His old workflow was to build a model, examine correctly and incorrectly classified examples, and group errors into buckets. He says this process has consistently produced insights that led to better models. Sorting data by a metric and examining extreme values can also expose useful patterns.
Fine-tuning remains useful when data expresses behavior better than prompts
Daniel agrees that prompting or retrieval can solve some problems without fine-tuning, and he would not recommend starting with fine-tuning when those approaches work. He still sees fine-tuning as useful for style control and specialized behavior. A company chatbot may need a particular tone or a consistent way to handle situations. Some of that can go in a prompt, but style can be easier to express through curated examples. The data then captures what the user wants the model to do.
Interfaces can bring domain experts into the model-building process
Daniel compares this possibility with Tableau, which let managers and executives explore data directly instead of delegating every question to a data scientist. He wants similarly accessible interfaces for model and dataset work. A radiologist may not understand the mathematics of training, but can recognize a misclassified X-ray and explain why it is wrong. That expertise can guide corrections. Daniel says the division between model builders and domain experts is partly false because better interfaces let the expert participate directly instead of handing work over a wall.
Human guidance and model automation need each other
Daniel says a purely model-driven dataset improvement service would miss the human who understands the intended use of the model. A user interface keeps that person involved and lets them express intent through data or evaluations. At the same time, no person can inspect a billion examples during a lifetime, so models are needed to examine the dataset at scale. He describes the useful system as one where the model finds patterns and error groups while the human decides what matters and what should change.
Data and evaluations remain valuable when models and infrastructure change
Daniel calls data and evaluations bottlenecks in model development. He says an evaluation is often a clean dataset that expresses what the model should do, even though the relationship has some nuance. If a model fails on a problem and the failure can be turned into an evaluation, he expects future models to improve on it. He also argues that better data keeps paying off as teams move from one model generation to the next. Work on GPUs, architectures, or model-specific tricks can be discarded when technology changes, while proprietary business data and domain context remain necessary.
Browser-native dataset tools can remove the download bottleneck
Daniel explains that Parquet is column-oriented and stores an index in its footer, which lets systems such as DuckDB query remote files without downloading them in full. He built Highpark, a JavaScript Parquet parser, after finding existing browser libraries abandoned or unsuitable. It uses HTTP range requests to fetch only the rows needed for browsing a dataset. Daniel says the compressed library is under 10 kilobytes and can display Hugging Face datasets in the browser. Hugging Face later provided an open-source grant to support its development.
"You still need access to this data, and the cleaner and better formatted and structured and junk removed it is, that's going to be better whether you're a person or a model looking at it."Kenny Daniel41:27
Who should watch
You are building or curating large language model datasets and need a practical reason to inspect examples instead of sending everything straight to training.
Your model works on specialist data, and domain experts need a way to find errors and guide corrections without becoming ML engineers.
You are evaluating whether to build data tooling, model infrastructure, or an AI application that should remain useful as models improve.
