Yash Sheth argues that teams can fix many model problems by finding gaps and errors in their training data instead of changing model architectures.
2
Unstructured data needs domain-specific analysis because sources, modalities, accents, noise, annotations, and semantic changes are difficult to capture with simple distribution metrics.
3
Galileo helps teams find problematic data cohorts after training and supports active-learning workflows that keep production models relevant.
Summary
Yash Sheth explains why data work is often the hardest part of building machine learning systems with unstructured data. From his experience leading speech recognition work at Google, he learned that teams need to understand their datasets and fix model blind spots before production. He describes problems across the full workflow: curating data before training, finding sampling and annotation errors after training, and tracking freshness and semantic drift after launch. Small teams often lack domain expertise and forget models after deployment, while larger teams rely on inconsistent scripts and metrics. Sheth argues that modern model architectures make data-centric work more valuable because many problems can be corrected in the dataset. He also discusses synthetic data, warning that it can make models learn artifacts from the generation process. Galileo is intended to surface useful data cohorts, provide actionable fixes, and support active-learning refresh cycles.
Speech recognition taught Sheth to fix data before production
Yash Sheth says his Google speech recognition team learned that world-class models depended on understanding datasets and proactively fixing gaps before launch. Their systems supported more than 20 Google products across 80 languages. That experience led him toward tools for understanding data, especially because modern machine learning teams lacked effective ways to inspect unstructured datasets. He connects this work to his earlier interest in using mathematics inside algorithms to learn features from data.
Unstructured data creates problems at every stage of the ML workflow
Sheth says more than 90 percent of the world's information is unstructured and that it is growing quickly. The difficulty starts with curation. Speech data, for example, varies by speaker, accent, source, modality, and environment. After training, teams need to find where models fail, yet many still use ad hoc scripts and inspect data dumps one sample at a time. After deployment, changing content creates another problem. Teams need to refresh datasets as the world changes, since semantic drift in unstructured data is difficult to predict.
Many model errors can be addressed by improving the dataset
Sheth says deep learning models are increasingly packaged as reusable libraries, so teams spend less time changing model architecture. When a model mispredicts, the first place to look can be the data. He describes common issues such as missing samples, insufficient noise variation, human annotation errors, and stale test sets. He compares this to garbage in, garbage out. A model can be treated as a black box while the team investigates the inputs that cause failures.
Small and large teams need different forms of data support
Small teams may have only two or three people carrying responsibility for AI. They often lack deep domain experience, focus on one model at a time, and move on after launch without maintaining what they built. Sheth wants tooling to act like a data scientist assistant by giving proactive and actionable suggestions. Large teams have a different issue. Individual engineers often run separate Python or SQL scripts, use their own metrics, and produce insights without shared review. Sheth says this creates inconsistent analysis across the organization.
Unstructured data needs domain-specific metrics before it can be standardized
Sheth agrees that the same ideas can help structured data, but says the need is more urgent for unstructured data. A data scientist can often calculate basic distribution insights for structured features. For audio, text, and other unstructured inputs, it is less obvious which metrics to track, and the right analysis depends on the domain. He sees a path toward unified tooling as structured-data teams increasingly create embeddings and use deep learning models, since the model can provide a common point for analysis.
Applied ML engineers can use deep learning without knowing calculus
Sheth distinguishes between researchers who develop new model methods and applied ML engineers who use existing models to build applications. He believes applied engineers do not need to understand the mathematics underneath every model, although researchers working on newer areas may need that knowledge. His analogy is software development: programmers can build useful software without understanding how the compiler works. He expects machine learning tools to become more abstract and general-purpose in a similar way.
Synthetic data can teach models the artifacts of its generator
Sheth warns that synthetic data can hurt performance when the model learns the heuristics used to generate it. In speech, text-to-speech output can bias a model toward the generation algorithm. In document parsing, synthetic W-2s and driver's licenses may differ from real documents in subtle ways. He says synthetic data should be mixed carefully with real data. The right augmentation also depends on the domain. Image data is easier to vary, speech can take added noise, and text is harder to augment safely.
Galileo connects data findings to model refresh workflows
Sheth says Galileo is used by small startups and Fortune 500 companies in areas such as conversational AI, contact centers, financial services, and healthcare. Users first adopt it to find errors after training, where Sheth says teams spend much of their time. More advanced teams connect it to active-learning pipelines so the system can identify new samples that should be labeled and added to the dataset. Galileo can also give a proactive signal when a production model needs a refresh instead of requiring constant manual monitoring.
Production failures often come from a changing world
Sheth describes speech recognition examples involving names and facts that change over time. A query about Barack Obama was useful for demos, but other names and current events created harder cases. The changing US president and annual Super Bowl winner require models to keep up with new entities. COVID-19 also forced teams to update models for new terminology while users and data teams were adjusting to lockdowns. Sheth's point is that proactive data signals could expose these problems before they become production failures.
"One of the key things that the team got right was the ability to effectively understand the data sets and proactively fix gaps in the model before launching anything in production."Yash Sheth05:13
Who should watch
You own an ML model built from audio, text, images, or other unstructured data and need a practical way to find bad or missing examples.
Your team spends time inspecting data dumps and writing one-off analyses after every training run.
You are considering synthetic data or production monitoring and want to hear where Sheth says those approaches can fail.
