Machine learning systems need evaluation across metrics, datasets, and slices throughout their lifecycle, rather than a single score from one test set.
2
An evaluation store could keep evaluation definitions consistent across testing, deployment, monitoring, data collection, labeling, and retraining.
3
Josh Tobin expects machine learning infrastructure to remain modular, with tools fitting into existing systems instead of replacing the whole stack.
Summary
Josh Tobin explains why production machine learning is harder than training a model and reporting an offline score. Models face shifting data, long-tail cases, delayed labels, and autonomous decisions, so teams need to understand performance at every stage of the model lifecycle. He proposes an evaluation store that centrally defines datasets, metrics, and slices, then makes those definitions available for offline evaluation, live testing, monitoring, data collection, labeling, and retraining decisions. The system would connect offline and online evaluation, since offline results provide context for interpreting production behavior. Tobin expects people to remain involved in these decisions for a long time, although better tools can reduce the work. He also argues for modular infrastructure. An evaluation store should fit with training pipelines, deployment systems, monitoring tools, and dataset tools. Its broader purpose is to give scientists, engineers, platform teams, product managers, and business stakeholders a shared view of model quality.
Production machine learning is a data product with a tight feedback loop
Josh Tobin distinguishes research and analytics-oriented machine learning from production machine learning. A production system puts predictions inside a product or service and lets them influence decisions autonomously or semi-autonomously. These systems often need real-time predictions, operate on changing distributions with long tails, and cannot rely on the independent and identically distributed assumptions common in academic machine learning. They also may not receive quick feedback from the people affected by their predictions, so the system needs to operate on its own and alert people when performance may be changing.
A single score from running a model on a large dataset does not describe the contours of its performance. Tobin recommends tracking several metrics, evaluating them across an overall evaluation set and smaller datasets for known edge cases, and slicing results by categories such as user groups where bias is a concern. This approach can expose long-tail failures, regressions from earlier model versions, and differences between groups. He is clear that testing does not solve bias and fairness, but broader testing can reduce the risk and impact of failures. He also says that the right metrics, slices, datasets, and significance tests remain open technical questions.
Evaluation should guide every decision in the model lifecycle
Tobin sees evaluation as the information needed to move a model through training, testing, deployment, monitoring, data collection, labeling, and retraining. Before promotion, teams need confidence that performance is better across the metrics, slices, and datasets they care about. Before full production traffic, they can use A/B tests and shadow mode for live evaluation. Monitoring asks how the model should perform on new data and whether degradation is occurring. Data collection and labeling should focus on cases where performance is low or uncertain. Retraining is ultimately an ROI decision, which requires an estimate of how much production performance has degraded.
An evaluation store would keep definitions consistent across tools
The proposed evaluation store would define datasets, metrics, and slices in a central place and apply them throughout the machine learning ecosystem. This avoids a situation where a product manager helps choose a metric for testing, then the monitoring system computes a different version because the definition was not updated. Tobin also sees the store as a place to execute these computations. Offline and online evaluation would share definitions and context, which could improve how teams estimate performance when production labels are delayed, expensive, or noisy.
Monitoring alone does not cover the full evaluation problem
Tobin says an evaluation store overlaps with machine learning monitoring, but differs in scope. Monitoring is often pictured as a one-way view into the current health of an online system. An evaluation store would connect online monitoring to offline evaluation for consistency and better interpretation. It would also feed information back into model development, data gathering, labeling, and promotion decisions. Evaluation therefore sits across the lifecycle instead of appearing only after deployment. Machine learning engineers may use the system most, but other stakeholders need visibility into how the model is judged and how it is performing.
The infrastructure should be modular rather than end-to-end
Tobin does not expect most companies to adopt one complete machine learning platform. Companies have different data, products, and integration needs, much as software teams use different combinations of tools. He wants the evaluation system to fit existing training pipelines, deployment systems, monitoring systems, and dataset exploration tools. It may replace parts of those systems, but it should remain a developer tool that connects to the ecosystem. He compares this approach with software infrastructure, where end-to-end platforms are uncommon.
Shared evaluation can reduce the wall between model builders and production teams
Tobin describes a common organizational problem: machine learning scientists build models, then hand them to another team that may not understand the model well enough to productionize it. The handoff can take months, cause communication problems, and introduce bugs when model definitions are translated into production. A shared evaluation system could give scientists, machine learning engineers, platform teams, product managers, and business stakeholders a common view of what the system is and how it is judged. He compares this possible effect with software observability tools, which made system health more accessible beyond a separate operations group.
Tesla's data engine uses detectors to find valuable edge cases
Discussing work described by Andrej Karpathy, Tobin explains how a team can improve a model by finding where it performs poorly and collecting more data for those cases. An engineer might identify a failure pattern such as a left-hand turn in the rain around sunset. Instead of sending drivers out manually to find every example, developers can write monitoring hooks, which are smaller, noisy machine learning models that detect likely instances of the edge case. When a hook fires on a vehicle, the image can be uploaded for labeling. The resulting dataset will contain false positives, but it is more likely to include the kind of scene the team wants to improve.
"The core decision that humans need to make should be informed by the question of, okay, do I have a good understanding of how well my model is performing at this stage so that I can continue to move it along?"Josh Tobin23:30
Who should watch
You are building models that work in a controlled evaluation but fail on edge cases or shifting production data.
Your team uses separate systems for testing, monitoring, data collection, and retraining, and model definitions can drift between them.
You are deciding whether an evaluation product should be a standalone system or a modular part of an existing machine learning stack.
