Aggregate evaluation metrics hide the input conditions and failure cases that matter when choosing or improving an AI model.
2
Zeno lets teams inspect inputs and outputs, filter data by metadata, quantify slices of model behavior, and combine the results into shareable reports.
3
Alex argues that evaluation should move from discovering patterns in data to encoding those patterns as repeatable tests.
Summary
Alex Cabrera argues that AI evaluation results lose much of their value when they are reduced to one score. A word error rate for an audio transcription model, for example, does not show whether the model fails for particular languages, accents, noise levels, or speaker backgrounds. Zeno is an open source platform designed to help people inspect the data behind an evaluation, compare model outputs, create slices, build interactive charts, and publish reports grounded in those analyses. Alex demonstrates this with audio transcription, machine translation, image generation, and sensor data. The platform supports qualitative inspection as well as reproducible quantitative comparisons. Alex also describes open problems, including automatic error discovery and turning discovered failure patterns into regression tests. He prefers using a small amount of labelled data, or labels generated by an LLM, over label-free evaluation when possible.
A single score hides the conditions where a model fails
Alex groups current evaluation into benchmarks, human feedback, and model-based evaluation, then explains why their results are often reduced to one aggregate number. For an audio transcription system, average word error rate does not show whether performance changes across languages, background noise, speaking volume, or speaker demographics. A product team needs those details to decide whether a model fits its users. Alex compares this with grading a student: a teacher can inspect the individual exam answers, while developers often see only the final score from a model evaluation.
Zeno starts with direct inspection of data and outputs
Zeno lets users open individual evaluation instances and inspect the inputs, outputs, and metadata together. In Alex's audio transcription example, people repeat the same phrase while differing in where they are from, their age, and when they learned English. Users can listen to the audio, compare Whisper model outputs, and filter the data interactively. The display makes it possible to see that word error rates are higher for some speaker groups and to combine filters, such as North American speakers who learned English later in life.
The platform supports cross-filtering so users can examine combinations of conditions rather than isolated categories. Alex shows a slice for North American speakers who learned English later in life, where the word error rate is 34, higher than the average in that example. Zeno uses the same filtering idea across different data types. For image generation, users can inspect two million prompt-image pairs and compare the not-safe-for-work level associated with prompts containing words such as 'girl' and 'boy'. The point is to find patterns that a total score would conceal.
Evaluation needs reproducible slices and visual comparisons
Alex says that one-off observations are not enough. A team should define slices that can be reused to compare models and track behavior. In a translation benchmark, Zeno can create a slice for French, or a more specific intersection such as short French inputs, and compare systems on that subset. Its chart tools turn those slices into bar charts, radar charts, and other interactive views. Users can change the selected models, metrics, and subsets without rebuilding the analysis by hand.
Reports connect raw evaluation data to an explanation
Zeno reports combine written context with charts that remain connected to the underlying project data. Alex shows a translation report that moves from overall performance to the reasons some language models perform worse than state-of-the-art translation systems. Repeated outputs and hallucinations can be isolated as a slice, then examined across models and languages. Alex wants a new model upload to update existing reports dynamically, so the same analysis can be reused when the systems under test change.
The next step is turning discovered errors into tests
Alex identifies error discovery as an open problem because a dataset can be sliced in many possible ways. A useful platform could find areas with unusually high error or large differences between models. After a team finds a pattern it cares about, the next step is to encode that pattern as a specific test. Alex connects this to regression testing and continuous integration for models, where a discovered weakness could become a check that runs whenever the model changes.
Small labelled sets often beat label-free evaluation
In response to a question about label-free evaluation, Alex says Zeno can use model-graded metrics without labels, but his experience favors having labels when possible. A small labelled dataset can provide a stronger signal than a completely label-free approach. Teams can also use an LLM to generate labels and then evaluate against that smaller labelled set. Zeno does not require labels, so users can still analyse whatever form of metric or output they have.
"I think what we found across a bunch of different use cases is it's better spend either some time with small data or even just use an LLM to generate the labels and then do evaluation."19:25
Who should watch
You are evaluating an LLM, transcription model, translation system, or image model and your current report is mostly one aggregate score.
You need to compare models on particular user groups, languages, input types, or other slices of an evaluation dataset.
You want evaluation findings to become shareable reports and later become repeatable regression tests.
