# Evaluating Quality and Improving LLM Products at Scale

Austin Bell, Slack | AI in Production 2024 | 15:11

Source: https://www.youtube.com/watch?v=F5t2XJaSpVY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale
Published: 2024-05-23
Tags: evals, structured-outputs, testing

## TL;DR
- LLM products need automated quality metrics because prompt, preprocessing, and model changes affect outputs in ways that a small hand-checked sample cannot capture.
- Evaluation should break broad concerns such as hallucination into smaller product-specific checks, including context use, citation accuracy, relevance, coherence, and Slack identifier formatting.
- A staged process using a golden set, a larger validation set, and A/B tests lets teams prototype quickly, reject weak changes, and deploy improvements with more confidence.

## Summary
Austin Bell explains how Slack evaluates generative AI products such as channel and thread summarization and natural-language search. The central problem is that each user interaction is different, while judgments about quality can be subjective. A short summary and a comprehensive summary may both be appropriate for different users. Slack therefore breaks quality into smaller checks, including factual accuracy, coherence, relevance, hallucination types, citation correctness, and formatting of Slack user, channel, and message IDs. Bell describes three evaluation stages: a small golden set for rapid prototyping, a larger validation set of roughly 100 to 500 samples, and A/B testing with quality metrics. He gives extractive summarization as an example of a preprocessing change that reduced quality while improving formatting, producing a more useful overall product tradeoff. The talk is practical and honest about the need to measure the whole product rather than rely on model benchmarks.

## Key ideas
### Prompt and model changes need measured impact
[01:56](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=116s)
Teams improve generative products through prompt engineering, preprocessing, postprocessing, and different machine learning models. Bell says each change needs an evaluation of its effect on the output. A small collection of examples is not enough because user experiences are random and unique. Without a way to compare changes, teams cannot tell whether a product is improving or safely move changes forward at scale.

### Quality depends on the user and the product context
[03:16](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=196s)
Bell uses summarization to show why quality is partly subjective. One user may want a short summary with a few bullet points, while another may want a comprehensive summary that removes the need to read the underlying messages. Both outputs can be good. Evaluation therefore needs to account for the context and the needs of the particular product, alongside objective properties such as accuracy, coherence, grammatical quality, and relevance.

### Broad quality concerns should become smaller checks
[05:20](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=320s)
Slack breaks large evaluation concepts into more tractable problems. Hallucination is one example, but the product also needs correctly formatted user IDs, channel IDs, and message IDs so generated outputs can connect to the rest of Slack. These smaller checks can run on each summary or question-and-answer response and produce either separate scores or a composite quality score.

### Hallucination evaluation works better when it is specific
[06:40](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=400s)
Rather than treating hallucination as one broad category, Bell describes checks that match the product's needs. An extrinsic hallucination occurs when an LLM generates text outside the supplied context. A question-answering system can also give the right answer with an incorrect citation, such as pointing to the wrong message. Large LLM evaluators can assess these cases, while natural language inference models can support evaluation at higher scale.

### Evaluation happens at three stages
[08:35](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=515s)
Slack uses a small golden set of messages and outputs for fast prototyping. It then uses a larger, more representative validation set, typically around 100 to 500 samples, where automated metrics replace direct inspection of the underlying data. The final stage uses A/B testing to determine whether a change improves the product in use. Each stage acts as a gate before the next one.

### Stage gates let teams fail fast
[09:09](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=549s)
The staged process lets teams try multiple approaches quickly and continue only with changes that meet their quality metrics. Bell connects this to a standard development cycle for generative products. The purpose is to avoid pushing changes forward based on intuition or a few examples, while keeping evaluation practical enough to run repeatedly.

### Preprocessing can improve one property while hurting another
[09:50](https://www.youtube.com/watch?v=F5t2XJaSpVY&t=590s)
Bell describes introducing extractive summarization as preprocessing when the context was too large for the LLM summarization flow. Automated metrics showed a significant reduction in quality alongside significant improvements in formatting capabilities. The example shows why teams need several quality measures and an overall product judgment rather than a single metric.

## Notable quotes
- "You can't just develop 10 examples, evaluate on those, and say you're kind of good to go." (02:35)
- "The evaluation should largely be driven by context and the user needs for that particular product." (04:38)
- "The goal here is to take these really specific and small problems and start to develop a set of automated quality metrics." (06:15)
- "By having this stage gate and prototyping capability, it allows us to fail fast and prototype a variety of different approaches very, very quickly." (09:29)

## Tools & references mentioned
- Slack
- extractive summarization
- natural language inference

## Who should watch
- You are changing prompts, preprocessing steps, or models and need evidence that the resulting product is actually better.
- Your team evaluates LLM outputs with a small hand-built test set and wants a staged path toward larger automated validation.
- You are building summarization or question-answering features where correctness, citations, formatting, and user preferences all affect quality.

## Related talks

- [Evaluation](https://mlopstalks.com/talks/evaluation-panel-1-large-language-models-in-production-conference-part-2) (Abi Aryan, Independent Consultant & Amrutha Gujjar, Structured & Josh Tobin, Gantry & Sohini Roy, NVIDIA, 38:19)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)
- [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50)
- [How Product Metrics Become LLM Evaluations](https://mlopstalks.com/talks/how-product-metrics-become-llm-evaluations) (Raza Habib, Humanloop, 53:07)
- [Systematically Test and Evaluate Your LLMs Apps](https://mlopstalks.com/talks/systematically-test-and-evaluate-your-llms-apps) (Gideon Mendels, Comet, 1:01:43)