# Making AI Reliable is the Greatest Challenge of the 2020s

Alon Bochman, RagMetrics | MLOps Podcast | Episode 312 | 1:01:38
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=d4PGxNM3Iis
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/making-ai-reliable-is-the-greatest-challenge-of-the-2020s
Published: 2025-05-06
Tags: evals, human-in-the-loop, synthetic-data

## TL;DR
- AI teams should test models, prompts, and agent configurations against their own data instead of relying on advice from vendors, labs, influencers, or papers.
- A useful eval starts with a small set of examples, grows from user failures, and must be run across the full set so fixes for one case do not break others.
- LLM judges become useful when their decisions are compared with human or domain-expert decisions and optimized for agreement with the people who matter for the task.

## Summary
Alon Bochman argues that AI teams have to discover what works through experiments on their own tasks. Vendors, research labs, influencers, and paper authors cannot know which model, prompt, retrieval setup, or agent configuration will work best for a particular application. He recommends starting with a small eval set, then expanding it as users expose new cases. The set also needs maintenance because user needs, laws, facts, and products change. Alon explains why domain experts should participate through a feedback loop rather than face a blank spreadsheet or a coding task. An LLM judge can evaluate most cases, while experts review its decisions and help improve it. Its quality should be measured by agreement with relevant humans. His FactSet example shows how disagreement between experts can reveal unclear business knowledge and improve both the knowledge base and the copilot.

## Key ideas
### Teams should discover their own best AI configuration through data
[01:46](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=106s)
Alon Bochman says engineers often follow advice from major labs, vector database vendors, or charismatic influencers because the field is new. That advice can help, but nobody knows what will work for a particular task, including OpenAI, Anthropic, or Weaviate. Teams should try different models, prompts, and agent configurations, then follow what their own data shows. He recommends defining the desired outcome for each step and writing a small evaluation set with examples of acceptable and unacceptable output. Around 30 or 40 examples can be enough to create an initial benchmark in a couple of hours. The benchmark lets a team compare choices on its actual task.

### An eval should expand from a happy path as users reveal failures
[07:05](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=425s)
Alon compares AI development with ordinary software development. A team may begin with a demo and one happy path, then add 10 or 20 examples before letting users in. Real users will put unexpected inputs into the system, exposing cases the builders never imagined. Fixing one edge case can break earlier cases because a prompt change affects the whole system. Alon recommends running every evaluation across all examples, rather than testing only the case currently being fixed. The right number of evals depends on the application's stakes and breadth. Teams should cover the paths most likely to fail and the cases that matter most to users, rather than chase a universal coverage target.

### Eval sets need both coverage and regular cleanup
[12:40](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=760s)
Alon says evals can become redundant, just like automated tests for conventional software. A large count does not create confidence if the examples all exercise the same small group of paths. Evals also encode an application's current view of the world, so outdated examples can become harmful. Facts, laws, users, and product behavior change at different speeds. A news application needs faster cleanup than a constitutional-law application, for example. Demetrios Brinkmann describes this as needing a garbage collector for the eval set. Alon agrees that teams need an update process whose frequency matches how quickly the relevant knowledge changes.

### Synthetic evals are useful early, while user examples matter more later
[17:15](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=1035s)
Alon distinguishes synthetic examples generated from scratch from examples derived from real user data. Early in a project, when the team has a happy path or a small prototype, synthetic evals can help the team move quickly and grow its test surface. They do not need to be perfect at that stage. As more users interact with the system, real inputs become more valuable because they contain the edge cases that synthetic generation may miss. AI can still help process a large log of user interactions. It can group semantic themes, reduce them to distinct examples, and identify areas where the existing eval set is thin. Alon calls the model a sparring partner that can create harder variations around observed failures.

### Domain experts need an interactive feedback loop instead of a blank page
[21:00](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=1260s)
Alon says many teams either generate synthetic data without domain expertise or ask experts to create an eval from a blank spreadsheet. Both approaches make participation difficult. Experts may reject synthetic examples because they look uninformed, while the word 'eval' and an empty set of columns give them no easy starting point. His proposed loop places LLM-judge scores beside user and domain-expert feedback. An expert can approve, reject, or explain a decision. The expert's explanation then helps improve the judge. This also gives the expert feedback about what the judge notices. Over time, the eval workflow brings knowledge from the experts into the application without requiring them to code or write prompts.

### Agreement between experts can reveal unclear business knowledge
[30:27](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=1827s)
Alon describes a FactSet copilot that answered questions about financial formulas. The team initially reached 70% accuracy, then found that repeated disagreements about the meaning of 'cash' were preventing further improvement. Different experts used the same term for different concepts, and changing the prompt simply moved the answer back and forth. Alon frames the remaining disagreement as a gray area in the knowledge base. The team used embeddings and semantic clustering to group similar questions, then looked for clusters with inconsistent answers. Experts could discuss those clusters and settle the disagreements. This improved the knowledge base and made answers more consistent for users, while also giving the copilot clearer information.

### LLM judges should be selected by human agreement on the target task
[48:00](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=2880s)
Alon proposes human agreement rate as the metric for comparing LLM-judge approaches. A team can test one judge, several judges, or a jury of different models, but it should measure each approach against the humans who matter for the application, such as users or domain experts. Different models may have different strengths, yet combining them may also reduce consistency. The answer has to come from the team's own eval and data rather than from a paper or an influential opinion. If the judge agrees with experts on most cases, experts can focus on exceptions instead of reviewing every result. Alon says this can reduce their workload substantially.

### Judge prompts and criteria should evolve with the application
[27:19](https://www.youtube.com/watch?v=d4PGxNM3Iis&t=1639s)
Alon gives two ways to improve an LLM judge. A team can update the prompt directly, or it can collect corrected judge responses and use them to fine-tune the judge. Prompt updates are quick early on, but a prompt full of edge cases can become slow and brittle. Fine-tuning becomes more suitable after collecting corrected input-output pairs. He also recommends breaking broad criteria such as 'accuracy' into observable dimensions. Domain experts often reveal that accuracy includes naming, detail, refusal behavior, tone, or persistence. Over time, those dimensions can become a rubric or checklist. The rubric itself needs feedback and revision because the application's definition of a good answer can change.

## Notable quotes
- Alon Bochman: "Nobody knows what's going to work for your task." (02:28)
- Alon Bochman: "Every time that you run an eval, don't just run it on case number 21, run it on all the cases." (09:29)
- Alon Bochman: "The metric I propose is whichever LLM judge approach reaches the highest human agreement rate." (48:52)
- Alon Bochman: "The exciting part, the economic value, the new part, the unlock, is you deliver legal expertise that used to cost 500 bucks an hour, you deliver it for 5 bucks an hour." (44:43)

## Tools & references mentioned
- RagMetrics
- ragmetrics.ai
- OpenAI
- Anthropic
- Weaviate
- Mistral
- FactSet
- Bloomberg
- MMLU
- ICLR
- DeepSeek

## Who should watch
- You are building a RAG application, copilot, or agent system and need a practical way to start evaluating it.
- Your AI product depends on finance, legal, medical, or another specialist domain, but the subject matter experts are difficult to involve in engineering work.
- You are comparing LLM-as-a-judge designs and need a measure tied to agreement with your users or domain experts.

## Related talks

- [Iterating on Your AI Evals](https://mlopstalks.com/talks/iterating-on-your-ai-evals) (Mariana Prazeres, 13:47)
- [Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development](https://mlopstalks.com/talks/navigating-the-ai-frontier-the-power-of-synthetic-data-and-agent-evaluations-in) (Boris Selitser, Okareo, 57:22)
- [Evaluating LLMs for AI Risk](https://mlopstalks.com/talks/evaluating-llms-for-ai-risk) (Finn Howell, Robust Intelligence, 10:38)
- [Reliable LLM Products, Fueled by Feedback](https://mlopstalks.com/talks/reliable-llm-products-fueled-by-feedback) (Chinar Movsisyan, Feedback Intelligence, 49:17)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)