# It's 2026, and We're Still Talking Evals Maggie Konstanty, Prosus | MLOps Podcast | Episode 372 | 40:57 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=9EjWR3QpJYk Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/its-2026-and-were-still-talking-evals Published: 2026-04-27 Tags: evals, monitoring, observability, testing ## TL;DR - Pre-production evals protect an agent from shipping obvious failures, while production evals must respond to real users and failure modes the team did not anticipate. - Aggregate accuracy scores hide important mistakes, so teams should define what good means for their product and connect evaluators to user behavior and business outcomes. - Evals need to remain part of product development because error analysis, prompt changes, regression checks, and team agreement about quality continue after launch. ## Summary Maggie Konstanty argues that evals should begin when a product idea begins and continue throughout development and production. Pre-ship tests can simulate personas and scenarios, but real users introduce requests and failure modes that teams did not imagine. She prefers evaluating specific failures, such as recommending pepperoni to a vegetarian user, over relying on an unexplained accuracy percentage. For food ordering, her team connects conversation evaluations with conversion, frustration, and drop-off behavior. She is skeptical of generic eval platforms when they limit exports, sampling, multi-turn analysis, or custom evaluators. Maggie says mature teams may build their own tooling, while the harder problem remains agreeing on what good means for the product. She is also direct about the workload: error analysis is slow, repetitive, and easy for teams to abandon after one pass. ## Key ideas ### Evals have to change after the agent meets real users [00:22](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=22s) Maggie says teams often build an eval suite before launch and then discover that it does not match production. A shopping agent might be tested with a request such as finding Adidas shoes in size 47, while a real user asks for shoes like LeBron James wears. Production brings new users, unusual requests, and failure modes that the team did not simulate. Pre-production evals help the team avoid shipping a product that is obviously poor. Production evals answer a different question: whether the product is delivering acceptable quality in actual use. A new feature or system-prompt change also needs testing so it does not damage existing behavior. ### Agent tests need repeated scenarios and varied user personas [03:16](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=196s) Maggie describes creating user profiles such as power users, lazy users, and busy professionals, then using an LLM to express those personas across predefined scenarios. The team may run a scenario 1, 5, 10, or even 100 times and examine the variance in outcomes. An agent can work well several times and then fail badly on the next attempt, often in a different place. She says current LLMs can understand the task while losing track after a sequence of actions. That makes these failures different from deterministic software bugs. Repetition and variation help the team find behavior that a single successful run would miss. ### A single accuracy percentage does not describe product quality [04:57](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=297s) Maggie does not rely heavily on an accuracy score such as 95 percent because the number only describes an average and leaves the meaning undefined. She prefers measures based on the relationship between human labels and the LLM's judgments, including true negative and true positive rates. Even then, she wants teams to examine concrete failure modes. Her example is a food recommendation agent that gives a vegetarian user five choices, including pepperoni pizza. That one mistake says more about the product's behavior than an overall score. Error analysis takes time, so teams often skip it even though it reveals what the evaluator is actually measuring. ### Production signals include users who leave without complaining [22:32](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=1352s) Maggie says many unhappy users do not report a problem. They simply stop using the product. Her team therefore studies drop-off, conversion, and conversation behavior together. A user can appear frustrated and still complete a purchase, while another user can leave without providing any direct feedback. She describes a satisfaction metric with satisfied users, dissatisfied users, and a large partially satisfied group whose behavior is ambiguous. Repeated requests such as 'More. Give me more. Give me more.' can indicate that the agent has failed several times. Caps lock and curse words may provide clues, but Maggie warns that they also create false positives because users can type that way accidentally or for reasons unrelated to the agent. ### Good evaluators begin with the product goal and a definition of good [16:53](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=1013s) Maggie says the right evaluators depend on the use case. Before choosing metrics, she asks what the product is meant to achieve, who uses it, and what good means in that context. For a food ordering agent, useful measures can include personalized recommendations, task execution, coherence, context retention, and whether the system remembers constraints such as a user's vegetarian preference. The same measures may not fit a car dealer product. In food ordering, her team connects conversation outcomes with conversion and checks which evaluator results occur in conversations that convert or end in frustration. Business metrics help determine which parts of an interaction deserve more attention. ### Prompt changes require regression checks and product judgment [26:46](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=1606s) When error analysis finds a failure, the cause may be a specification problem, incorrect prompting, missing context, or context supplied at the wrong time. A prompt change can reduce one failure mode while creating another, which makes the process feel like whack-a-mole. Maggie uses evals to track regressions between prompt versions and then prioritizes the failures that matter most for the product. Teams must decide whether a minor hallucination is less damaging than an incorrect recommendation, for example. She does not expect a trustworthy system to reach 100 percent. If a metric is nearly perfect all the time, she would question the data and move attention to another meaningful area. ### A large collection of disconnected evaluators can make teams fail [30:27](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=1827s) Maggie warns against setting up 20 kinds of evaluators without tying them to a product goal. Such a system measures many overlapping things, creates more failures to investigate, and encourages teams to chase 100 percent on metrics that may not matter. She recommends starting with a small core of evaluators connected to business metrics and desired behavior. Teams can add more sophisticated checks, regression testing, and agent simulation later. The evaluator set will differ by product, but the core should remain tied to the outcomes the team is trying to improve. Her criticism is practical: an oversized, disconnected eval suite can become an instruction to fail. ### Eval tooling often gets in the way of the work [32:23](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=1943s) Maggie mainly uses tooling for observability, especially viewing conversations turn by turn. She wants fast data exports through APIs, sampling, support for multi-turn conversations, and dashboards that show error rates and regressions. Some platforms slow down when exporting more than 1,000 traces, force batch processing, or encourage teams to evaluate every trace when sampling would be cheaper. She works with Arize but found creating evaluators in its interface harder than writing the code herself. She spent a day trying to configure a multi-turn evaluator and became frustrated. Her preferred tool would support failure-mode discovery, trace review, dataset sampling, training and validation splits, evaluator creation, and production use. ### The hard part of evals is team alignment, not the mechanics [38:47](https://www.youtube.com/watch?v=9EjWR3QpJYk&t=2327s) Maggie says the basic eval tasks are manageable once the process is structured. The difficult part is agreeing across the team about what matters and what good means for the product. She expects mature product teams to choose custom solutions because their use cases and evaluation methods differ. She also rejects outsourcing labeling, since labeling is an opportunity to learn how the product behaves and requires judgment about the product's standards. Evals are often treated as boring work and abandoned after one pass, even though teams need to return to them as the product changes. For Maggie, the process has to become part of the team's regular practice rather than a launch task. ## Notable quotes - Maggie Konstanty: "Evals should be constant within the development team." (02:00) - Maggie Konstanty: "What does it mean? You know, my agent is accurate 95% of the time. What does it mean?" (06:22) - Maggie Konstanty: "The biggest loss that you can have is the user that can talk to you and suddenly drop off because they're not satisfied." (23:52) - Maggie Konstanty: "Setting up 20 types of different evaluators that are not connected to your goal, I would say it's an instruction to fail." (30:47) - Maggie Konstanty: "The alignment within team members and alignment on what matters for your product is the part that is the trickiest part." (39:50) ## Tools & references mentioned - Prosus - Tiago - Arize - Phoenix - LinkedIn ## Who should watch - You are building an LLM agent and need to separate pre-launch testing from the signals that matter once users arrive. - Your team reports high eval scores but cannot explain whether conversations lead to useful outcomes, conversions, or silent user drop-off. - You are comparing eval platforms and want a practical critique of exports, sampling, multi-turn traces, custom evaluators, and outsourced labeling. ## Related talks - [Evals Aren't Useful? Really?](https://mlopstalks.com/talks/evals-arent-useful-really) (Chiara Caratelli, Prosus Group, 25:25) - [Stop Shipping on Vibes: How to Build Real Evals for Coding Agents](https://mlopstalks.com/talks/stop-shipping-on-vibes-how-to-build-real-evals-for-coding-agents) (Jessica Wang, Braintrust, 29:09) - [Iterating on Your AI Evals](https://mlopstalks.com/talks/iterating-on-your-ai-evals) (Mariana Prazeres, 13:47) - [Evaluating AI Agents: Why It Matters and How We Do It](https://mlopstalks.com/talks/evaluating-ai-agents-why-it-matters-and-how-we-do-it) (Annie Condon & Jeff Groom, Acre Security, 13:27) - [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43)