# Maximize Your Productivity with LLMs: Task Utility Explained Julia Kiseleva, MultiOn | MLOps Community | 28:24 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=jA6AKow9pvs Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/maximize-your-productivity-with-llms-task-utility-explained Published: 2024-12-11 Tags: agents, evals ## TL;DR - Agentic applications need metrics that describe how useful they are to users, especially when task success has no single correct answer. - AgentEval uses critic, quantifier, and verifier agents to propose, score, and filter task-utility criteria from application examples and logs. - Developers can use these criteria to compare agent versions and optimize for user needs beyond latency and binary task success. ## Summary Julia Kiseleva presents AgentEval, a framework for assessing the utility of LLM-powered applications. She starts with a problem in user-facing machine learning: many tasks, such as drafting an email, do not have a single ground-truth answer, so binary success metrics are insufficient. AgentEval uses a critic agent to propose criteria from a task description plus successful and unsuccessful examples. A quantifier agent scores unseen solutions against those criteria, while a verifier agent filters criteria that are unstable or difficult to assess. Kiseleva demonstrates the approach with mathematical problem solving and compares baselines including React, vanilla GPT-4, and AutoGen. The framework can expose criteria such as clarity, completeness, efficiency, and verbosity. She is clear that the method still needs research, especially around criteria completeness and unreliable assessments such as error analysis. The practical aim is to give developers a fast way to inspect agent logs, compare versions, and align applications with what users actually value. ## Key ideas ### User-facing agents need utility measures beyond ground-truth accuracy [00:55](https://www.youtube.com/watch?v=jA6AKow9pvs&t=55s) Kiseleva says developers building LLM-powered applications need to know which metrics to optimize for users. Object recognition has labels and ground truth, but an email-writing assistant has no simple answer to whether the result succeeded. A user may copy the email and edit it, so the application's contribution is harder to measure. Some tasks, such as solving a mathematics problem, have a known correct answer. Other tasks have several possible solutions, and the developer must decide which qualities matter. AgentEval targets this broader problem of understanding the utility an application may bring to users. ### AgentEval proposes task criteria automatically from examples [08:43](https://www.youtube.com/watch?v=jA6AKow9pvs&t=523s) The framework asks a critic agent to explain a task and use one successful and one unsuccessful example from the application. It then proposes criteria with descriptions and accepted values. In the mathematics example, the output contains four criteria, although Kiseleva says many tasks produce up to 25. She describes using the approach in robotics, where a domain expert later verified a fairly complete set of criteria. This reduces the amount of user research developers need before they can begin assessing a new application. ### Quantifier agents score unseen solutions against the proposed criteria [13:27](https://www.youtube.com/watch?v=jA6AKow9pvs&t=807s) After criteria are defined, a quantifier agent assesses new solutions that were not used to create the criteria. Developers can introduce a new baseline or change the underlying agent and then compare how the solution performs across the dimensions of utility. Kiseleva says the criteria should be kept stable during this comparison, including the accepted values suggested by the critic. The application's logs provide material for identifying which criteria matter, while the quantifier produces scores that support comparisons between solutions. ### Verifier agents remove criteria that cannot be assessed reliably [14:38](https://www.youtube.com/watch?v=jA6AKow9pvs&t=878s) The verifier agent checks whether proposed criteria are stable, assessable, and suitable for adversarial examples. Kiseleva gives error analysis as a criterion that did not work well with the quantifier in the presented experiment. It may still be a useful property for an application, but the current method could not assess it properly. The verifier therefore helps distinguish a criterion worth caring about from one that the quantifier can measure reliably. This separates the desired utility of an application from the current limits of the evaluation method. ### Successful and unsuccessful examples change which criteria appear [19:13](https://www.youtube.com/watch?v=jA6AKow9pvs&t=1153s) Kiseleva distinguishes task-based criteria from solution-based criteria. A task description alone might suggest general properties for solving a mathematics problem, while successful and unsuccessful examples can reveal additional properties of the actual solutions. Code efficiency did not appear when the task was described generally, but it emerged when examples showed agents using code to solve the problems. A new solution can also introduce a new criterion, so developers need to review the criteria again when they change their agent. ### The framework tests whether scores rank degraded solutions lower [22:26](https://www.youtube.com/watch?v=jA6AKow9pvs&t=1346s) To test whether the quantifier reacts to quality differences, the researchers add noise to existing solutions. Their hypothesis is that disturbed samples should receive lower scores than the same examples without noise. Kiseleva says this happened in the experiment across several criteria. That result supports using the quantifier to rank solutions by relative quality, although it does not remove the need to examine whether each criterion is stable or meaningful. ### AgentEval gives developers an early utility assessment without requiring ground truth [23:30](https://www.youtube.com/watch?v=jA6AKow9pvs&t=1410s) Kiseleva describes AgentEval as a way to get an initial view of an agentic application's potential utility from its logs. Developers can use it when ground truth is unavailable and set up an initial assessment quickly. In the mathematics example, verbosity emerged as a possible criterion. A teaching application might prefer verbal solutions, while another application might not. Once criteria are in place, developers can optimize for them, or ask users which forms of utility they prefer. ### Utility should include how users receive an application [26:25](https://www.youtube.com/watch?v=jA6AKow9pvs&t=1585s) Kiseleva closes by arguing that developers should inspect more than latency and whether a task technically succeeds. AgentEval is intended to expose strengths and weaknesses from real interactions and show how those properties change as new solutions are introduced. The framework remains an early direction with open work around the completeness and reliability of its criteria. Its practical value is giving developers a structured way to connect agent behavior with user-facing properties. ## Notable quotes - Julia Kiseleva: "If you're building something for the end user, you need to care about how your end user says satisfied or not satisfied with an experience and what your application is even done for." (02:50) - Julia Kiseleva: "We basically trying to define kind of multi-dimensional task utility." (10:18) - Julia Kiseleva: "You don't have to get the ground truths for this, so if you have that's great, but if you don't then just want to explore it, there's something that you can in couple of minutes set up and get the first idea." (16:09) - Julia Kiseleva: "We need to optimize our applications not only towards the obvious metrics like latency, even success, but also this underlying multi-dimensional way of how users receive our application." (26:25) ## Tools & references mentioned - AgentEval - AutoGen - LLM-as-a-judge - React - GPT-4 - NeurIPS - Interactive Grounded Language Understanding (IGLU) - MultiOn ## Who should watch - You are building an LLM-powered application and do not know which user-facing properties to measure. - Your task has several acceptable answers, so binary success or a single ground-truth label does not describe quality well. - You want to compare agent versions from logs before investing in extensive user research. ## Related talks - [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50) - [How Product Metrics Become LLM Evaluations](https://mlopstalks.com/talks/how-product-metrics-become-llm-evaluations) (Raza Habib, Humanloop, 53:07) - [Evaluation](https://mlopstalks.com/talks/evaluation-panel-1-large-language-models-in-production-conference-part-2) (Abi Aryan, Independent Consultant & Amrutha Gujjar, Structured & Josh Tobin, Gantry & Sohini Roy, NVIDIA, 38:19) - [LLM Evaluation with Arize AI's Aparna Dhinakaran](https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran) (Arize AI's Aparna Dhinakaran, 55:48) - [Building Products](https://mlopstalks.com/talks/building-products) (Sam Charrington, TWIML AI Podcast & George Mathew, Insight Partners & Asmitha Rathis, PromptOps & Natalia Burina, Meta & Sahar Mor, Stripe, 45:18)