Production agents need evaluation loops that reduce reliance on human review and turn failures into improvements within hours or minutes.
2
AI products have three gaps to bridge: specifying intent, generalizing from that specification to real tasks, and comprehending what the system actually did.
3
Teams should separate alerts they can solve now, alerts the agent can learn to solve, and alerts it is unlikely to solve.
Summary
Shreya Shankar and Willem Pienaar discuss why production AI agents are difficult to evaluate and improve. Willem describes Cleric, an agent that investigates production alerts by searching observability systems, calling APIs, and assembling findings. The central problem is weak ground truth. Users may approve a root cause without knowing whether it is correct, so teams need implicit signals and automated analysis that reduce dependence on manual review. Shankar frames human-AI interaction around three gulfs: specification, generalization, and comprehension. These gaps affect prompt design, transfer to real data, and validation of outputs. The conversation also covers feedback databases, prompt suggestions, traces, task-level heat maps, and simulated production environments. Both speakers reject the idea that more evaluation automatically produces 100% performance. Pienaar instead groups tasks into what the agent can solve, can learn, or will probably never solve, then focuses on making the first group valuable and expanding the second.
Willem Pienaar says production diagnosis does not have the clean reference answers available in code or writing tasks. When an engineer reviews a proposed root cause, they may say it looks good without knowing whether it is real. That makes user feedback insufficient as the only evaluation signal. He argues that the human should be removed from the review and labeling loop as much as possible, because otherwise the system remains blocked on people before it can improve. The desired loop takes a production failure, incorporates it into the system, and learns from it within hours or minutes rather than days or weeks.
AI data pipelines expose the same verification problems as agents
Shreya Shankar describes data processing over PDFs, transcripts, and other unstructured data as a useful test case for agent design. In her system, an LLM executes map, reduce, filter, and related operations over documents. Verification is difficult because the user may not know all the information in the source data. They must check whether an insight was extracted correctly, whether it exists in the data, and whether the system missed something. A pipeline can also fail because of an upstream issue, which makes it hard to know what observation should be encoded back into a prompt.
Prompt quality is only one of the gaps between intent and results
Shankar builds on Don Norman's ideas about the gulfs of execution and evaluation. She separates the AI problem into specification and generalization. Specification is the work of communicating intent clearly to the model. Generalization is the gap between a well-specified request and the model applying it correctly to a particular task or dataset. Her team is building tools that mark bad outputs and suggest prompt changes for the specification gap. Better task decomposition, agent workflows, and models are aimed at the generalization gap. A good prompt does not guarantee that the model will execute it correctly.
Interfaces should collect useful feedback without demanding constant manual review
The speakers discuss interface controls that produce implicit feedback. If a user expands a finding or searches further from it, the system can treat that action as evidence that the result was useful. Ignoring a finding is also a signal. Shankar describes combining binary actions with open-ended comments, where users highlight a bad document or output and explain what is wrong. Those comments are stored in a database. An assistant can read the accumulated feedback and suggest targeted prompt changes or diffs, which users can accept or reject. This approach preserves user input while reducing the need for people to remember every past failure mode.
Users need a mental model of the system to steer it
Shankar argues that hiding an AI system behind a black box can make it harder to use. A developer working in Cursor, for example, can steer it more effectively after understanding that it uses retrieval over code, that certain files may be indexed, and that open tabs or recent edits affect the context. The same issue appears in other agent interfaces. Users need to understand enough about the system's behavior to provide useful guidance. The speakers also discuss giving users less information at first, with controls such as expand or search further, so they can explore outputs without being overwhelmed.
The three gulfs are specification, generalization, and comprehension
Shankar adds a third gap to the specification and generalization gaps: comprehension. Users must understand what happened, inspect long tails of outputs, and decide whether the result is correct. This applies to data processing, coding tools, and other complex AI products. The three gaps form a cycle. After a user comprehends an output, they need to specify a correction, then check whether it generalizes to the real task. The speakers argue that product interfaces and evaluation tools will need to address all three rather than focusing only on prompt steering.
Evaluation reaches a ceiling, so teams need explicit task categories
The speakers reject the idea that adding more evaluations will always push an agent toward 100% performance. Shankar describes a saturation point where new scenarios no longer change the result. Pienaar groups alerts into three categories: cases Cleric is already confident it can solve, cases it expects to learn in production, and cases it may never solve. The first category establishes whether the product is useful enough to deploy. The second drives improvement. The third defines the boundary. They prefer confidence around specific behaviors or measures over a single accuracy percentage that hides what the system can actually do.
Task heat maps reduce the need for humans to inspect every trace
Willem explains that Cleric starts with traces showing tool calls, prompts, reasoning, and other details from an individual run. Manual trace review is too slow, so the team post-processes traces using deterministic and AI-based summarization. It clusters recurring tasks, such as querying logs or inspecting alert conversations, then measures properties such as API failures, loops, distraction, efficiency, and task success. The results form heat maps with tasks as rows and metrics as columns. A bright problem area can show that the agent consistently fails at one task, such as querying metrics, without requiring a person to review every run.
Simulated environments speed testing but introduce their own failures
Pienaar says live production environments made evaluation slow because creating test scenarios could take a week. The team tried real cloud projects and clusters, then mocked APIs, but mocked systems introduced nondeterminism. They moved toward simulations that reproduce enough of an observability system to test the agent without recreating every API. The trade-off is fidelity. A simulation can omit unusual behavior, and some models can detect that they are in a test environment and stop investigating. The team therefore has to make simulated names, logs, and system behavior realistic while also checking whether changes came from the agent or from the environment.
"There are three gaps: specifying, then generalizing from your specification to your actual task or data, then third is comprehension, understanding what the hell happened."Shreya Shankar26:02
Who should watch
You are building an agent that must diagnose production issues and need an evaluation loop that does not depend on engineers manually checking every answer.
Your AI application produces plausible outputs, but your team cannot say whether failures come from prompts, models, data, tools, or the environment.
You are deciding how much control to expose to users and need practical ideas for feedback controls, traces, heat maps, and simulated tests.
