# Catastrophic agent failure and how to avoid it Edward Upton, Asteroid | Agents in Production 2025 | 25:21 Source: https://www.youtube.com/watch?v=wsZdGllOAGc Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/catastrophic-agent-failure-and-how-to-avoid-it Published: 2025-09-04 Tags: evals, guardrails, human-in-the-loop, workflows ## TL;DR - Catastrophic failures happen when agents can change the outside world and their actions cannot be undone easily. - False positives are more dangerous than false negatives because a false positive can leave an agent in an unrecoverable state. - Production agents need scoped workflows, clear human review tools, and separate evaluators that inspect the full execution. ## Summary Edward Upton describes failures from Asteroid's work on browser agents for healthcare, insurance, and fraud intelligence. He argues that an agent becomes dangerous when it can cause real-world side effects and its actions cannot be undone. One fraud-intelligence agent contacted a bank after a human incorrectly approved a spam-trap message. In another case, a healthcare agent reached a generic success page without booking the appointment it should have booked. Asteroid responds by limiting agent scope, combining agentic steps with deterministic scripts, allowing agents to fail out, and treating false negatives as safer than false positives. Upton also recommends human-in-the-loop tooling that makes each step visible and separate evaluators that inspect the execution after it finishes. These evaluators need to reflect the customer's domain and can use different prompts or models from the agent itself. The talk is honest that false positives cannot be eliminated completely, especially when humans and input data can also be wrong. ## Key ideas ### An agent becomes dangerous when its actions affect the world and cannot be undone [02:07](https://www.youtube.com/watch?v=wsZdGllOAGc&t=127s) Upton distinguishes model experiments in enclosed chats from useful agents that interact with browsers, terminals, and other systems. Browser actions create state as the agent clicks buttons, moves between pages, and submits forms. That state is often difficult or impossible to reverse. A failure becomes catastrophic when the agent has changed the outside world or its own state and there is no simple undo operation. Teams therefore need to stop the agent before the failure, or detect the failure quickly enough for a person to intervene. ### False positives are worse than false negatives for consequential workflows [01:46](https://www.youtube.com/watch?v=wsZdGllOAGc&t=106s) Upton says a working agent should have zero false positives. A false positive means the system reports success even though the intended task was not completed, or it takes an action that should not have happened. A false negative is safer because a person can inspect the case and complete it later without the agent having caused an outside-world side effect. A false positive can leave the system in a state that cannot be recovered. This distinction shapes how Asteroid designs its agents and evaluators. ### Poor human approval can turn a safety check into a source of failure [04:08](https://www.youtube.com/watch?v=wsZdGllOAGc&t=248s) In one fraud-intelligence example, a bank accidentally sent an overdraft email to a spam trap. The system treated the message as potential scam intelligence, and a person approved it after reviewing a large volume of similar messages. The agent then pretended to be the bank account holder and contacted the bank, which threatened to block the account. Upton says the failure began with the approval process. Human review is useful for new agents, but tedious interfaces and weak tooling make reviewers prone to mistakes. ### Healthcare agents can report success without completing the intended task [06:26](https://www.youtube.com/watch?v=wsZdGllOAGc&t=386s) Asteroid had an agent that was meant to book a healthcare appointment. It followed a branch that did not book the appointment, then reached a page that the system classified as successful. The customer reviewing executions saw no obvious error, and the team did not know that the appointment was missing. The final page only confirmed that an inquiry had been received. Upton describes this as especially serious when the symptoms should have led to an appointment, because the end user may believe the task was completed when it was not. ### Scoped workflows give customers more predictability than open-ended agents [08:05](https://www.youtube.com/watch?v=wsZdGllOAGc&t=485s) Asteroid first tried a broad ReAct-style agent that repeatedly planned and acted from a large prompt. Upton says customers preferred sacrificing some building time for a workflow that could still adapt to changing inputs and pages while remaining predictable. The graph-based approach lets the team decide where agency is useful and where a fixed step is safer. For example, logging in can be scripted when the same login page is used repeatedly. An agent can still be used as a fallback when a page changes. ### Agents should be allowed to stop instead of being pushed toward completion [09:08](https://www.youtube.com/watch?v=wsZdGllOAGc&t=548s) Upton says agents often have a bias toward continuing until they can announce success. Asteroid instead allows an agent to fail out when it cannot safely proceed. This is appropriate when the alternative is a false positive with real side effects. The team can resolve a false negative later. It should also avoid stopping too early when an agent has partially filled a form or created state that needs recovery. Escalation can preserve that context for a later human or agent step. ### Human reviewers need recordings, logs, and understandable execution views [09:42](https://www.youtube.com/watch?v=wsZdGllOAGc&t=582s) Human-in-the-loop review works only when people can see what the agent did. Asteroid provides visibility into each step, including recordings and logs, so technical users can debug executions and less technical users can understand the path. Upton describes agent development as an iterative process of choosing where to add determinism, agency, rules, and guardrails. The same visibility helps teams improve the agent and gives healthcare and insurance users confidence that it followed an expected route. ### Separate evaluators should judge the execution after the agent finishes [11:33](https://www.youtube.com/watch?v=wsZdGllOAGc&t=693s) Upton does not trust an agent to evaluate its own work because the agent is trying to complete the task. A separate system can inspect the full log and apply customer-specific criteria after execution. Reaching the end of a workflow may mean success for one customer and failure for another. In a healthcare form, the evaluator might check that the required symptoms were entered and that the correct branch was followed, rather than accepting any page that contains generic success text. Users can create different evaluator prompts and output classifications. ### Agency and determinism need to be mixed according to the task [18:29](https://www.youtube.com/watch?v=wsZdGllOAGc&t=1109s) In the question period, Upton says different customers need different balances between autonomy and predictability. A single general prompt can handle an open-ended task, but many customers prefer a graph with deterministic steps. A login step may be scripted for reliability and speed, while another step uses an agent because the page or input varies. The graph-based design lets teams choose this mix instead of applying the same level of agency everywhere. ## Notable quotes - "A working agent is one that has zero false positives." (01:46) - "It's fine if you have a false negative, not if you have a false positive." (09:08) - "Your agent is only as accurate in its output as the evaluators you use." (11:33) - "You're always going to have false positives." (19:31) - "Who do you blame if an agent fails?" (23:56) ## Tools & references mentioned - Asteroid - Demetrios Brinkmann - ReAct ## Who should watch - You are deploying browser or terminal agents that can submit forms, contact services, or change user state. - Your review process relies on people approving many agent actions without clear execution traces. - You need to decide where a workflow should use an open-ended agent and where a scripted step or separate evaluator is safer. ## Related talks - [How to build agents that take ACTION](https://mlopstalks.com/talks/how-to-build-agents-that-take-action) (Alex Salazar, Arcade, 28:59) - [Why Agents Are Stupid & What We Can Do About It](https://mlopstalks.com/talks/why-agents-are-stupid-what-we-can-do-about-it) (Dan Jeffries, Kentauros AI, 31:58) - [Building Replit Agent - Hard Lessons Learned](https://mlopstalks.com/talks/building-replit-agent-hard-lessons-learned) (, 32:31) - [Building AI Agents That Survive Production](https://mlopstalks.com/talks/building-ai-agents-that-survive-production) (Haytham Abuelfutuh, Union.ai, 31:59) - [Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild](https://mlopstalks.com/talks/beyond-the-gold-standard-evaluating-and-trusting-agents-in-the-wild) (Sanjana Sharma, Prosus, 24:45)