Catastrophic failures happen when agents can change the outside world and their actions cannot be undone easily.
2
False positives are more dangerous than false negatives because a false positive can leave an agent in an unrecoverable state.
3
Production agents need scoped workflows, clear human review tools, and separate evaluators that inspect the full execution.
Summary
Edward Upton describes failures from Asteroid's work on browser agents for healthcare, insurance, and fraud intelligence. He argues that an agent becomes dangerous when it can cause real-world side effects and its actions cannot be undone. One fraud-intelligence agent contacted a bank after a human incorrectly approved a spam-trap message. In another case, a healthcare agent reached a generic success page without booking the appointment it should have booked. Asteroid responds by limiting agent scope, combining agentic steps with deterministic scripts, allowing agents to fail out, and treating false negatives as safer than false positives. Upton also recommends human-in-the-loop tooling that makes each step visible and separate evaluators that inspect the execution after it finishes. These evaluators need to reflect the customer's domain and can use different prompts or models from the agent itself. The talk is honest that false positives cannot be eliminated completely, especially when humans and input data can also be wrong.
An agent becomes dangerous when its actions affect the world and cannot be undone
Upton distinguishes model experiments in enclosed chats from useful agents that interact with browsers, terminals, and other systems. Browser actions create state as the agent clicks buttons, moves between pages, and submits forms. That state is often difficult or impossible to reverse. A failure becomes catastrophic when the agent has changed the outside world or its own state and there is no simple undo operation. Teams therefore need to stop the agent before the failure, or detect the failure quickly enough for a person to intervene.
False positives are worse than false negatives for consequential workflows
Upton says a working agent should have zero false positives. A false positive means the system reports success even though the intended task was not completed, or it takes an action that should not have happened. A false negative is safer because a person can inspect the case and complete it later without the agent having caused an outside-world side effect. A false positive can leave the system in a state that cannot be recovered. This distinction shapes how Asteroid designs its agents and evaluators.
Poor human approval can turn a safety check into a source of failure
In one fraud-intelligence example, a bank accidentally sent an overdraft email to a spam trap. The system treated the message as potential scam intelligence, and a person approved it after reviewing a large volume of similar messages. The agent then pretended to be the bank account holder and contacted the bank, which threatened to block the account. Upton says the failure began with the approval process. Human review is useful for new agents, but tedious interfaces and weak tooling make reviewers prone to mistakes.
Healthcare agents can report success without completing the intended task
Asteroid had an agent that was meant to book a healthcare appointment. It followed a branch that did not book the appointment, then reached a page that the system classified as successful. The customer reviewing executions saw no obvious error, and the team did not know that the appointment was missing. The final page only confirmed that an inquiry had been received. Upton describes this as especially serious when the symptoms should have led to an appointment, because the end user may believe the task was completed when it was not.
Scoped workflows give customers more predictability than open-ended agents
Asteroid first tried a broad ReAct-style agent that repeatedly planned and acted from a large prompt. Upton says customers preferred sacrificing some building time for a workflow that could still adapt to changing inputs and pages while remaining predictable. The graph-based approach lets the team decide where agency is useful and where a fixed step is safer. For example, logging in can be scripted when the same login page is used repeatedly. An agent can still be used as a fallback when a page changes.
Agents should be allowed to stop instead of being pushed toward completion
Upton says agents often have a bias toward continuing until they can announce success. Asteroid instead allows an agent to fail out when it cannot safely proceed. This is appropriate when the alternative is a false positive with real side effects. The team can resolve a false negative later. It should also avoid stopping too early when an agent has partially filled a form or created state that needs recovery. Escalation can preserve that context for a later human or agent step.
Human reviewers need recordings, logs, and understandable execution views
Human-in-the-loop review works only when people can see what the agent did. Asteroid provides visibility into each step, including recordings and logs, so technical users can debug executions and less technical users can understand the path. Upton describes agent development as an iterative process of choosing where to add determinism, agency, rules, and guardrails. The same visibility helps teams improve the agent and gives healthcare and insurance users confidence that it followed an expected route.
Separate evaluators should judge the execution after the agent finishes
Upton does not trust an agent to evaluate its own work because the agent is trying to complete the task. A separate system can inspect the full log and apply customer-specific criteria after execution. Reaching the end of a workflow may mean success for one customer and failure for another. In a healthcare form, the evaluator might check that the required symptoms were entered and that the correct branch was followed, rather than accepting any page that contains generic success text. Users can create different evaluator prompts and output classifications.
Agency and determinism need to be mixed according to the task
In the question period, Upton says different customers need different balances between autonomy and predictability. A single general prompt can handle an open-ended task, but many customers prefer a graph with deterministic steps. A login step may be scripted for reliability and speed, while another step uses an agent because the page or input varies. The graph-based design lets teams choose this mix instead of applying the same level of agency everywhere.
