Real agents must handle complex, open-ended tasks in the real world over hours, days, or weeks without unrecoverable errors.
2
Agent failures come from big-brain reasoning, little-brain tactical decisions, and tool-brain interfaces, with mistakes cascading through long tasks.
3
Better agents need reinforcement learning, generalizable algorithms, scalable data, useful memory retrieval, specialized skills, and more precise tools.
Summary
Dan Jeffries argues that current frontier models are far from general agents. A real agent has to complete open-ended tasks in the physical or digital world, often over long periods, without an unrecoverable error. He groups failures into three areas: big-brain reasoning and planning, little-brain tactical decisions, and tool-brain interfaces such as APIs, coordinates, and visual controls. Errors compound across a task, so a small mistake can become a serious failure many steps later. Jeffries describes common-sense gaps, weak abstraction, poor world models, unreliable memory, and imprecise computer-use tools. His proposed path combines reinforcement learning, general-purpose algorithms, scalable workflow data, shared memory, task-specific reasoning, hot-swappable skills, and a model split into strategist, tactician, critic, translator, and arm. He is direct that multi-agent systems and retrieval methods help with some failures but do not provide a magical fix.
A real agent must survive long, open-ended work in a messy world
Jeffries defines a real agent as an AI system that can perform complex, open-ended tasks in the real world. The harder version must work over many hours, days, or weeks without unrecoverable errors. The setting can be physical, such as a robot dealing with the unpredictable world, or digital, such as an agent operating online. Simple systems that answer questions about a PDF or scrape a website with Playwright are useful, but he does not consider them complex reasoning systems. The difficulty appears when an agent has to keep making correct decisions as conditions change.
Agent failures fall into big-brain, little-brain, and tool-brain problems
The big brain covers higher-level reasoning, strategic planning, abstraction, expert knowledge, and common sense. The little brain covers tactical actions, such as choosing a checkout line, stepping forward, answering a question, and paying. Tool brain covers the quality and precision of the interfaces and appendages the system uses. An agent can reason correctly and still fail because its hand, API, coordinate system, or visual interface is unreliable. Jeffries compares this with knowing how to pick up a banana while having a broken hand.
Errors compound when an agent carries one bad decision into later steps
A wrong answer in a chat can often be corrected with another prompt. In an agent, the same mistake can affect every later step. Jeffries gives the example of code that reverts from the current OpenAI API client to the older chat completion format because the model remembers the older pattern. The resulting failure may only become visible many steps later, when someone is forced to hunt through the generated code. He also describes a browser agent that reached the right Wikipedia page, then clicked a language button, scrolled around, and spent about two minutes trying to solve a problem that was not there.
Models lack the abstract world models that give people common sense
Jeffries says common sense depends on recognizing abstract patterns across different situations. A person who is cut by a knife can connect pain and sharpness to a jagged rock, a fence spike, or a sword. A multimodal model may need to see many separate examples instead of applying the concept broadly. He uses an example from comparative psychology in which a fish trained to recognize a frog can identify the frog from a new angle. Models do not reliably have that built-in three-dimensional understanding, which causes ordinary reasoning failures.
Computer-use agents need specially designed tools because visual precision breaks down
Jeffries describes a system that placed a grid over an image so a model could identify an approximate location, such as a compose button. The method worked reasonably well until calendar dates were covered by the grid labels. OCR then missed some numbers, which left the system without reliable coordinates. The team had to infer the calendar layout mathematically and recreate the grid so the agent could click accurately. This example shows why agents often need custom recognizers, coordinate systems, and interface tooling rather than a single general-purpose vision prompt.
Reinforcement learning is better suited to tasks with a definite outcome than fuzzy reasoning
Jeffries describes systems such as OpenAI o1 as deterministic policies that work well when the answer has a clear outcome, such as mathematics or some scientific tasks. He compares this with learning to move through a fixed game path. That approach does not automatically solve fuzzy reasoning. He also explains the bitter lesson as more than simply adding compute. General-purpose algorithms such as backpropagation, reinforcement learning, and the Transformer are difficult to build, but once available they can outperform systems filled with hand-written expert knowledge. He points to AlphaZero learning without prior expert training as the direction teams should eventually pursue.
A useful agent architecture separates planning, tactics, criticism, translation, and action
Jeffries presents a five-part model for the agent's brain. The strategist handles the large plan. The tactician breaks it into smaller steps. The critic checks whether the agent is stuck or heading in the wrong direction. The translator turns an instruction such as clicking the blue submit button into coordinates through tool interfaces. The arm executes the action. He says the parts could be called multiple agents, but his team thinks of them as parts of one brain. Splitting responsibilities helps isolate failures, although it does not remove the underlying problems.
Memory depends more on retrieval and learning from failures than on storing more documents
Jeffries says retrieval is the important part of an agent memory system. Human memory searches for related experiences and analogies rather than simply storing information. His team creates synthetic memories, annotates successful and failed sequences, and records corrections such as avoiding an unnecessary language-button click. Shared memory can let one agent's learning reach the other agents in a swarm. He expects workflow data from these memories to support fine-tuning, while warning that retrieval alone will not solve every task. Sometimes the agent needs to learn a better behavior from the data.
