Watch on YouTubeAn LLM is only the reasoning engine; a useful agent also needs memory, tools, APIs, and the ability to plan and revise its actions.
Agents become harder to evaluate as they move from structured APIs to web browsing, computer control, and robotics.
The practical reason to build agents is that they can perform tasks beyond question answering, although current interfaces and APIs often break under agent use.
The talk explains AI agents through a Mars rover built from a Raspberry Pi, sensors, a camera, and memory. An LLM provides the reasoning, while an agent needs tools to access current or private data, memory to retain interactions, and the ability to plan, critique its work, and revise its steps. The speakers describe several forms of agents, including computer-use agents, web agents, API-based agents, voice agents, and possible game characters. API-based agents are the easiest starting point because their inputs and outputs are structured and testable. Web browsing, computer control, and robotics add more open-ended behavior and greater difficulty. The discussion also covers evaluation, latency, and cost. Agents offer more possible tasks than traditional question-answer systems, but many websites and APIs were not designed for them. Even simple services can be unreliable, change over time, or expose inconsistent interfaces.
The speaker compares an LLM with the Raspberry Pi in a Mars rover. The LLM can reason, but it does not by itself have sensors, memory, wheels, or a way to affect the world. An agent adds those missing parts so the system can interact with external data and carry out actions. This changes the engineering problem from producing an answer to connecting a reasoning model reliably to everything it needs around it.
An agent needs memory of earlier interactions and tools for actions such as fetching information through an API, creating a file, or storing data. Training data has an endpoint, so the system needs access to proprietary information and events that happened after training. It may also need to write its own information to memory. The hard design question is when that memory should be retrieved and how it should shape the next answer or action.
The described system can make a plan from a request, follow its steps, inspect what happened, critique the result, and return to an earlier step when the plan did not work. That loop has to work reliably rather than only once in a demonstration. Shipping it to real customers adds production requirements around the whole system, including its connections to tools, data, and memory.
The speakers group agents by how they interact with the world. Computer-use agents control a whole computer, web agents work through a browser, and API agents use external services directly. Voice agents are another form, and game characters may also fit the category. APIs are the natural first tool because they are documented and structured, with defined inputs and expected outputs that make testing and evaluation easier.
Web browsing is harder than calling a well-defined API because the agent has to operate in a less predictable environment. Computer-use agents go further by controlling a desktop. The speakers mention systems that can create pivot tables, download files, and process them from brief instructions. Robotics adds interaction with the physical world, which they place beyond browsing and computer control on the path toward more sophisticated agents.
Traditional question-answer systems can answer questions reasonably well, but tools let an agent do more. The examples include booking trips, ordering food, helping select products, and supporting users across e-commerce platforms. The benefit is the range of actions available after the system can access APIs, applications, or the web. That range also brings more ways for the system to fail.
Many existing interfaces were built for people rather than autonomous systems. The speakers describe APIs that break, change over time, and differ from one another. Even obtaining weather data through an apparently simple weather API can be difficult. If an agent is not kept up to date with those changes, a workflow that depends on several services can become unreliable.
"What we basically have today with these powerful gen models, large language models, is just the reasoning engine, basically the Raspberry Pi without anything else."00:40