# Web Agents: The Cutting Edge of AI is Here?

Paul van der Boor & Chiara Caratelli, Prosus Group | Agents in Production Series | 45:53
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=QbxN_PN7kZc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/web-agents-the-cutting-edge-of-ai-is-here
Published: 2025-02-26
Tags: agents, evals, planning, tool-use

## TL;DR
- Paul van der Boor argues that web agents are a natural step between API-based agents and systems that use an entire computer or the physical world.
- Chiara Caratelli found that reliable web agents require narrow tasks, limited action spaces, stored successful trajectories, and a clear split between planning and execution.
- The Prosus Group team treats web-agent work as software engineering, using deterministic code wherever an agent is unnecessary and testing against representative e-commerce tasks.

## Summary
Paul van der Boor and Chiara Caratelli explain why Prosus Group is building agents that use websites through browsers. APIs do not expose every action involved in shopping, booking travel, ordering food, or communicating with sellers. A web agent can follow the same visible steps as a person, although websites contain popups, dynamic content, large DOMs, CAPTCHAs, and unexpected state changes. Caratelli describes building a food-ordering agent and then creating a narrower internal framework after existing tools performed inconsistently. Their design separates a planner from an executor, limits the executor's tools, stores successful paths, and switches between slower visual interaction, faster known flows, and automated reflex actions. The speakers are direct about the limits. Agents often get stuck, and benchmark results did not predict performance on their own tasks. They recommend treating production systems as software projects, with modular components, deterministic steps, targeted evaluations, and as few LLM calls as possible.

## Key ideas
### Web agents extend agent access from APIs to browser interactions
[00:53](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=53s)
Paul van der Boor defines agents as LLMs that interact with the world. The web is one of the ways people do that, so browsers are a natural next interface after simple APIs and function calling. Prosus Group's companies reach consumers through websites and apps, which makes web interaction directly relevant to tasks such as discovering products, buying goods, and completing transactions. He describes this progression as gradual: APIs first, then browsers, then broader computer use, and eventually interaction with the physical world.

### Web agents can handle workflows that do not have a usable API
[05:01](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=301s)
The speakers say many actions people take online are not exposed through APIs. Ordering a flight, getting a pizza, or buying a gift involves searching, inspecting results, clicking through pages, reading reviews, selecting options, and completing several dependent steps. Paul gives Amazon and LinkedIn as examples where there is no single request that means 'find a gift and add it to my basket'. The browser already contains the workflow that humans use, so an agent can potentially operate it without a separate integration for every action.

### Websites create difficult conditions for agents
[10:50](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=650s)
Chiara Caratelli says the food-ordering project looked simple until the team tried it. The agent had to understand dietary restrictions, location, time, local events, promotions, and the available options across food platforms. Existing tools struggled because websites are built for people. Dynamic content, loading behavior, large and changing DOMs, popups, CAPTCHAs, filters, toppings, and other unexpected interactions all make execution harder. The team found that agents could fail even on tasks that seemed much less complex than their data-analysis systems.

### A narrow action space makes a web agent more reliable
[19:42](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=1182s)
Caratelli says the team improved performance by optimizing for food ordering instead of trying to serve the whole web. They selected strategies that matched the platforms and task, loaded page-specific instructions, and combined actions that normally occur together. Typing a search and pressing Enter became one tool because the agent rarely needed to type without submitting. They also adapted scrolling for long menus and restaurant lists. Her advice is to remove capabilities the task does not need and design around the actual workflow.

### Stored trajectories let agents move faster on familiar sites
[22:27](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=1347s)
The team manually performed the relevant flows before building the agent, then stored the paths and page state from successful runs. Caratelli describes three operating modes. Traditional mode uses a screenshot and page content before choosing the next action. Faster mode skips the screenshot on pages and tasks the system already knows. Reflex mode automates parts of the interaction like a macro. The agent tries the fast path first and falls back to slower, more thorough interaction when it fails. Paul compares this learned familiarity to a person who already knows how to use a familiar site.

### Planning and execution need different responsibilities
[27:14](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=1634s)
Caratelli describes a planner that turns a user request into steps and an executor that performs limited browser actions such as clicking, entering text, and scrolling. The executor can receive a specific instruction, such as opening a site and searching for a product, then stop when the task is complete. Paul says planning needs strong reasoning about user intent, while execution needs a good understanding of the available actions in a particular environment. This separation also makes it easier to use smaller or more constrained components for execution.

### Simulation exposes inconsistent web-agent behavior
[28:42](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=1722s)
The team found that WebArena results did not predict how systems performed on their own tasks. Agents could succeed once and fail the next time, so the researchers ran repeated simulations of the same kind of task. They sent agents to explore sites, defined states such as finding an item or adding it to a cart, and examined which trajectories reached those states. Agents often entered loops because the task, plan, or available action space was unclear. Mapping a site's actions can make later execution easier.

### Production systems should use deterministic code wherever possible
[34:34](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=2074s)
Caratelli's main lesson from six months of work is to approach web agents as software engineering projects. Teams should keep modules separate, control the parts that matter, and limit LLM calls. She gives the example of retrieving a user's dietary information: an agent could query a database, but ordinary code can fetch the data and place it in the prompt more simply. Paul adds that agents are useful for quick prototypes, while systems intended for tens or hundreds of millions of users need to be distilled into workflows that are easier to control and scale.

### Agent-friendly sites can expose useful structure and marketplace information
[41:23](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=2483s)
Paul says a web agent that visits an existing marketplace can only use what the site exposes. A platform could provide more useful help by giving its own agent access to seller reputation, location, search history, supply and demand, reasonable category prices, and customer reviews. Chiara adds that even simple page structure matters. Markdown descriptions can make large e-commerce pages easier for agents to process. Both argue that websites will need clearer instructions and more deliberate access to information if agents are expected to complete transactions.

## Notable quotes
- Paul van der Boor: "Agents are LLMs that interact with the world." (00:53)
- Chiara Caratelli: "The way to make an agent succeed is to limit the amount of choices it has to do as much as possible." (19:42)
- Chiara Caratelli: "This is not data science project but a software engineering project where there are some LLM steps in." (35:00)
- Chiara Caratelli: "If there are things that can go through a deterministic approach it's much better." (36:00)

## Tools & references mentioned
- Prosus Group
- OLX
- iFood
- Amazon
- LinkedIn
- Uber
- Google Maps
- Booking.com
- Google Flights
- WebVoyager
- WebArena
- Selenium

## Who should watch
- You are deciding whether browser interaction adds value beyond API integrations for shopping, booking, or other multi-step web workflows.
- Your team is prototyping an agent and needs practical guidance on narrowing the task, separating planning from execution, and handling known website paths.
- You are building an e-commerce platform and want to understand what makes pages easier for agents to inspect and operate.

## Related talks

- [AI Agents: The Future of Productivity, or Just a Fad?](https://mlopstalks.com/talks/ai-agents-the-future-of-productivity-or-just-a-fad) (Sam Partee, Arcade AI, 35:18)
- [AI Agents: The Evolution of Our Digital Friends](https://mlopstalks.com/talks/ai-agents-the-evolution-of-our-digital-friends) (, 10:16)
- [We're Using AI Agents at Work (and it's amazing)](https://mlopstalks.com/talks/were-using-ai-agents-at-work-and-its-amazing) (Euro Beinat, Prosus Group, 27:37)
- [A New Way of Building with AI](https://mlopstalks.com/talks/a-new-way-of-building-with-ai) (Jiquan Ngiam, Lutra AI, 1:04:46)
- [The Agent Landscape - Lessons Learned Putting Agents Into Production](https://mlopstalks.com/talks/the-agent-landscape-lessons-learned-putting-agents-into-production) (Paul van der Boor & Floris Fok, Prosus Group, 1:08:41)