Replit had to optimize its agent for the people building software from scratch, rather than mainly for public coding benchmarks such as SWE-bench.
2
Failure detection needs to be part of the product because standard monitoring cannot identify agents that loop, wander into unrelated work, or bypass restrictions.
3
A private evaluation harness gives engineers a way to measure regressions, test new models, and build intuition about how changes affect agent behavior.
Summary
James describes the lessons Replit learned while building and scaling Replit Agent, a coding agent inside its online development environment. The first lesson is to define the user clearly. An agent for an engineering manager working asynchronously on a large codebase has different needs from an AI-first coder building a product interactively. Replit responded by adding Rapid Build mode, prompt rewriting, and support for tight user feedback loops. James then explains why agents need application-specific failure detection, including rollback tracking, sentiment signals, trace inspection, and reports from users. He argues that public benchmarks are too general for many agent products, so teams need evaluations that behave like integration tests. Finally, he describes how Replit expanded the AI effort from three engineers to about 20 and helped engineers learn through evaluation runs and example traces in pull requests. The talk is candid about the limits of current debugging, monitoring, and guard rails.
Replit initially optimized for SWE-bench, which tests turning GitHub issues into pull requests across large repositories. James says that score did not match what many Replit users wanted. They wanted to start an idea from scratch, get a working result quickly, and iterate closely with the agent. An engineering manager working asynchronously on a large codebase has different needs from an AI-first coder building a product on a weekend. Running five agents in parallel can improve accuracy for long tasks, but it costs five times as much and slows down users who want to stay in the loop. Product choices such as supported frameworks also depend on the target user.
Rapid Build mode shortened the first working result
The first SWE-bench-oriented agent was methodical and could take six or seven minutes to create a small application. Replit built Rapid Build mode to dump out a cohesive starting point quickly, using templates, guidance, and custom prompting. It could create 10 or 15 files, accept that some small issues would remain, and then hand the work to the normal agentic loop for repair. James says this reduced the time to a working application to under two minutes. Replit also rewrote user prompts before coding. For a request to build a mailing-list site for an Italian restaurant, the rewrite added details such as using Flask, which gave the user a chance to correct the framework choice early.
James says agents fail in unusual ways and often continue pursuing a bad direction. An agent might get distracted by unrelated content, become stuck in a loop, or bypass a blocked stack after a user claims to be the CEO. Tools such as Datadog can show that an application crashed, but they cannot show that an agent is looping or that a user left in frustration. Replit generates thousands of traces each day, so engineers cannot read everything manually. The team uses LangSmith for trace monitoring and adds failure signals to the application itself. Rollbacks are especially useful because users get immediate value from them, while frequent rollbacks point engineers toward problematic behavior. User sentiment, social media, and customer support provide further evidence.
Public benchmarks do not replace product-specific evaluations
James compares public evaluations to the SAT. They measure something related to capability, but they do not capture the exact work a product must do. SWE-bench may fit an agent that handles GitHub issues and pull requests, but it does not test an agent helping someone move an image on a marketing site or replace a forest with a palm tree. Replit treats its evaluations more like integration tests. They are expensive to build and run, yet they provide a safety net for large changes and a way to compare new models. The test set can also grow whenever the team finds a new failure. Replit used its evaluations to give Anthropic feedback on Computer Use before public release.
Evaluation replaces vibes-based development with measurable change
The team initially used what James calls vibes-based development: play with the agent, decide whether a change feels better, and repeat. That process created a patchwork of prompts and hacks and made regressions hard to detect. If a trace takes several minutes to test, it is difficult to tell whether a failure rate improved from 50 percent to 90, 95, or 99 percent. James also points out that a five percent failure may require about 20 runs before a failure is expected to appear. Replit therefore built a harness that can run multiple prompts in parallel and compare behavior after a change. The AI team owns the harness, while engineers working on individual integrations can contribute their own evaluation cases.
Scaling the team required teaching engineers how tools affect models
Replit's AI team grew from about eight engineers across several AI efforts to roughly 20 people involved in the agent and its platform integrations. Many were experienced software engineers who had not worked with language models. Familiar problems, such as memory leaks, were straightforward for them to tackle after learning the codebase. Designing tools for an agent was different. Adding fields to a normal API may improve communication without changing the API's behavior much, while adding options to an agent tool can make the model more likely to go off track. Replit used evaluation runs to show how a prompt or planning change affected other parts of the system. Including example traces in every pull request that touched the agent also helped engineers develop intuition about tool use.
The current debugging interface exposes behavior without full debuggers
In the questions, James explains that Replit gives the agent views of what the user can see. These include files processed through an LSP, symbol errors, console logs, and other live information. The agent does not yet use breakpoints or step through code. When a task fails after several exchanges, it tends to add print statements instead. A consistent debugger would need to work across the languages and environments Replit eventually wants to support, which James describes as an extremely difficult problem. Replit instead relies on the information exposed through the application and on the agent's existing tools. The current approach is useful, but James is direct that the team has not solved debugging yet.
"You need to be paying a lot of attention to your traces and what your agent is doing, but if you look at every trace, you're just simply going to be overwhelmed."11:22
Who should watch
You are building a coding agent and need to choose between benchmark performance and the experience of people starting projects from scratch.
Your traces look healthy, but users still get stuck, loop, or leave without an obvious application error.
Your team is adding engineers to an agent project and needs practical ways to teach tool design and model behavior.
