SWE-bench evaluates whether an AI system can fix real GitHub issues in existing codebases and verify its changes with tests.
2
SWE-agent improves performance by giving a language model an interface designed for software engineering tasks instead of connecting it directly to a basic terminal or editor.
3
SWE-bench has become a useful evaluation for autonomous coding systems, although it covers only one part of software engineering and remains focused on Python repositories.
Summary
John Yang presents three connected pieces of work on AI systems for software engineering: SWE-bench, SWE-agent, and later work on broader evaluation. SWE-bench turns real GitHub issue and pull request pairs into testable tasks. An agent receives an issue and a codebase, produces a fix, and runs tests against it. Yang explains why this is harder than short code-generation benchmarks: the system must find the right files, understand a large codebase, match its coding style, and make changes that pass tests. SWE-agent adds an interface designed around the actions language models need, including searching, editing, navigating, and running tests. Yang reports a large improvement over simple retrieval-based approaches. He also describes emerging solution patterns and says later systems have made substantial progress on the SWE-bench leaderboard. He is honest that the benchmark captures only one part of software engineering and that planning, test-time computation, and new workflows remain open areas.
Software engineering offers a harder test than saturated language benchmarks
Yang says older language-model benchmarks measured separate abilities such as translation, named-entity recognition, and sentiment analysis. Their results had reached 70, 80, or 90 percent and above, so they no longer separated newer models well. Short code-generation tasks such as implementing merge sort or a heap created a more useful test for a while, but HumanEval-style problems also became saturated. He proposes software engineering because it is challenging, realistic, and has built-in ways to check a solution through unit, integration, and end-to-end tests.
SWE-bench turns real GitHub maintenance work into an evaluation
SWE-bench is built from the open-source development process. A user reports an issue in a repository, and a developer responds with a pull request that fixes it. The benchmark gives a system the repository and issue, then asks it to generate a fix. Tests associated with the pull request check whether the behavior is correct. Yang describes this as a direct version of a normal maintenance workflow, rather than a self-contained programming exercise.
The benchmark pairs issue descriptions with code, patches, and executable tests
The dataset construction filters GitHub issue and pull request pairs from 12 popular Python repositories. A suitable pull request needs an associated issue and must add tests that verify the fix. The team also builds execution environments, including Docker containers, where candidate patches can be applied and tested. Each instance contains the original issue, a reference solution called the gold patch, and a test patch that captures the expected behavior.
Finding the right files is often as difficult as writing the fix
Yang describes the initial results as very low for simple retrieval-augmented generation systems that did not execute code or use multiple steps. One immediate problem is file localization. Giving a system the correct files improves performance compared with asking retrieval to find them. The system also has to reason over a large codebase and understand its existing classes, objects, higher-order functions, and coding conventions. A short interview-style Python problem usually avoids these issues.
SWE-agent improves results through a language-model-friendly computer interface
Yang explains that directly connecting a language model to a Bash terminal, VS Code, or familiar tools does not work well for complex software engineering. The action space becomes large because the system may need to navigate directories, search symbols, edit files, and run selected tests. SWE-agent combines a language model with an agent-computer interface whose commands are designed around these needs. Yang's hypothesis is that models perform better when the interface and tools are adapted to their operating patterns.
Agents settle into repeated search, edit, and test patterns
With the SWE-agent tools, Yang observes recognizable problem-solving behavior. Early turns may involve recreating the bug in a new Python file or locating the relevant code. The middle phase usually consists of editing and running tests repeatedly. After several rounds, the system begins submitting solutions. These patterns make the interaction easier to study than a single prompt followed by a single generated answer.
SWE-bench progress has been rapid, while the benchmark remains incomplete
Yang says that many teams submitted systems after SWE-agent and that OpenAI and Anthropic adopted SWE-bench as an evaluation. He reports that the leading result on one SWE-bench Verified split had passed 50 percent at the time of the talk. In the question period, he says the benchmark covers an important but limited part of software engineering. Enterprise workflows and tasks in other domains could provide additional evaluations.
More planning and recovery may reduce cascading agent errors
In response to a question about what comes after multi-turn agents, Yang points to planning, search, and test-time computation. He says models can make a change, see that it fails, and keep repeating the same approach, while a person might step back and reconsider. More explicit pipelines could require stages such as finding files, editing, and testing. He also expects evaluation to expand beyond a single agent and benchmark split as researchers examine additional software engineering workflows.
"The hypothesis that we have is that in the same way that as a user we kind of co-evolve where humans learn to use tools but then those tools become improved and optimized to make human operations on the task a little bit easier, language models can improve more and get better performance by simply giving it a better interface."18:29
Who should watch
You are building or evaluating a coding agent and need tasks that involve real repositories, issue descriptions, code changes, and executable tests.
You work on language-model evaluation and want an example of how an existing software workflow can become a benchmark.
You are deciding whether an agent needs better prompts, more turns, or a redesigned computer interface to work on large codebases.
