# Few Shot Code Generation to Autonomous Software Engineering Agents John Yang, Stanford University | MLOps Community | 26:05 Source: https://www.youtube.com/watch?v=f6D5eHNr_VI Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/few-shot-code-generation-to-autonomous-software-engineering-agents Published: 2024-12-02 Tags: agents, benchmarks, coding-agents, evals, tool-use ## TL;DR - SWE-bench evaluates whether an AI system can fix real GitHub issues in existing codebases and verify its changes with tests. - SWE-agent improves performance by giving a language model an interface designed for software engineering tasks instead of connecting it directly to a basic terminal or editor. - SWE-bench has become a useful evaluation for autonomous coding systems, although it covers only one part of software engineering and remains focused on Python repositories. ## Summary John Yang presents three connected pieces of work on AI systems for software engineering: SWE-bench, SWE-agent, and later work on broader evaluation. SWE-bench turns real GitHub issue and pull request pairs into testable tasks. An agent receives an issue and a codebase, produces a fix, and runs tests against it. Yang explains why this is harder than short code-generation benchmarks: the system must find the right files, understand a large codebase, match its coding style, and make changes that pass tests. SWE-agent adds an interface designed around the actions language models need, including searching, editing, navigating, and running tests. Yang reports a large improvement over simple retrieval-based approaches. He also describes emerging solution patterns and says later systems have made substantial progress on the SWE-bench leaderboard. He is honest that the benchmark captures only one part of software engineering and that planning, test-time computation, and new workflows remain open areas. ## Key ideas ### Software engineering offers a harder test than saturated language benchmarks [03:02](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=182s) Yang says older language-model benchmarks measured separate abilities such as translation, named-entity recognition, and sentiment analysis. Their results had reached 70, 80, or 90 percent and above, so they no longer separated newer models well. Short code-generation tasks such as implementing merge sort or a heap created a more useful test for a while, but HumanEval-style problems also became saturated. He proposes software engineering because it is challenging, realistic, and has built-in ways to check a solution through unit, integration, and end-to-end tests. ### SWE-bench turns real GitHub maintenance work into an evaluation [07:39](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=459s) SWE-bench is built from the open-source development process. A user reports an issue in a repository, and a developer responds with a pull request that fixes it. The benchmark gives a system the repository and issue, then asks it to generate a fix. Tests associated with the pull request check whether the behavior is correct. Yang describes this as a direct version of a normal maintenance workflow, rather than a self-contained programming exercise. ### The benchmark pairs issue descriptions with code, patches, and executable tests [10:07](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=607s) The dataset construction filters GitHub issue and pull request pairs from 12 popular Python repositories. A suitable pull request needs an associated issue and must add tests that verify the fix. The team also builds execution environments, including Docker containers, where candidate patches can be applied and tested. Each instance contains the original issue, a reference solution called the gold patch, and a test patch that captures the expected behavior. ### Finding the right files is often as difficult as writing the fix [12:59](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=779s) Yang describes the initial results as very low for simple retrieval-augmented generation systems that did not execute code or use multiple steps. One immediate problem is file localization. Giving a system the correct files improves performance compared with asking retrieval to find them. The system also has to reason over a large codebase and understand its existing classes, objects, higher-order functions, and coding conventions. A short interview-style Python problem usually avoids these issues. ### SWE-agent improves results through a language-model-friendly computer interface [16:48](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=1008s) Yang explains that directly connecting a language model to a Bash terminal, VS Code, or familiar tools does not work well for complex software engineering. The action space becomes large because the system may need to navigate directories, search symbols, edit files, and run selected tests. SWE-agent combines a language model with an agent-computer interface whose commands are designed around these needs. Yang's hypothesis is that models perform better when the interface and tools are adapted to their operating patterns. ### Agents settle into repeated search, edit, and test patterns [20:00](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=1200s) With the SWE-agent tools, Yang observes recognizable problem-solving behavior. Early turns may involve recreating the bug in a new Python file or locating the relevant code. The middle phase usually consists of editing and running tests repeatedly. After several rounds, the system begins submitting solutions. These patterns make the interaction easier to study than a single prompt followed by a single generated answer. ### SWE-bench progress has been rapid, while the benchmark remains incomplete [21:08](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=1268s) Yang says that many teams submitted systems after SWE-agent and that OpenAI and Anthropic adopted SWE-bench as an evaluation. He reports that the leading result on one SWE-bench Verified split had passed 50 percent at the time of the talk. In the question period, he says the benchmark covers an important but limited part of software engineering. Enterprise workflows and tasks in other domains could provide additional evaluations. ### More planning and recovery may reduce cascading agent errors [23:38](https://www.youtube.com/watch?v=f6D5eHNr_VI&t=1418s) In response to a question about what comes after multi-turn agents, Yang points to planning, search, and test-time computation. He says models can make a change, see that it fails, and keep repeating the same approach, while a person might step back and reconsider. More explicit pipelines could require stages such as finding files, editing, and testing. He also expects evaluation to expand beyond a single agent and benchmark split as researchers examine additional software engineering workflows. ## Notable quotes - "The task is given an issue and a codebase, the language model must generate a pull request, or just generally speaking a fix, that corresponds to resolving that issue." (08:41) - "It's not just a matter of knowing the exact solution, because it's not usually readily apparent, understanding the rest of the codebase and what context to include and what to exclude and what to not consider is an incredibly important and also challenging part of this benchmark." (15:04) - "The hypothesis that we have is that in the same way that as a user we kind of co-evolve where humans learn to use tools but then those tools become improved and optimized to make human operations on the task a little bit easier, language models can improve more and get better performance by simply giving it a better interface." (18:29) - "Sometimes there's cascading errors, so it's if it tries to make an edit and it doesn't work, it'll kind of keep hammering home versus a person might kind of take a step back." (24:20) ## Tools & references mentioned - John Yang - Diyi Yang - Karthik Narasimhan - SWE-bench - SWE-agent - SWE-bench Verified - Devon - Cognition Labs - OpenAI - Anthropic - HumanEval - GLUE - SuperGLUE - GitHub - Django - Flask - Matplotlib - scikit-learn - pytest - SymPy - Astropy - NumPy - Pandas ## Who should watch - You are building or evaluating a coding agent and need tasks that involve real repositories, issue descriptions, code changes, and executable tests. - You work on language-model evaluation and want an example of how an existing software workflow can become a benchmark. - You are deciding whether an agent needs better prompts, more turns, or a redesigned computer interface to work on large codebases. ## Related talks - [AI Coding Agents Change Software Development Forever](https://mlopstalks.com/talks/ai-coding-agents-change-software-development-forever) (Michele Catasta, Replit & Aparna Dhinakaran, Arize AI & Erik Schluntz, Anthropic, 20:32) - [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43) - [The Open Source AI Coding Revolution](https://mlopstalks.com/talks/the-open-source-ai-coding-revolution) (Graham Neubig, All Hands AI, 28:00) - [Building Replit Agent - Hard Lessons Learned](https://mlopstalks.com/talks/building-replit-agent-hard-lessons-learned) (, 32:31) - [The Future of AI Agents are Sandboxes](https://mlopstalks.com/talks/the-future-of-ai-agents-are-sandboxes) (Jonathan Wall, Runloop.ai, 58:04)