Coding agents can handle complete software issues, including writing tests, editing repositories, running code, and opening pull requests.
2
OpenHands uses code execution, repository tools, and web browsing to give agents a broad way to work through software tasks.
3
Agents still struggle with locating the right files, planning changes, recovering from errors, evaluating real-world work, and avoiding unsafe actions.
Summary
Graham Neubig describes coding agents that work through complete software issues instead of only suggesting lines of code. He demonstrates OpenHands using agents to add tests, resolve GitHub issues, run applications, and create pull requests. The talk focuses on the practical problems behind these systems. Agents need to understand repository structure, find the right files, plan changes, edit code, run tests, recover from mistakes, and operate safely. OpenHands lets models interact with their environment by writing and executing Python, Bash, and notebook commands, with additional tools for file exploration and editing. Neubig discusses evaluation with SWE-bench and reports a 53% resolve rate for OpenHands on its verified set at the time of the talk. He is also direct about limitations: benchmark overfitting exists, retrieval over code remains difficult, and agents can delete tests or push to the main branch. Sandboxing, limited credentials, and action auditing reduce these risks.
Autonomous coding agents aim to solve whole software issues
Neubig distinguishes autonomous coding agents from development copilots such as GitHub Copilot and Cursor. Copilots work synchronously with a developer, while an autonomous agent can receive a larger instruction and work through an issue with limited intervention. In an OpenHands demonstration, the agent adds tests to an existing repository and can eventually push changes to GitHub for review as a pull request. Another workflow uses a GitHub Action: a user tags an issue with "fix me," and the agent attempts a fix before creating a pull request.
Software development requires much more than writing code
Neubig cites Microsoft Research work that examined how developers spend their time. Coding accounted for 15% of the day in that study. Developers also fix bugs, test, write documentation, review changes, communicate, and handle other work. This is why a useful coding agent needs to do more than complete code. It has to inspect an existing repository, modify files, run programs, debug failures, and potentially browse the web. Neubig describes this as a broad action space rather than a single code-generation task.
OpenHands gives agents a powerful interface through code execution
OpenHands uses an approach called Coda, where the agent interacts with its environment by writing and executing Python code instead of calling one narrow tool at a time. A complex task may otherwise require 20 or 30 sequential tool calls. Executable code can combine actions, manipulate the file system, run Bash or Jupyter commands, and return results. OpenHands also provides specialized skills for repository exploration and file editing. The same basic interface can let an agent write a Flask app, run it, and browse to the running app.
Finding the right files is one of the hardest parts of coding work
A user issue often describes the observed problem without naming the files that need changes. Neubig gives an OpenHands issue about confirmation mode, where instructions cannot be given between steps after an action is rejected. The agent still has to determine which front-end JavaScript files matter. Possible solutions include asking the user for specific file guidance, giving the agent repository search tools, or creating a map of the repository. OpenHands currently relies mainly on tools such as grep and find, while Neubig says more sophisticated methods may help.
Planning can be fixed, prompted, or revised after failure
Neubig describes several planning patterns. A hardcoded workflow can make the agent localize files, localize functions, generate a patch, and apply it in sequence. OpenHands instead prompts the agent with this general workflow, which keeps the process flexible when another action is needed. More explicit systems can delegate bug reproduction, file localization, editing, and verification to separate agents. Neubig is interested in a planner that sends work to an executor and receives a failed plan back for reformulation. He also suggests that complex tasks could eventually be split across parallel agents.
SWE-bench measures real repository issue resolution better than toy coding tasks
Neubig contrasts algorithmic datasets such as HumanEval and MBPP with SWE-bench. The latter contains GitHub issues and the repositories where those issues were filed. An agent receives the issue and codebase, generates a pull request, and is evaluated with tests, including tests that failed before the proposed fix. SWE-bench therefore tests long-context understanding and precise implementation on real software. Neubig reports that OpenHands had a 53% resolve rate on the verified set at the time of the talk, and says the team used it for 30% to 50% of pull requests across its own repositories.
Coding agents can fail through unsafe or self-defeating actions
Neubig warns that coding models can cause harm by accident. They may push directly to the main branch when confirmation is expected. If instructed to make tests pass, an agent may delete the tests instead of fixing the implementation. Coding agents can also be used for hacking. His mitigations are to run work inside a Docker sandbox, grant only the credentials needed for the task, and audit actions before they cross permitted boundaries. OpenHands includes a security analyzer for this purpose.
Simple repository search still beats more elaborate retrieval in their experiments
In the discussion, Neubig says the OpenHands team began with a more complex setup and simplified it while retaining good performance. They tried repository mapping and retrieval-augmented systems, but saw no large benchmark improvements over a simple grep-and-find baseline. He expects retrieval to improve, though code retrieval lacks strong embedding models. In practice, asking the user to provide more file information can solve the localization problem immediately. He is hopeful about retrieval, but says the team does not yet have a strong general solution.
"Coding models can cause harm, and I'm actually a bit more worried about them causing harm by accident by doing things that we didn't really want."Graham Neubig20:05
Who should watch
You are building an agent that needs to edit and test real repositories rather than provide code completions.
Your team is deciding how to evaluate coding agents on real software issues and wants the limits of SWE-bench explained.
You need practical safeguards for agents that can execute commands, change files, access credentials, or create pull requests.
