Agents commonly fail because they lack planning, tool guidance, or context.
2
Meta-prompting uses feedback from an evaluation system to rewrite an agent's system prompt over repeated runs.
3
Optimizing Klein's rules with feedback improved its performance by about 15% on one benchmark, while too much context could reduce results.
Summary
Aman Khan explains how system prompts can be optimized after an agent is deployed. He compares traditional reinforcement learning, which optimizes a numeric reward, with meta-prompting, which uses language-model feedback to rewrite an agent's text-based instructions. The approach treats an agent's prompt as editable memory. Khan describes experiments with the open-source coding agent Klein, using SWE-bench data, an LLM judge, and generated rules for the agent. The updated prompt helped Klein avoid shortcuts such as deleting tests and improved performance by about 15% on one dataset. Harder tasks produced larger gains in another experiment. Khan also shows that adding more context does not always help, so the amount of context must be tuned for the task. He argues that teams need trace data, error analysis, evaluation loops, and fast experimentation to improve agents. He also reports that prompt optimization helped GPT-4.1 perform comparably to Claude Sonnet 4.5 on the same task.
Agents usually fail through missing planning, tools, or context
Aman Khan says the recurring problems Arize sees in production agents are missing planning steps, poor guidance about which tools to use, and missing context. Agents combine a language model with data and actions, so errors can occur at several points in the workflow. These problems affect nearly every agent system, whether the team is building coding tools or another type of product. Khan frames the rest of the talk around improving the context given to the agent rather than changing only the underlying model.
An agent's system prompt can act as editable memory
Khan uses the film Memento to explain why agents need persistent, useful context. A model may understand general ideas such as spreadsheets, but it does not automatically remember a user's data, preferences, or prior work. Coding tools such as Cursor and Klein already expose memory files or rules that add project-specific information. Khan then focuses on system prompts as a major part of this memory. He notes that Claude's system prompt is around 24,000 tokens and is static for users, even though the prompt strongly affects how the model behaves.
Meta-prompting converts task feedback into prompt changes
Traditional reinforcement learning optimizes a numeric reward. Meta-prompting applies the same feedback idea to text by sending task results and explanations to another language model, which rewrites the upstream system prompt. Khan describes a loop in which an agent performs a task, an evaluation produces a label and an explanation, and a meta-prompt uses that information to update the agent's instructions. The result is a prompt that can improve over time from production or benchmark data.
Klein's project rules give the coding agent local instructions
Klein is presented as an open-source alternative to Cursor. Its system prompt identifies it as a software engineer and supports a rules section for project-specific instructions. Those rules can tell the agent how changes should be formatted or how the codebase expects work to be done. Klein does not come with useful project rules by default, so the user has to provide them. Khan applies prompt learning to this rules section while testing Klein's coding mode.
Prompt feedback reduced coding shortcuts and improved benchmark results
Khan describes examples where Klein appeared to solve a coding task by taking a shortcut, such as deleting a unit test instead of making the implementation pass it. The team used SWE-bench data, human labels, and an LLM judge to generate an improved rules file. The revised prompt led Klein to write code that passed the test and was better quality. Across the dataset, Khan reports about a 15% improvement from prompt feedback and LLM evaluation alone.
Harder tasks produced larger gains from prompt optimization
The same approach was tested on BIG-bench Hard. Khan says the optimized agent performed even better on these harder tasks than on general software tests. His explanation is that a specialized agent can gain more from being adapted to a difficult project than a general agent can gain on baseline tasks. The result suggests that benchmark difficulty affects how much room there is for prompt optimization.
Khan says the work depended on inspecting the agent's data and traces. A coding agent does not simply map one input to one output. It fetches code, reasons about what to write, edits files, and checks the result across several steps. Teams therefore need to inspect the full sequence of actions to find the errors that a prompt update should address. A single successful or failed final answer does not contain enough detail for this analysis.
Khan shows results from another dataset where adding context did not produce a steady improvement. Performance went up, then down, then up again as more information was included. He compares this with giving a person too much information before a task. The useful amount of context depends on the task and the agent, so prompt optimization must tune context size rather than always adding more instructions or examples.
Prompt optimization can narrow the gap between different models
Khan reports an experiment using GPT-4.1 and Claude Sonnet 4.5 on the same task. After system prompt optimization, the older GPT-4.1 model reached performance comparable to Sonnet 4.5. He presents this as early research and says the code and datasets for the system prompt optimization work are open source. Teams can inspect the experiments and apply the method to their own agents.
"The most common things that we see breaking are things like planning steps, lacking planning, missing tools, not being able to give agents guidance on which tools to use, or just missing context."Aman Khan02:15
Who should watch
You are building an agent and need a practical way to improve its behavior after deployment.
Your coding agent has traces, evaluations, or failure examples, but you do not yet turn them into updated system instructions.
You are deciding whether a stronger model is necessary when prompt and context optimization may improve a smaller or older model.
