# How to Optimize AI Agents in Production

 | Agents in Production 2025 | 27:06

Source: https://www.youtube.com/watch?v=--le-yBdVPk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-optimize-ai-agents-in-production
Published: 2025-09-27
Tags: agents, cost, evals, prompt-engineering

## TL;DR
- AI agent development involves many interacting choices across prompts, models, retrieval, temperatures, examples, and error correction, so changing one setting can improve one measure while damaging another.
- A small configuration space can grow from 12 combinations to 1,200 or more, while some combinations are invalid or make little sense together.
- Traigent uses guided exploration and multi-objective evaluation to spend an experimentation budget on promising configurations instead of testing every combination.

## Summary
Nimrod argues that improving an AI agent in production is usually handled through biased, manual trial and error. Engineers change a prompt or model after seeing a failure, then check a few similar examples. This can fix one case while damaging accuracy, cost, or response time elsewhere. The number of possible configurations also grows quickly as teams add prompt styles, retrieval settings, examples, temperatures, and models. Nimrod compares this problem with A/B testing and hyperparameter tuning, which he says do not fit the cost, dependencies, and mixed parameter types of agent systems. Traigent is designed to evaluate alternatives across several objectives and use machine learning methods to explore promising configurations within a fixed budget. The talk includes an example where replacing GPT-4 with GPT-3.5 kept accuracy similar and improved performance by about 5 percent. Nimrod also describes an SDK with a decorator, customizable evaluators, and configurable sampling strategies.

## Key ideas
### Agent configuration has many interacting choices
[01:34](https://www.youtube.com/watch?v=--le-yBdVPk&t=94s)
An agent can vary across the model, temperature, retrieval method, prompting strategy, prompt style, role, examples, and error-correction steps. Nimrod says accuracy is often several measures rather than one score, while runtime and cost matter at the same time. A change that helps one measure can hurt another. He wants engineers to compare alternatives across these objectives instead of choosing settings from intuition.

### Manual fixes can improve one example while harming the system
[04:00](https://www.youtube.com/watch?v=--le-yBdVPk&t=240s)
Nimrod describes a common workflow where an engineer sees a few production failures, changes the prompt, and checks the same example or a few similar ones. This creates a biased evaluation. The team may fix one area while breaking other areas or hurting the overall task. Measures that conflict with one another then fluctuate as engineers test one combination at a time.

### The configuration space grows exponentially
[05:42](https://www.youtube.com/watch?v=--le-yBdVPk&t=342s)
Trying every combination with nested loops quickly becomes impractical. Three prompts and two temperatures already create 12 combinations. Adding more choices can produce 60 or 1,200 options, before accounting for further settings. Some choices also depend on one another. Nimrod gives zero-shot selection and top-k as an example of a combination that does not make sense, so an optimizer must handle invalid configurations as well as the size of the search space.

### Better models do not automatically give better value
[11:23](https://www.youtube.com/watch?v=--le-yBdVPk&t=683s)
Nimrod says many randomly chosen configurations are inefficient because another configuration can provide more accuracy for the same money. In the text-to-SQL system he worked on, the team found after months that GPT-3.5 could replace GPT-4 with the same accuracy and about 5 percent better performance. He says this change moved their result from number 16 to number 5 in an international competition.

### A/B testing and standard hyperparameter tuning fit poorly
[13:00](https://www.youtube.com/watch?v=--le-yBdVPk&t=780s)
A/B testing compares two versions at a time, so it does not help engineers choose effectively across a large configuration space. Nimrod also says standard hyperparameter tuning requires substantial setup, an objective function, and careful constraint definition. It is awkward for parameters such as prompts and discrete choices, and evaluating thousands of configurations can become too expensive when each evaluation consumes paid model calls.

### Traigent optimizes from an evaluation set
[15:17](https://www.youtube.com/watch?v=--le-yBdVPk&t=917s)
Traigent is intended to let an engineer define an objective and an evaluation set with limited setup. Nimrod states the central rule as, "If we can evaluate it, we can optimize it." A decorator can wrap a function containing LLM code, after which the engineer calls an optimize function. The system can return the best configuration or score, along with detailed statistics.

### The system spends the budget on promising areas
[17:36](https://www.youtube.com/watch?v=--le-yBdVPk&t=1056s)
Traigent does not explore the full combination space. It uses machine learning techniques to identify promising strategies, avoid wasting evaluation examples and model calls, prioritize valid regions, and sample configurations efficiently. Nimrod says the aim is to make a fixed budget produce more useful exploration, with the exact improvement depending on the case.

### The planned SDK combines standard and custom optimization
[21:53](https://www.youtube.com/watch?v=--le-yBdVPk&t=1313s)
In the question period, Nimrod says the SDK will support popular techniques such as grid search and Bayesian optimization. Users will be able to customize the system or extend the library with their own algorithms. The Traigent team is also developing algorithms specifically for agent optimization and is studying whether they can provide guarantees about the most promising experiment for the money spent.

## Notable quotes
- "The implications are that we often time fix a single area and break other areas or we just hurt the overall task that we had." (04:48)
- "If we can evaluate it, we can optimize it." (15:41)
- "We're actually using different machine learning techniques in order to identify the most promising strategies as fast as we can." (17:48)
- "We want to prove that for the money that you spent you get the most promising experiment that can be done." (22:22)

## Tools & references mentioned
- Traigent
- IBM Research
- Accenture Labs
- GPT-4
- GPT-3.5
- OpenAI SDK
- Spider competition
- University of Ottawa
- THA University
- Bayesian optimization
- grid search

## Who should watch
- You are maintaining an AI agent and changes to prompts, models, or retrieval settings keep producing mixed results across accuracy, cost, and latency.
- Your team has an evaluation set but lacks a practical way to compare many agent configurations within a limited model-call budget.
- You are building optimization tooling and want to think about invalid combinations, dependent parameters, prompt optimization, and guarantees about search efficiency.

## Related talks

- [How to Make AI Agents that ACTUALLY WORK](https://mlopstalks.com/talks/how-to-make-ai-agents-that-actually-work) (Patrick Marlo, Google, 25:43)
- [Meta-Prompting: The Hack That's Changing Production AI](https://mlopstalks.com/talks/meta-prompting-the-hack-thats-changing-production-ai) (Aman Khan, Arize, 19:28)
- [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [Advancing the Cost-Quality Frontier in Agentic AI](https://mlopstalks.com/talks/advancing-the-cost-quality-frontier-in-agentic-ai) (Krista Opsahl-Ong, Databricks, 12:45)