Watch on YouTubeAI agent development involves many interacting choices across prompts, models, retrieval, temperatures, examples, and error correction, so changing one setting can improve one measure while damaging another.
A small configuration space can grow from 12 combinations to 1,200 or more, while some combinations are invalid or make little sense together.
Traigent uses guided exploration and multi-objective evaluation to spend an experimentation budget on promising configurations instead of testing every combination.
Nimrod argues that improving an AI agent in production is usually handled through biased, manual trial and error. Engineers change a prompt or model after seeing a failure, then check a few similar examples. This can fix one case while damaging accuracy, cost, or response time elsewhere. The number of possible configurations also grows quickly as teams add prompt styles, retrieval settings, examples, temperatures, and models. Nimrod compares this problem with A/B testing and hyperparameter tuning, which he says do not fit the cost, dependencies, and mixed parameter types of agent systems. Traigent is designed to evaluate alternatives across several objectives and use machine learning methods to explore promising configurations within a fixed budget. The talk includes an example where replacing GPT-4 with GPT-3.5 kept accuracy similar and improved performance by about 5 percent. Nimrod also describes an SDK with a decorator, customizable evaluators, and configurable sampling strategies.
An agent can vary across the model, temperature, retrieval method, prompting strategy, prompt style, role, examples, and error-correction steps. Nimrod says accuracy is often several measures rather than one score, while runtime and cost matter at the same time. A change that helps one measure can hurt another. He wants engineers to compare alternatives across these objectives instead of choosing settings from intuition.
Nimrod describes a common workflow where an engineer sees a few production failures, changes the prompt, and checks the same example or a few similar ones. This creates a biased evaluation. The team may fix one area while breaking other areas or hurting the overall task. Measures that conflict with one another then fluctuate as engineers test one combination at a time.
Trying every combination with nested loops quickly becomes impractical. Three prompts and two temperatures already create 12 combinations. Adding more choices can produce 60 or 1,200 options, before accounting for further settings. Some choices also depend on one another. Nimrod gives zero-shot selection and top-k as an example of a combination that does not make sense, so an optimizer must handle invalid configurations as well as the size of the search space.
Nimrod says many randomly chosen configurations are inefficient because another configuration can provide more accuracy for the same money. In the text-to-SQL system he worked on, the team found after months that GPT-3.5 could replace GPT-4 with the same accuracy and about 5 percent better performance. He says this change moved their result from number 16 to number 5 in an international competition.
A/B testing compares two versions at a time, so it does not help engineers choose effectively across a large configuration space. Nimrod also says standard hyperparameter tuning requires substantial setup, an objective function, and careful constraint definition. It is awkward for parameters such as prompts and discrete choices, and evaluating thousands of configurations can become too expensive when each evaluation consumes paid model calls.
Traigent is intended to let an engineer define an objective and an evaluation set with limited setup. Nimrod states the central rule as, "If we can evaluate it, we can optimize it." A decorator can wrap a function containing LLM code, after which the engineer calls an optimize function. The system can return the best configuration or score, along with detailed statistics.
Traigent does not explore the full combination space. It uses machine learning techniques to identify promising strategies, avoid wasting evaluation examples and model calls, prioritize valid regions, and sample configurations efficiently. Nimrod says the aim is to make a fixed budget produce more useful exploration, with the exact improvement depending on the case.
In the question period, Nimrod says the SDK will support popular techniques such as grid search and Bayesian optimization. Users will be able to customize the system or extend the library with their own algorithms. The Traigent team is also developing algorithms specifically for agent optimization and is studying whether they can provide guarantees about the most promising experiment for the money spent.
"We're actually using different machine learning techniques in order to identify the most promising strategies as fast as we can."17:48