Evals replace subjective impressions with structured measurements of an agent's answers, reasoning, tool use, and behavior.
2
Real-world agent evaluation needs broad rubrics that cover accuracy, personality, interpretability, latency, task completion time, eagerness, consistency, and explainability.
3
High-quality graders can support reinforcement fine-tuning with relatively small datasets, including a chemistry agent that learned to predict chemical properties and justify its answers.
Summary
Robert explains how OpenAI evaluates collaborative agents with graders. He contrasts structured evals with judging models by personal impressions, then describes a progression from vibes-based testing to manual labels, automated evals, and automated CI/CD flywheels. Real-world tasks create problems because they are ambiguous, require outside information, and often involve multiple steps. Coding agents make these issues visible through long tasks, tool calls, and changing behavior. Robert recommends diverse rubrics that measure accuracy alongside personality, interpretability, latency, efficiency, eagerness, consistency, and explainability. He then describes good graders through the generator-verifier gap, where checking an answer is easier than producing one. These graders can support reinforcement fine-tuning, which explores multiple solutions and uses rewards to update a model. A chemistry-agent example shows the approach improving both property predictions and explanations.
Evals replace model judgment by vibes with structured measurements
Robert defines an eval as a structured, objective way to measure an agent or model. A grader evaluates the model's output, including its answer, reasoning, and justification. He contrasts this with trying a model and deciding that it feels good or bad, a judgment affected by the user's mood and recent experiences. He uses GPT-5 as an example: its launch may have felt underwhelming for ordinary email drafting, while harder tasks reveal larger performance differences. The point is that an evaluation must match the work people actually need agents to do, rather than relying on a single casual interaction.
Evaluation practice moves from manual labels toward automated flywheels
Robert describes four stages of evaluation maturity. The first is vibes-based testing in interfaces such as ChatGPT or Gemini. The second uses domain experts to label data, which is slow and expensive to repeat. The third adds well-defined automated evals while retaining manual checks before production. The fourth, which OpenAI calls an automated flywheel, puts comprehensive evals into CI/CD so agents can move toward production automatically. He also mentions inference tables that expose production metrics, such as guardrail behavior over time.
Public benchmarks miss ambiguity and multi-step work
The GDPval benchmark illustrates why real-world evaluation is difficult. A task might ask a manufacturing engineer to create an exploded view for an automotive assembly line, but the prompt cannot include every detail that a person would seek from colleagues or internal systems. Robert says current models perform strongly on prominent coding and mathematics competitions, yet remain below human experts across many occupations. These benchmarks are often one-shot, do not require tool calls or multi-step reasoning, and cover only part of the work people do.
Coding agents need rubrics that measure behavior beyond accuracy
Robert's central recommendation is to use diverse, comprehensive rubrics. Accuracy measures the percentage of tests that pass, but agents also need evaluation for personality, such as whether they ask useful follow-up questions. Interpretability can include a short explanation before each tool call, giving users an audit trail for actions such as large codebase edits. Other measures include time to first edit, time to complete a task, eagerness, consistency, and explainability. These dimensions can reveal behavior regressions after changes to tools or system prompts.
Robert uses Codex to show why accuracy alone can produce awkward model settings. Many systems offer low, medium, and high thinking modes because they optimize for accuracy without handling effort dynamically. In his example, Codex responds quickly to a small task such as creating a terminal command and spends much longer on a large software request. The evaluation goal is to measure this relationship between task difficulty, speed, and quality instead of forcing users to choose a thinking level every time.
Graders benefit from the gap between generating and verifying answers
Robert describes a good grader through the generator-verifier gap. Some tasks are harder to solve than to check. He gives Sudoku, mathematics, programming, and summarization as examples. Evaluating whether a document and its summary form a good pair is easier than producing a good summary repeatedly. This difference allows teams to spend more computation during inference and use model judges for some rubric-based checks. The same idea helps turn difficult agent outputs into measurable rewards.
Reinforcement fine-tuning explores solutions through graded rewards
Reinforcement fine-tuning explores a solution space instead of directly copying labeled outputs, as supervised fine-tuning does. For each task, the model can produce multiple parallel runs with different reasoning paths. A grader assigns different rewards to those outputs, and the rewards update the model. Robert says this can work with tens to hundreds of examples rather than thousands to tens of thousands, though it is not automatically better than supervised fine-tuning. A weak grader or inadequate task instructions can give every attempt a zero reward, leaving the model unable to learn.
A chemistry agent can learn both predictions and explanations
Robert ends with a chemistry-agent use case. The model receives molecules expressed in natural language and SMILES representations, then predicts chemical properties and justifies its answers. The rubric therefore measures more than prediction accuracy. It also checks whether the explanation is useful and whether the agent can generate other possibilities. In the demo, the model improves its chemical-property predictions and produces better justifications with a small number of examples. Robert attributes this progress to the strong graders developed for the task.
