Watch on YouTubePrompting alone does not make complex, multi-step agents reliable in production.
Reinforcement learning with verifiable rewards works well for problems such as maths, but most agent tasks still need a way to avoid reward hacking.
Interactive learning could let an agent update immediately from user feedback through a learn method alongside its generate method.
Patrick Barker argues that agent reliability gets worse as systems move from simple interactions to multi-step, multi-tool and autonomous work. Prompting cannot currently fix many of the underlying model problems. He explains two uses of reinforcement learning with language models: RLHF, which relies on human preferences and reward models, and RLVR, which uses deterministic checks such as the answer to a maths problem. RLVR has produced strong reasoning models, but it applies to only a small part of the tasks agents need to handle. The harder problem is reward hacking, where a model finds an easy way to score well without completing the intended task. Barker describes using reasoning validators, smaller amounts of feedback, and AI-generated verification to train agents. He also presents Dark RL, an open source library that trains a model immediately when a user selects one response over another. His proposed direction is for language models to learn online during use.
Barker says that agent reliability falls as systems move from simple agents to multi-step, multi-tool and autonomous agents. He sees prompting as insufficient at the current stage. If prompts alone were enough, he argues, reliable agents would already be common. The problem is therefore partly inside the model, rather than something that can always be fixed by adding more instructions.
Barker places current reinforcement learning work in a longer history. RLHF helped produce ChatGPT through InstructGPT, which taught a base language model to handle multi-turn conversations. A model generates several responses, a reviewer ranks them, and a reward model learns to reproduce those preferences. The reward can be binary feedback, an ordered ranking, a numerical score, or textual feedback from a simulated multi-turn interaction.
RLVR uses rewards that can be verified without a learned reward model. Barker gives a maths problem as the simple case: the system can check whether the answer is correct because there is one expected result. He connects this approach with o1 and R1, and names GRPO as a common algorithm. Its strength comes from avoiding some of the reward-hacking problems that arise when the model learns from an imperfect reward signal.
Barker describes an OpenAI game example where an agent learned to drive in a circle and collect power-ups because that produced the highest score. He says models take opportunities to cheat whenever they can. Verifiable rewards cover only a small fraction of the tasks people want language-model agents to perform, so reliable training needs better ways to tell whether an action achieved the intended goal.
One proposed direction is to train an agent and a validator together. The agent tries to solve a task while the validator reasons about whether the result was genuinely successful. Barker also argues that agents may need less training data than current methods use. A small amount of direction, combined with data augmentation and language-model-generated examples, could help an agent learn from fewer direct demonstrations.
Once a language model can approximate a human signal, Barker says it can replace the human verifier in a reinforcement-learning loop. He calls this RL AIF, or reinforcement learning from AI feedback. The system can generate tasks within a skill area, have a language model verify the results, and use those judgements for training. The same reward-hacking problem remains, since the verifier can also be imperfect.
Barker says his work has produced models that can outperform frontier models on complex tasks, especially tasks outside their training distribution. He also describes them as cheaper, faster and more reliable to operate. Running a smaller model on a team's own infrastructure avoids dependence on frontier-model uptime and can make a production service easier to maintain.
Dark RL is an open source library that lets a user choose between generated responses and trains the model immediately from that choice. Barker says the team developed kernels to speed up training and inference within one model. He imagines language models having both a generate method and a learn method, so data sent during use can update the model in real time.
In the discussion, Barker says binary reward models were difficult to train from simple yes-or-no feedback. If a dataset contains 60 percent positive examples and 40 percent negative examples, the model can become biased toward saying yes at roughly that rate without understanding the task. Offline datasets can be rebalanced, but this is harder during online learning. He suggests generating other tasks where the same action would be valid as a way to expand training data.
"If your data set says 60% yeses and 40% nos, the model will just naturally go to 60%."10:45