# I Let An AI Play Pokémon! - Claude plays Pokémon Creator David Hershey, Anthropic | MLOps Podcast | Episode 297 | 46:59 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=nRHeGJwVP18 Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/i-let-an-ai-play-pokemon-claude-plays-pokemon-creator Published: 2025-03-05 Tags: agents, evals, fine-tuning, prompt-engineering ## TL;DR - David Hershey built Claude Plays Pokémon as a side project to learn how agents handle long-running tasks and changing environments. - Prompting is usually worth pushing much further before a team considers fine-tuning, because fine-tuning takes time and can make models worse. - Agent adoption will accelerate when models become reliable enough that users do not need to inspect and repair each step of the work. ## Summary David Hershey explains how he built Claude Plays Pokémon to explore agents in a setting that requires thousands of decisions over many hours. Claude presses game controls, receives screenshots and game information, and updates a knowledge base that summarizes earlier actions. The project became a useful way to compare models because beating a gym leader is an observable milestone, unlike a short prompt-and-response benchmark. David argues that most teams should spend more time improving prompts before fine-tuning. Fine-tuning can help with unusual data formats, simple classification, or tasks where the final few percent of performance matters, but it is often a costly research project. He also describes a threshold for useful agents: once they work reliably enough that users do not have to investigate every failure, a workflow can suddenly become valuable. Managed model platforms reduce the infrastructure burden and let more engineers build AI features, although blank chatbots still leave too much cognitive load on users. ## Key ideas ### Claude Plays Pokémon became a long-horizon test for model behavior [02:00](https://www.youtube.com/watch?v=nRHeGJwVP18&t=120s) David started the project in June while working with Anthropic customers on agents. He wanted a playground where he could build agents himself, and Pokémon was appealing because it made the work enjoyable. Earlier models could leave the house or obtain a starter Pokémon, but they struggled to make sustained progress. Newer models reached a point where Claude could move through the game and beat gym leaders. David uses those milestones to examine whether a model can stay coherent across hours of actions, absorb new information, try different approaches, and update its behavior. ### The agent uses a small set of tools and compresses its history [09:06](https://www.youtube.com/watch?v=nRHeGJwVP18&t=546s) David says he gradually removed complexity from the system. Claude receives a short prompt about playing Pokémon and can press buttons, maintain a knowledge base, and use a Navigator that points to a location on the screen. After a button press, it gets a screenshot plus limited information read directly from the game, such as its current location. Since thousands of screenshots cannot fit in the context window, the system summarizes groups of 30 actions. The knowledge base lets Claude preserve longer-term information, such as the fact that electric attacks are effective against water. ### Prompting usually deserves more work before fine-tuning [12:45](https://www.youtube.com/watch?v=nRHeGJwVP18&t=765s) David says he often advises teams to push prompting much further before fine-tuning. Prompting can have a short iteration cycle, while changing a trained model requires data, evaluation, and difficult engineering work. Retrieval and other ways of adding information to the prompt can also be easier than fine-tuning. He has seen teams assume they have reached the limit of a prompt when they have not yet tested enough variations. For most use cases, he thinks fine-tuning should not even be considered until the team is confident it has extracted the most from prompting. ### Fine-tuning makes sense for narrow data and output problems [15:46](https://www.youtube.com/watch?v=nRHeGJwVP18&t=946s) David identifies cases where fine-tuning can be reasonable. A model may need to follow a particular output format, understand a document format it has rarely seen, or perform a simple classification task on specialized data. Training a smaller model to imitate a larger one can also reduce inference costs, but David calls this a possible trap because cheaper and more capable models may arrive before the fine-tuning work pays off. Teams must account for the time spent building data, testing the result, and avoiding damage to other model capabilities. ### Improving an already strong model is a research project [20:20](https://www.youtube.com/watch?v=nRHeGJwVP18&t=1220s) David spends much of his time on the hardest category of fine-tuning, taking a good model and making it better at a particular task. He says most teams can get good enough performance from an off-the-shelf model, while the final improvement may require research into data, methods, and tooling. Fine-tuning data does not automatically improve a model. David has watched models get worse more often than better on average data sets, so he treats the work as an uncertain and expensive investigation rather than a routine engineering step. ### Agents become useful after they cross a reliability threshold [24:03](https://www.youtube.com/watch?v=nRHeGJwVP18&t=1443s) David expects agents to appear in areas where a model suddenly becomes good enough to complete a workflow. Coding agents crossed that threshold when models could produce useful code without constant inspection and repair. The same pattern could apply to legal work, accounting workflows, and spreadsheet manipulation. A single small capability can change the experience, such as clicking the correct spreadsheet cell. If an agent fails often enough that the user must reconstruct what happened and finish the task themselves, much of the value disappears. ### Model updates can make old prompts worse by leaving them in place [33:14](https://www.youtube.com/watch?v=nRHeGJwVP18&t=1994s) David says new models often make his Pokémon prompt simpler. Older versions needed instructions that acted like Band-Aids for recurring mistakes. When a new model handles those situations on its own, keeping the old instructions can hurt performance. Benchmark changes may not explain the practical difference. A model can appear only slightly better in conversation while becoming much better at one small action, such as selecting the correct spreadsheet cell. That small improvement can make an entire workflow possible. ### Managed AI platforms lower the barrier for software engineers [40:57](https://www.youtube.com/watch?v=nRHeGJwVP18&t=2457s) David sees more people building with language models, including engineers who are not machine learning specialists. They can expose organizational tools through MCP and assemble workflows without building a large platform. He thinks machine learning skills still matter, especially creating evaluation data, tracking experiments, and iterating carefully. Managed services also remove much of the work of hosting GPUs, routing requests, and scaling inference. Paying for API calls is much simpler than building and operating the infrastructure directly, although users still need better guidance than a blank chatbot provides. ## Notable quotes - David Hershey: "Pokemon it's like I don't know you beat a gym leader like that's a thing that happens after 10 hours right." (16:16) - David Hershey: "Until you are really confident that you've gotten the most out of a prompt I think it's almost never a good idea to even consider fine-tuning for most use cases." (14:53) - David Hershey: "As soon as it crosses that threshold of more often than not you don't need to check it, then it's like, 'Oh, this is the coolest thing I've ever seen.'" (31:34) - David Hershey: "The people who let go and embrace the free infrastructure are getting to some extent the free management they're getting, I think, able to make a ton of progress on this stuff." (45:52) ## Tools & references mentioned - Anthropic - Claude - Claude Plays Pokémon - Pokémon - Karpathy - TensorFlow - Voyager - Claude Code - Codex - Cursor - Model Context Protocol - MCP ## Who should watch - You are deciding whether to improve prompts, add retrieval, or fine-tune a model and need a practical way to judge the trade-offs. - You are building an agent that works in demos but still needs a person to inspect every step and repair failures. - You are moving from traditional machine learning infrastructure to managed model APIs and want to understand which engineering skills still carry over. ## Related talks - [How to Make AI Agents that ACTUALLY WORK](https://mlopstalks.com/talks/how-to-make-ai-agents-that-actually-work) (Patrick Marlo, Google, 25:43) - [Meta-Prompting: The Hack That's Changing Production AI](https://mlopstalks.com/talks/meta-prompting-the-hack-thats-changing-production-ai) (Aman Khan, Arize, 19:28) - [The Claude Code Story: How Two Guys Changed AI Forever](https://mlopstalks.com/talks/the-claude-code-story-how-two-guys-changed-ai-forever) (Siddharth Bidasaria, Anthropic, 50:29) - [How Reinforcement Learning Can Improve Your Agent](https://mlopstalks.com/talks/how-reinforcement-learning-can-improve-your-agent) (Patrick Barker, 12:25) - [We're All Finetuning Incorrectly](https://mlopstalks.com/talks/were-all-finetuning-incorrectly) (Tanmay Chopra, Emissary, 1:00:31)