# How AI Will Change Gaming Forever

Pietro Gagliano, Transitional Forms | MLOps Community | 26:25

Source: https://www.youtube.com/watch?v=5HuMYLpzDkk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-ai-will-change-gaming-forever
Published: 2024-12-02
Tags: agents, multi-agent, multimodal

## TL;DR
- Pietro Gagliano uses autonomous systems in film, music, games, and television to make entertainment respond to characters and audiences.
- Little Learning Machines lets players train neural networks inside a game through positive and negative rewards.
- Long-running character agents can produce flat stories, so Transitional Forms uses a director agent to introduce conflict and manage the narrative.

## Summary
Pietro Gagliano presents projects from Transitional Forms that treat AI as an active part of entertainment production. He describes Agents, a dynamic film in which reinforcement-learning characters make decisions inside a virtual world; Project Malachi, which generates music from curves for emotion, energy, complexity, and depth; Little Learning Machines, a game where players train neural networks through rewards; and Robots Make TV, which uses GPT-2 and Unity to generate interactive television. He then introduces Simia, a planned multimodal, multi-agent platform for creating and remixing content from a smartphone. Gagliano is honest about the limits of autonomous characters. When agents follow their backstories too closely, stories become flat because nobody acts irrationally, gets frustrated, or creates conflict. His current approach adds a director agent that controls the story and introduces tension. The talk connects machine-learning experiments with questions about agency, authorship, empathy, and narrative design.

## Key ideas
### Transitional Forms treats autonomous systems as part of entertainment production
[04:51](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=291s)
Gagliano defines an AI agent broadly as an autonomous system, not only as a large language model. Transitional Forms has used reinforcement learning and heuristic AI in film, music Transformers in real-time music production, reinforcement learning in video games, and language models in television. He says the company is also developing a multimodal, multi-agent system that can run from a smartphone. His focus is the way creative machine intelligence could change entertainment itself, rather than using AI only behind the scenes.

### Agents made a film whose characters learned how to survive
[06:00](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=360s)
In Agents, Transitional Forms created artificially intelligent characters inside a virtual film. The characters could make decisions, and those decisions changed what happened, so the film could be different each time. The team built a training world with reinforcement learning. Gravity went straight down, and the characters had to cooperate to survive on top of a planet. They repeatedly trained, failed, and died while learning how to understand and act in the environment. Gagliano says the project proved the concept of a dynamic film, although audiences often thought it was a game.

### Project Malachi mapped story curves to live-generated music
[10:40](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=640s)
Project Malachi explored whether real-time music Transformers could create music from the shape of a story. Using Google's Magenta, the team built an interface where curves controlled musical properties. The first curve represented emotional valence, with high values for positive emotion and low values for negative emotion. They later added energy, complexity, and depth. Gagliano says the system produced music in near real time, which mattered for experiences whose events unfold while the audience is present. The team shelved the concept for the time being, though he expects to revisit it.

### Little Learning Machines puts neural-network training inside a game
[12:26](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=746s)
After seeing how difficult it was to build the training environment for Agents, the team asked whether a game could let players train neural networks themselves. Little Learning Machines uses positive and negative rewards, compared by Gagliano to training a puppy. Players express love or fear toward actions in the world, such as petting a dog, cutting down trees, falling from an edge, or being hit by fire. He says the game achieved its goal: anyone who can play a video game can train an agent inside it. His young son trained his first neural network through the game.

### Robots Make TV used GPT-2 to generate interactive shows
[16:13](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=973s)
Robots Make TV connected GPT-2 to the Unity game engine and added rules for the fictional world. The system generated dialogue and other content, with users able to seed an exchange and let the language model produce possible next lines. The team built the system in modules, so environments, characters, and backstories could be swapped. Users could also trigger effects such as throwing food, setting fires, or changing gravity. Gagliano says the hardest part was making language generation work in real time. The project streamed continuously for 14 days.

### Simia lets people generate and remix content from a phone
[19:27](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=1167s)
Gagliano introduces Simia as a new product based on the experiments in Robots Make TV. He describes it as a multimodal, multi-agent framework for generating and remixing content, with sharing built into the experience. The presentation shows the product as a way for content creators and audiences to generate endless variations, including deliberately silly material such as fart jokes. He asks viewers to sign up for early access and says the system is designed to run through the smartphone in their pocket.

### Character agents alone produce stories without enough conflict
[21:56](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=1316s)
In response to a question about persistent agents, Gagliano explains that Simia and Robots Make TV give characters their own backstories. The team tested making each character an agent that acted according to that history. The result was interesting but narratively weak. Characters avoided actions that were out of character, flawed, frustrated, or confused, so the stories became flat. Gagliano's current solution is a director agent that controls the story and introduces tension between characters. Character consistency is useful, but it does not automatically create drama.

### Creative interaction can help people understand machine behavior
[24:31](https://www.youtube.com/watch?v=5HuMYLpzDkk&t=1471s)
Gagliano says Transitional Forms wants humans to understand what it means to be a machine and machines what it means to be human. He argues that interactive art gives people a different way to observe machine behavior than charts and graphs do. During Agents, researchers could see characters take action inside the film world instead of only checking whether they were learning efficiently. He believes real-time creative systems can support empathy with machines because the audience engages directly with the behavior.

## Notable quotes
- "It is my goal today to show you as many examples of agents in production output by one small team." (01:09)
- "What if we could build a game that allowed people to train neural networks right within the game itself?" (13:14)
- "What we're finding now is in our new multi-agent system is having a director agent that is controlling the story and saying conflict here." (22:58)
- "The artfulness and the real-time capabilities of it really help you to engage with the content because it's literally engaging you." (25:01)

## Tools & references mentioned
- Transitional Forms
- Secret Location
- Entertainment One
- Canada Media Fund
- Ontario Creates
- National Film Board of Canada
- Venice Film Festival
- MIT Technology Review
- Google Brain
- Google DeepMind
- Google Magenta
- Agents
- Project Malachi
- Little Learning Machines
- Robots Make TV
- Simia
- GPT-2
- Unity
- Steam
- Kurt Vonnegut

## Who should watch
- You are building interactive entertainment and want concrete examples of reinforcement learning, language models, or multi-agent systems used in production.
- You want to understand how AI-generated characters behave over time and why consistent personalities do not automatically produce good stories.
- You are interested in games that teach machine-learning concepts through direct interaction rather than through technical interfaces.

## Related talks

- [The Coming Revolution of AI Agents](https://mlopstalks.com/talks/the-coming-revolution-of-ai-agents) (Jazmia Henry, Iso AI & Rogerio Bonatti, Microsoft & Julia Kroll, Deepgram & Joshua Alphonse, PremAI, 28:59)
- [Creative AI: Using ML to Create Art, Music, and Jokes](https://mlopstalks.com/talks/creative-ai-using-ml-to-create-art-music-and-jokes) (Suyash Joshi, MLOps Community, 1:12:57)
- [Beyond the Matrix: AI and the Future of Human Creativity](https://mlopstalks.com/talks/beyond-the-matrix-ai-and-the-future-of-human-creativity) (Fausto Albers, AI Builders Club, 55:09)
- [A New Way of Building with AI](https://mlopstalks.com/talks/a-new-way-of-building-with-ai) (Jiquan Ngiam, Lutra AI, 1:04:46)
- [How Agentic Workflows Will Change Everything](https://mlopstalks.com/talks/how-agentic-workflows-will-change-everything) (Raj Rikhy, Microsoft, 49:13)