# What is Data / ML Like on League?

Ian Schweer, Riot Games | MLOps Coffee Sessions | Episode 132 | 1:00:39
Hosted by Skylar Payne

Source: https://www.youtube.com/watch?v=JjMc8TguPvQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/what-is-data-ml-like-on-league
Published: 2022-11-15
Tags: data-engineering, feature-engineering, feature-stores, model-serving, testing

## TL;DR
- League produces a complete end-of-game JSON record, which lets Riot analyze a match without reconstructing it from many services and databases.
- Riot runs inference in several places, including inside the game, at service time, and in data-processing jobs, because real-time inference inside an older game engine has strict performance and security limits.
- Ian's team treats production models as software products, with Python packages, tests, linting, typing, monitoring, and regular reviews instead of leaving them in notebooks.

## Summary
Ian Schweer describes the data and ML work behind League of Legends at Riot Games. League's game servers produce a compact, referentially complete record at the end of each match, which gives data teams a full account of what happened without rebuilding the story from separate systems. Riot stores derived data in Hive, S3, and a RocksDB-based feature store, and uses models in game binaries, services, and data-processing jobs. Ian explains why live inference inside the game is difficult: frame-time limits, network security, and a nine- or ten-year-old C++ codebase all constrain the design. He also compares software engineering with data work, where the quality of the data and model often matters more than the elegance of the code. His team reduces production risk by moving work out of notebooks and treating models as maintainable products. Ian is candid that the right balance between code quality and delivery is still unsettled.

## Key ideas
### Riot's mental-health support grew from an awareness of how engineering culture can harm people
[06:06](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=366s)
Ian connects his interest in engineering mental health to growing up around macho expectations and feeling pressure to prove himself through technical achievement. University, therapy, and learning to enjoy his career without constantly trying to one-up other people helped him leave that state behind. At Riot, he found a company that openly recognizes how bro-y and burnout-driven cultures can affect employees. He says Riot offers extra support for therapy and has been the most supportive workplace he has experienced for mental well-being. He did not join for that reason, though. At first, the attraction was simpler: Riot seemed exciting and had free coffee.

### Ian's career moved from customer-facing software work into data infrastructure and ML operations
[11:25](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=685s)
Ian started at Adobe through a co-op arrangement, working part time during school and full summers as an intern. His early work involved building video technology solutions with Adobe customers. Around 2018, he moved into data engineering and worked on automation, metrics-layer problems, data observability, orchestration, and a custom stack built on Zookeeper and Postgres. At DoorDash, he handled data privacy work, Cassandra replication, Flink applications over Debezium streams, containers, and operations work. He later joined Riot in game build and SRE before moving to data. Earlier ML infrastructure work at DoorDash included EMR, Airflow, live inference, and keeping Snowflake out of the request path.

### Problems in ML operations often resemble older problems from SRE and data systems
[18:33](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=1113s)
Ian says his main lesson from SRE is that many ML operations and data-engineering problems have the same shape as problems solved in other fields. Point-in-time calculations in feature stores resemble slowly changing dimensions. Snapshotting and SQL macros can recreate historical views of data, while configuration systems also need to recover what existed before an incident and track changes over time. He has also learned that large-company tools often depend on many internal systems that users do not see. DoorDash had its own Flink, Docker, and container deployment infrastructure. This background changes how Ian designs systems at Riot, where he sees older solutions and historical constraints embedded throughout the stack.

### League gets a complete match record that avoids reconstructing the game from scattered sources
[29:03](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=1743s)
League uses Java services and C++ game systems to handle login, the client, purchases, matchmaking, team creation, game play, and end-of-game summaries. Riot collects data through database scrapes and service telemetry sent to Kafka. The unusual source is the end-of-game record. Because the game server makes authoritative decisions, it logs the match into a compact JSON file that fully describes what happened. Ian contrasts this with DoorDash, where teams often had to query several services and databases to rebuild the story of a delivery. Riot can use the complete game record for player- and game-level aggregation without first hitting the data warehouse.

### Riot places features and inference in different systems depending on the use case
[30:32](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=1832s)
After ingestion, Riot sends data into Hive, including an on-premise warehouse that is moving toward AWS Glue for management. Feature and decision-science jobs may write results back to Hive, package them as tar.gz files in S3 for services to consume, or place them in a RocksDB implementation on top of S3. The latter gives some transformations stronger isolation and idempotence properties. That store supports models concerned with feeding in-game, intentional poor play, and banning decisions. Riot also serializes some models into the game code so a later patch carries a new version. One example infers which lane a player is using from play activity, since the player does not explicitly provide that information.

### Real-time inference inside the game is constrained by frame time, security, and legacy C++
[34:45](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=2085s)
Ian explains that updating a decision tree frequently would be useful, but running the computation during a game can affect rendering. Spawning a thread may consume too much frame time and cause player hitches, so the inference code must be highly optimized and fit within a fixed time budget. Network security adds another obstacle. Riot protects a large online game against attacks, and game servers may not even be allowed to make TCP connections. The game also has substantial historical inertia. League's roughly nine- or ten-year-old C++ codebase includes several custom unique-pointer implementations, which makes it difficult for an ML engineer to add a safe module. These limits are why Riot does not simply update every model in real time.

### Data work puts more weight on the data and model than on the elegance of the code
[39:31](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=2371s)
Ian says a software engineer moving into data eventually sees that the data matters more than the code in many situations. People may not care how inelegant PySpark code is if the analysis is clean or the model has strong accuracy and provides player value. Product software has a different expectation because the product is assumed to remain in place, so its code must be maintainable and treated almost as gospel. Ian expects ML engineering to adopt more software practices, such as unit tests, mocked training data, and model observability. He still expects ML engineers in the future to spend more time discussing their data than their code.

### Riot reduces production risk by moving models out of notebooks and treating them as products
[48:48](https://www.youtube.com/watch?v=JjMc8TguPvQ&t=2928s)
Ian's team is trying to reduce the number of notebooks that reach production. Jupyter notebooks work well for analysis, but a PySpark job that will be called by a service should become a Python package, Java service, or similar production artifact. The item recommender is the team's example. It recommends League items using information about the player's champion, opponents, teammates, and game state. Riot built it with modules, unit tests, linting, typing, and standards for code use, and Ian says players like it while the team can maintain and observe it. The team also wants regular model reviews, since a model can degrade or encounter shifted data long before anyone notices that it has failed.

## Notable quotes
- Ian Schweer: "Because of how online video games work, everything gets logged out in the game server into this kind of nicely compact JSON file." (29:47)
- Ian Schweer: "The best answer I've read for that is the fact that your code doesn't matter as much as your data matters." (39:56)
- Ian Schweer: "If you know this is going to production, don't write anything in a notebook." (53:00)
- Ian Schweer: "Prefer composition over inheritance." (57:04)

## Tools & references mentioned
- Riot Games
- League of Legends
- Teamfight Tactics
- Adobe
- DoorDash
- Zookeeper
- Postgres
- dbt
- Airflow
- Flink
- Debezium
- Cassandra
- Snowflake
- Hive
- AWS Glue
- Kafka
- S3
- RocksDB
- PySpark
- Python
- Java
- C++
- Jenkins
- Facebook
- Feather
- Shreya Shankar
- Google SRE
- LinkedIn
- Uber
- Laszlo
- Serge
- Mihael
- Legend of Zelda: Majora's Mask

## Who should watch
- You work on data or ML systems for an online product and want to see how game data changes the usual ingestion and inference assumptions.
- Your team is deciding whether notebooks, data-science code, and models are ready for production use.
- You are dealing with real-time constraints, network restrictions, or a long-lived C++ product and need examples of the tradeoffs Ian makes at Riot.

## Editor's note

Ian Schweer says Riot reduces production risk by moving models out of notebooks and treating them as products. ZenML lets teams write workflows as Python steps and records each run's steps, inputs, outputs, and code version. That gives a production job a traceable history instead of leaving its behavior tied to an exploratory notebook.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [ML in Production: A DS from Ubisoft Perspective](https://mlopstalks.com/talks/ml-in-production-a-ds-from-ubisoft-perspective) (Jean-Michel Daignan, Ubisoft, 49:26)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [Luigi in Production](https://mlopstalks.com/talks/luigi-in-production) (Luigi Patruno, ML in Production, 47:23)
- [Machine Learning at Atlassian](https://mlopstalks.com/talks/machine-learning-at-atlassian) (Geoff Sims, Atlassian, 58:23)