Podcast

What is Data / ML Like on League?

Ian Schweer, Riot GamesEpisode 132 · 1:00:39 · Nov 2022 · 478 viewsHosted by Skylar Payne
Thumbnail for What is Data / ML Like on League? Watch on YouTube
TL;DR
  1. 1

    League produces a complete end-of-game JSON record, which lets Riot analyze a match without reconstructing it from many services and databases.

  2. 2

    Riot runs inference in several places, including inside the game, at service time, and in data-processing jobs, because real-time inference inside an older game engine has strict performance and security limits.

  3. 3

    Ian's team treats production models as software products, with Python packages, tests, linting, typing, monitoring, and regular reviews instead of leaving them in notebooks.

Summary

Ian Schweer describes the data and ML work behind League of Legends at Riot Games. League's game servers produce a compact, referentially complete record at the end of each match, which gives data teams a full account of what happened without rebuilding the story from separate systems. Riot stores derived data in Hive, S3, and a RocksDB-based feature store, and uses models in game binaries, services, and data-processing jobs. Ian explains why live inference inside the game is difficult: frame-time limits, network security, and a nine- or ten-year-old C++ codebase all constrain the design. He also compares software engineering with data work, where the quality of the data and model often matters more than the elegance of the code. His team reduces production risk by moving work out of notebooks and treating models as maintainable products. Ian is candid that the right balance between code quality and delivery is still unsettled.

Key ideas
06:06

Riot's mental-health support grew from an awareness of how engineering culture can harm people

Ian connects his interest in engineering mental health to growing up around macho expectations and feeling pressure to prove himself through technical achievement. University, therapy, and learning to enjoy his career without constantly trying to one-up other people helped him leave that state behind. At Riot, he found a company that openly recognizes how bro-y and burnout-driven cultures can affect employees. He says Riot offers extra support for therapy and has been the most supportive workplace he has experienced for mental well-being. He did not join for that reason, though. At first, the attraction was simpler: Riot seemed exciting and had free coffee.

11:25

Ian's career moved from customer-facing software work into data infrastructure and ML operations

Ian started at Adobe through a co-op arrangement, working part time during school and full summers as an intern. His early work involved building video technology solutions with Adobe customers. Around 2018, he moved into data engineering and worked on automation, metrics-layer problems, data observability, orchestration, and a custom stack built on Zookeeper and Postgres. At DoorDash, he handled data privacy work, Cassandra replication, Flink applications over Debezium streams, containers, and operations work. He later joined Riot in game build and SRE before moving to data. Earlier ML infrastructure work at DoorDash included EMR, Airflow, live inference, and keeping Snowflake out of the request path.

18:33

Problems in ML operations often resemble older problems from SRE and data systems

Ian says his main lesson from SRE is that many ML operations and data-engineering problems have the same shape as problems solved in other fields. Point-in-time calculations in feature stores resemble slowly changing dimensions. Snapshotting and SQL macros can recreate historical views of data, while configuration systems also need to recover what existed before an incident and track changes over time. He has also learned that large-company tools often depend on many internal systems that users do not see. DoorDash had its own Flink, Docker, and container deployment infrastructure. This background changes how Ian designs systems at Riot, where he sees older solutions and historical constraints embedded throughout the stack.

29:03

League gets a complete match record that avoids reconstructing the game from scattered sources

League uses Java services and C++ game systems to handle login, the client, purchases, matchmaking, team creation, game play, and end-of-game summaries. Riot collects data through database scrapes and service telemetry sent to Kafka. The unusual source is the end-of-game record. Because the game server makes authoritative decisions, it logs the match into a compact JSON file that fully describes what happened. Ian contrasts this with DoorDash, where teams often had to query several services and databases to rebuild the story of a delivery. Riot can use the complete game record for player- and game-level aggregation without first hitting the data warehouse.

30:32

Riot places features and inference in different systems depending on the use case

After ingestion, Riot sends data into Hive, including an on-premise warehouse that is moving toward AWS Glue for management. Feature and decision-science jobs may write results back to Hive, package them as tar.gz files in S3 for services to consume, or place them in a RocksDB implementation on top of S3. The latter gives some transformations stronger isolation and idempotence properties. That store supports models concerned with feeding in-game, intentional poor play, and banning decisions. Riot also serializes some models into the game code so a later patch carries a new version. One example infers which lane a player is using from play activity, since the player does not explicitly provide that information.

34:45

Real-time inference inside the game is constrained by frame time, security, and legacy C++

Ian explains that updating a decision tree frequently would be useful, but running the computation during a game can affect rendering. Spawning a thread may consume too much frame time and cause player hitches, so the inference code must be highly optimized and fit within a fixed time budget. Network security adds another obstacle. Riot protects a large online game against attacks, and game servers may not even be allowed to make TCP connections. The game also has substantial historical inertia. League's roughly nine- or ten-year-old C++ codebase includes several custom unique-pointer implementations, which makes it difficult for an ML engineer to add a safe module. These limits are why Riot does not simply update every model in real time.

39:31

Data work puts more weight on the data and model than on the elegance of the code

Ian says a software engineer moving into data eventually sees that the data matters more than the code in many situations. People may not care how inelegant PySpark code is if the analysis is clean or the model has strong accuracy and provides player value. Product software has a different expectation because the product is assumed to remain in place, so its code must be maintainable and treated almost as gospel. Ian expects ML engineering to adopt more software practices, such as unit tests, mocked training data, and model observability. He still expects ML engineers in the future to spend more time discussing their data than their code.

48:48

Riot reduces production risk by moving models out of notebooks and treating them as products

Ian's team is trying to reduce the number of notebooks that reach production. Jupyter notebooks work well for analysis, but a PySpark job that will be called by a service should become a Python package, Java service, or similar production artifact. The item recommender is the team's example. It recommends League items using information about the player's champion, opponents, teammates, and game state. Riot built it with modules, unit tests, linting, typing, and standards for code use, and Ian says players like it while the team can maintain and observe it. The team also wants regular model reviews, since a model can degrade or encounter shifted data long before anyone notices that it has failed.

"Because of how online video games work, everything gets logged out in the game server into this kind of nicely compact JSON file."Ian Schweer29:47
Who should watch
  • You work on data or ML systems for an online product and want to see how game data changes the usual ingestion and inference assumptions.
  • Your team is deciding whether notebooks, data-science code, and models are ready for production use.
  • You are dealing with real-time constraints, network restrictions, or a long-lived C++ product and need examples of the tradeoffs Ian makes at Riot.