# Chronon: Airbnb's Open-Source Data Platform

 | MLOps Community | 12:35

Source: https://www.youtube.com/watch?v=0wGLWWZuq6I
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/chronon-airbnbs-open-source-data-platform
Published: 2024-10-09
Tags: data-pipelines, feature-engineering, open-source, platform-teams

## TL;DR
- Chronon lets machine learning practitioners define computations over raw data while the platform generates the infrastructure needed for training and production serving.
- Pre-aggregating metrics and generating retrieval embeddings requires batch pipelines, streaming systems, storage, serving, orchestration, and monitoring, which makes small prototypes difficult to productionize.
- Chronon brings together infrastructure for different machine learning data tasks by making it easier to deploy and use existing open-source projects.

## Summary
Nikhil introduces Chronon, an open-source data platform built at Airbnb and Stripe for machine learning and related data use cases. He explains why a simple chatbot feature, such as calculating a merchant's recent order-issue rate or retrieving similar issues, can require batch processing, streaming, storage, serving, orchestration, and monitoring. This infrastructure burden often means that data scientists use less context than their models could support, or wait months for engineering teams to turn prototypes into production systems. Chronon changes the workflow so users define computations over raw data, while the platform generates the infrastructure and exposes endpoints for training, evaluation, iteration, and application use. Nikhil says Chronon supported growth from about 3,000 to nearly 30,000 features at Airbnb and was used for ranking, fraud, customer support, marketing, rules, and metrics. He positions Chronon alongside existing tools rather than as a replacement for each one.

## Key ideas
### Chronon supports machine learning and non-machine-learning data use cases
[00:32](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=32s)
Nikhil describes Chronon as a data platform built for machine learning at Airbnb and Stripe. At Airbnb, it is used for content and search ranking, account and payments fraud, customer support, and marketing technology. He also says the platform supports non-ML cases such as rule engines and metrics. This range matters because the same data computations can feed models, prompts, or ordinary application logic. Chronon was open sourced after being tested at Airbnb and Stripe, and Nikhil directs viewers to the project through a QR code and link.

### Chronon helped Airbnb expand its feature use while reducing dependence on systems engineers
[01:21](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=81s)
Nikhil says Airbnb grew from using about 3,000 features to nearly 30,000 in roughly three years. The number of models grew, and models also used more features. He attributes this to systems becoming faster to build, more scalable, and more performant out of the box. Previously, a data scientist could prototype in a few weeks or a month, then need a systems engineering team for many months or even a year to build the production ML system. Chronon aims to let practitioners work more independently.

### A simple prompt metric can require a large production system
[03:33](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=213s)
For an e-commerce chatbot handling a late order, Nikhil uses the percentage of a merchant's orders with issues during the last week as an example of useful context. Running a query over production order and issue tables may work initially, but scanning a merchant's full order history does not scale. The scalable approach pre-aggregates counts instead of reading raw orders on every request. That requires a batch pipeline over historical data, a streaming pipeline for recent data, a key-value store such as DynamoDB, and a service that supplies the result to the ML system or prompt.

### Retrieval adds another set of pipelines and services
[06:30](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=390s)
Nikhil's second example is finding issues similar to the current issue. The system must search issues across users, generate embeddings, store vectors, and retrieve the relevant results for the prompt. Embeddings may come from a deployed model or an external vendor such as OpenAI. The design also needs a vector store, a serving service, orchestration with Airflow, and monitoring with Grafana. More prompt elements, such as delivery-person issue rates or market-specific policy documents, multiply the infrastructure that must be built and operated.

### Infrastructure complexity causes teams to use less data
[07:43](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=463s)
Nikhil says the production burden has a direct effect on model and prompt quality. When each new data element requires its own pipelines, stores, services, orchestration, and monitoring, users often choose not to add enough data. The resulting context is weaker, so the model's responses are weaker. A notebook can be quick to read and build, but turning it into a production system is painful for one data scientist and remains a large undertaking even for a small systems engineering team.

### Chronon turns user-defined computations into production endpoints
[09:06](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=546s)
The central goal of Chronon is for ML systems to be generated rather than hand-built. Users define computations over raw data. Chronon generates the supporting infrastructure, which users then use to train and evaluate models and iterate on them. The platform also exposes production endpoints for applications. From the user's point of view, the workflow looks like writing Python code and reaching production endpoints without handing the system to an intermediate team that has to build everything manually.

### Chronon connects existing tools instead of replacing every category
[10:32](https://www.youtube.com/watch?v=0wGLWWZuq6I&t=632s)
In the closing discussion, Nikhil divides the surrounding ecosystem into offline training-data generation, model training, feature serving, model serving, observability, and orchestration. He names Databricks and Snowflake for data pipelines, Tecton and Fennel for online feature serving, Vertex AI and SageMaker for model training and serving, and Arize and Fiddler for monitoring. Chronon's approach is to make it easier for organizations to deploy strong open-source projects in these areas and use them to generate the required infrastructure.

## Notable quotes
- "We want the ML system to be generated, not handbuilt." (09:06)
- "The amount of infrastructure you need to support this is pretty massive." (06:05)
- "People simply don't use enough data in their models or in their prompts." (08:40)
- "Our approach is slightly different. We think there are really good open source projects that solve each of these things really well." (11:46)

## Tools & references mentioned
- Chronon
- Airbnb
- Stripe
- zipline.ai
- Meta
- Amazon
- Walmart
- Turbine
- Stylus
- DynamoDB
- Spark
- Flink
- Airflow
- Grafana
- OpenAI
- Databricks
- Snowflake
- Tecton
- Fennel
- Vertex AI
- SageMaker
- Arize
- Fiddler

## Who should watch
- You are building feature or prompt data pipelines and keep finding that production infrastructure takes much longer than the prototype.
- Your team is deciding whether to adopt Chronon alongside existing tools for batch data, online serving, model serving, monitoring, or orchestration.
- You want a concrete explanation of why seemingly small ML or retrieval features require so many separate systems.

## Editor's note

Nikhil says a data scientist can prototype in a few weeks or a month, then wait many months or even a year for a production ML system. ZenML lets teams write workflows as Python pipelines and run the same pipeline code on different infrastructure, so the path from an experiment to a scheduled production workflow does not require rewriting it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Real-time features, AI search, Agentic similarities](https://mlopstalks.com/talks/real-time-features-ai-search-agentic-similarities) (Varant Zanoyan & Nikhil Simha Raprolu, Zipline AI, 29:28)
- [Scaling your data and AI from 0-100 with open source](https://mlopstalks.com/talks/scaling-your-data-and-ai-from-0-100-with-open-source) (Maarten Breddels, Pycafe & Pranav Aurora, Mooncake & Simba Khadder, Featureform, 1:09:51)
- [AI Innovations: The Power of Feature Platforms](https://mlopstalks.com/talks/ai-innovations-the-power-of-feature-platforms) (Mahesh Murag, Tecton & Jose Navarro, Cleo & Nikhil Garg, Fennel, 1:05:28)
- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)
- [Data Engineering: The Missing Piece of Your Data Science Puzzle](https://mlopstalks.com/talks/data-engineering-the-missing-piece-of-your-data-science-puzzle) (, 12:10)