# Global Feature Store: Optimizing Locally and Scaling Globally at Delivery Hero

Gottam Sai Bharath & Cole Bailey, Delivery Hero | MLOps Podcast | Episode 263 | 50:19
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=c9L8vAvCUyA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/global-feature-store-optimizing-locally-and-scaling-globally-at-delivery-hero
Published: 2024-09-24
Tags: data-quality, feature-engineering, feature-stores, platform-teams, team-adoption

## TL;DR
- Delivery Hero moved away from a single central machine learning platform team because it could not prioritize the different needs of many departments.
- The Global Feature Store combines local implementations, such as Pandora's batch store and Logistics' streaming store, while allowing departments to keep some local control.
- Adoption depends on proving value department by department, supporting different cloud environments, and making migration from existing systems worth the effort.

## Summary
Gottam Sai Bharath and Cole Bailey describe how Delivery Hero changed its approach to machine learning platform work. A central team initially tried to build common tools for the company, but demand from many departments became difficult to prioritize. Local teams then built their own systems, including Pandora's batch feature store and Logistics' real-time feature store. The Global Feature Store brings these efforts together through inner sourcing. Departments contribute components, operate them locally, and share improvements. The guests explain that a feature is a data point usable by multiple consumers, while the feature store moves computed data to model training and inference. They discuss batch and streaming quality checks, feature discoverability, cloud compatibility, governance, and the problem of replacing established systems. The Global Feature Store was still at the MVP stage, with batch work planned first and streaming work later. Cole is candid that platform adoption is an organizational problem as much as a technical one.

## Key ideas
### A central platform team could not serve every department well
[02:29](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=149s)
Gottam Sai Bharath says Delivery Hero's original global machine learning platform team had a broad plan to centralize services and reduce cost. It supported teams such as fraud and science, but requests soon came from many departments with different priorities. When one team needed one feature and another team needed something else, the central group could not decide what to build first. The model became difficult to manage. Delivery Hero shifted toward local teams solving their own problems, then contributing useful pieces to a shared global system.

### Local feature stores grew around different use cases
[09:14](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=554s)
Cole Bailey explains that Logistics and Pandora had built different feature store capabilities. Logistics had a live feature store backed by Flink pipelines that aggregated streams and served the results to models. Pandora had a batch store that ran SQL against the data warehouse and loaded results into caches. Logistics needed batch features, while Pandora needed live features, so the two implementations were natural complements. Logistics handles problems such as delivery times, rider staffing, and delivery fees, while Pandora focuses on personalization and recommendations across its markets.

### The Global Feature Store uses inner sourcing instead of adding another team
[11:02](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=662s)
Cole describes the Global Feature Store as one of Delivery Hero's first large inner-sourcing efforts. Pandora owns the batch implementation and works to make it scalable so other departments can install, operate, and maintain it locally. Those departments can contribute changes back when they need improvements. Representatives from about 12 departments met for three days and compared seven feature store implementations. The goal was to combine the useful parts without simply stitching them together into a system that nobody could operate.

### The teams had to agree on basic definitions before designing the platform
[20:18](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=1218s)
During the workshop, the participants discovered that they did not agree on what a feature or a feature store meant. Cole says they had to return to the basics and define a feature as a single column or data point that can be used by multiple consumers. He describes the feature store as the engine that moves computed data to model training and inference. Computation remains in the data platform, while the feature store provides metadata and access through offline and online SDKs. The design avoids creating a separate storage layer when the data warehouse already holds the data.

### Batch features already provide value, while streaming is still being built
[23:49](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=1429s)
Gottam says Delivery Hero has already seen value from the feature store, including batch features used by Pandora and Logistics models. Pandora currently does not have real-time feature serving, so streaming features are being developed to support personalization and recommendations that can react more quickly. Logistics already has streaming experience, which gives Pandora and the global effort a starting point. Reusing that work could reduce engineering effort. The Global Feature Store was still an MVP, with batch work planned first and streaming expected later.

### Feature quality checks are much harder for streaming systems
[33:46](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=2026s)
Cole says streaming features require consistency checks across different data sources. Models need historical data for training, while online features come from streams, so Logistics compares data in its warehouse with data from Kafka. Features are run offline, deployed to a shadow environment, and compared with online results after collecting logs. The comparison is never perfectly identical because Flink and BigQuery can compute results differently. The team also monitors job health, checkpointing, latency, and consumer lag. These checks remain difficult to scale across departments.

### Adoption requires pragmatic decisions and support for different users
[28:02](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=1682s)
Cole says the teams made progress by using evidence and choosing what already worked. When most departments were using Redis, the group chose Redis rather than continuing an abstract debate about an alternative. Pandora's engineers prefer technical workflows and complex SQL, while other departments need a user interface, a catalog, and simple feature creation. The Global Feature Store is intended to remain component-agnostic so departments can contribute capabilities that others can use. The teams still need governance for ownership, reviews, and large changes.

### Replacing existing systems will be the hardest part of going global
[42:13](https://www.youtube.com/watch?v=c9L8vAvCUyA&t=2533s)
Cole expects the Global Feature Store to be adopted one department at a time. A department with no engineers and a new use case may adopt it easily, while a department with years of existing Airflow workflows has to justify migration effort and risk. One group may require batch features and a catalog, while another may only adopt after live features are available. Delivery Hero also has to support different production cloud setups, including AWS and Google Cloud. The long-term plan depends on proving that the shared system provides enough value to replace older standards.

## Notable quotes
- Cole Bailey: "I think Global feature store is one of the first examples where we're trying to really do this inner sourcing at a bigger scale." (11:17)
- Cole Bailey: "A feature should be essentially one column or data point if you think of like a table view." (21:42)
- Cole Bailey: "The feature store is kind of the whole engine that moves data around from where the data is computed to where it's served to models." (22:03)
- Cole Bailey: "It's not as easy as just develop a new standard, deploy your new system and everyone will adopt it." (48:34)
- Gottam Sai Bharath: "The whole initiative is very relevant for especially companies structured like Delivery Hero, where there's central teams and then there's so many local teams." (46:40)

## Tools & references mentioned
- Delivery Hero
- foodora
- foodpanda
- Pandora
- Google Cloud
- Vertex AI
- Kubeflow
- Flink
- Kafka
- BigQuery
- Feast
- Airflow
- Redis
- AWS
- Terraform

## Who should watch
- You run a central ML platform team and are struggling to prioritize requests from departments with different workflows.
- Your company has several local feature store or ML platform implementations and needs a practical way to share components without forcing one design everywhere.
- You are migrating from batch features to streaming features and need to understand the testing, data consistency, and operational work involved.

## Editor's note

Cole Bailey says Delivery Hero's central platform team could not decide what to build first for departments with different priorities. ZenML lets teams keep the same pipeline code while choosing the infrastructure stack for a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider. That supports local ownership without forcing every department onto one deployment setup.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Machine Learning Feature Store Panel Discussion](https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion) (Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit, 1:05:16)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)
- [Feature Stores at Shopify and Skyscanner](https://mlopstalks.com/talks/feature-stores-at-shopify-and-skyscanner) (Matt Delacour, Shopify & Mike Moran, Skyscanner, 49:36)
- [A Conversation Around Feature Stores](https://mlopstalks.com/talks/a-conversation-around-feature-stores) (Venkata Pingali, Scribble Data, 1:03:18)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)