# Machine Learning Feature Store Panel Discussion

Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit | MLOps Coffee Sessions | Episode 26 | 1:05:16
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=-TGp2qKz8tA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion
Published: 2021-01-19
Tags: data-engineering, data-quality, feature-engineering, feature-stores, platform-teams

## TL;DR
- A feature store is only worthwhile when it solves a specific problem, such as online and offline consistency, feature reuse, or low-latency availability.
- iFood is building its feature store around declarative transformation pipelines, while it evaluates whether other parts should be bought or built.
- Feature reuse depends on metadata, lineage, data quality, and processes that make teams responsible for the data they produce.

## Summary
This panel examines when teams need a feature store and how they should approach building or buying one. Matias Dominguez describes Rappi's move from a MySQL-backed fraud system toward a proof of concept with a feature store vendor. The existing system works for its current latency needs, but duplicated features and growing machine learning use cases make a shared platform more attractive. Daniel Galinkin explains that iFood's earlier Redis-based online store had no online and offline parity, so training data was calculated differently from serving data. iFood is concentrating first on feature transformations and declarative feature definitions. Simarpal Khaira describes how Intuit gathers requirements by persona, since fraud teams, data scientists, and analysts need different interfaces and latency models. The group also discusses feature lineage, data quality, versioning, review processes, and adoption. Their advice is practical: define the problem first, build around actual users, and treat a feature store as several related components rather than one fixed product.

## Key ideas
### A working database can be enough until duplication and scale create a shared-platform need
[03:58](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=238s)
Matias Dominguez says Rappi's fraud team has used its own online feature store for almost two years. The original requirement was modest: query values such as how much money a customer tried to send during the previous day so the team could improve a rule-based system. MySQL serves those values in real time, and the team has not seen latency or contention problems. The need for a broader feature store emerged later, as Rappi added recommendation, segmentation, pricing, and delivery-estimate use cases. Similar features were being rebuilt inside separate teams. A shared platform could make canonical features available to many models, although that would not have justified the budget when the first system was built.

### The online and offline stores need the same feature logic
[13:08](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=788s)
Daniel Galinkin describes iFood's earlier Redis-based online store. It provided low-latency access for predictions, but training teams had to retrieve historical data and calculate features through a separate process. The resulting values could differ from the online features, which made the system inconsistent and error-prone. iFood's internal feature store therefore focuses heavily on the transformation pipeline. Users declare what a feature is, rather than specifying every step used to calculate it. The platform handles the processing and supports online, offline, point-in-time, and time-travel queries. Daniel also describes feature stores as modular, with transformation, storage, serving, monitoring, and registry components that can be adopted separately.

### Feature-store requirements change with the people and use cases involved
[17:31](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=1051s)
Simarpal Khaira explains that Intuit has products such as TurboTax and QuickBooks, along with teams that work across products on security, risk, and fraud. He gathers requirements from different personas instead of assuming that every user needs the same platform. Fraud teams may need real-time features with very short availability times. Analysts often work with batch jobs, dashboards, and reports. Machine learning engineers may want APIs and access to data-processing technologies, while analysts may need a more guided interface. The product has to support self-service without forcing every user to work directly with backend systems. Simarpal says the platform started with technical users but is gradually expanding toward a wider group of people building machine learning use cases.

### Transformation is the hardest feature-store component to get right
[25:32](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=1532s)
Daniel says iFood has good enough solutions for storage, serving, registry, and monitoring, but transformation remains the hardest part. Features can come from batch data, streaming data, backfilled data, or sources where only the newest value matters. The platform must handle these different shapes while preserving point-in-time correctness and online and offline parity. Daniel's preferred order is to solve transformation first, then storage and serving, followed by monitoring and registry. Storage and serving still matter because a feature store must make features queryable and available to production systems. Monitoring can initially be deferred, although teams eventually need to see feature costs, calculation times, data skew, and pipeline problems.

### Feature reuse requires metadata, lineage, and trust
[33:21](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=2001s)
Simarpal says Intuit captures metadata during feature registration. Teams describe the feature set, its use cases, associated entities, individual feature descriptions, and data types. That information is stored in a feature catalog where users can search for features alongside other data assets. Intuit wants users to follow the lineage from a feature to the code in the Git repository that produced it. This connection would help data scientists understand the transformation logic and trust a feature built by another team. Simarpal argues that reuse depends on this information because teams will not share features confidently when the meaning, source, or transformation is unclear.

### Data quality cannot be fixed by the feature-store framework alone
[37:47](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=2267s)
Matias argues that poor input data can invalidate a feature even when its transformation code is correct. He compares the issue to garbage in and garbage out, then points to upstream teams that generate the data as part of the solution. Daniel agrees that quality checks should exist throughout the flow, from an application event to ingestion, the data lake, and the feature store. A failing check should stop or interrupt a pipeline before bad values spread. iFood was still working on this at the feature-store level and was evaluating vendors for data observability and quality checks. Both panelists describe data quality as a people and awareness problem as well as a technical one.

### Declarative definitions and review processes make bad practices harder
[46:26](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=2786s)
Daniel describes iFood's declarative approach as a way to bake good practices into the tool. A feature definition includes its name, aggregation type, data source, attributes, and window. The user does not write the full calculation logic. The platform handles calculation, storage, and processing, and exposes APIs for offline point-in-time and online access. Feature definitions are currently short Python files in a centralized repository, and iFood reviews them before creation. Daniel is unsure whether centralized manual review will scale, but it gives the team a control point while the platform is still developing. Simarpal makes a similar case for self-service interfaces that guide users toward the right metadata and workflows.

### A proof of concept should include the teams that will eventually use the platform
[45:38](https://www.youtube.com/watch?v=-TGp2qKz8tA&t=2738s)
Matias says Rappi is not building its feature-store proof of concept in isolation. People from five or six teams are part of the feature-store group, even though the first migration targets a fraud use case. Those participants can explain how their teams would migrate existing systems, interface with the platform, and use its features. They can also expose requirements that would be missed by a single team. Matias's view is that a company-wide platform needs participation from much of the company if it is expected to deliver value across the company. This approach lets the team add different groups' practices while it defines the architecture.

## Notable quotes
- Daniel Galinkin: "Feature store is a really broad concept and it's actually lots of components and lots of different moving parts." (16:02)
- Simarpal Khaira: "The first very first requirement was around feature availability." (32:02)
- Demetrios Brinkmann: "Reuse can only happen with consistency." (36:31)
- Daniel Galinkin: "Everyone that produces data has to be responsible for the quality of the data that's been produced." (41:55)
- Matias Dominguez: "If we want to make this work for the whole company we have to take people from most of the company to actually work on it." (56:20)

## Tools & references mentioned
- Tesseract Health
- Rappi
- iFood
- Intuit
- AWS SageMaker Feature Store
- Feast
- Tecton
- Redis
- MySQL
- Snowflake
- dbt
- Spark
- Flink
- Cassandra
- GitLab
- Docker Registry
- Tableau
- Protobuf
- Neil Lathia
- Kevin Stumpf

## Who should watch
- You are deciding whether a feature store solves a real problem in your machine learning platform or would add unnecessary infrastructure.
- Your online predictions and offline training use different feature calculations, and you need a practical route to parity and point-in-time correctness.
- Several teams are rebuilding similar features and you need advice on metadata, data quality, adoption, and involving users in a proof of concept.

## Editor's note

Daniel Galinkin says iFood's earlier online and offline feature calculations could produce different values, making the system inconsistent and error-prone. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a transformation's results can be traced to the code and data that produced them. Its pipelines can also run on different infrastructure without changing the pipeline code.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [A Conversation Around Feature Stores](https://mlopstalks.com/talks/a-conversation-around-feature-stores) (Venkata Pingali, Scribble Data, 1:03:18)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [Global Feature Store: Optimizing Locally and Scaling Globally at Delivery Hero](https://mlopstalks.com/talks/global-feature-store-optimizing-locally-and-scaling-globally-at-delivery-hero) (Gottam Sai Bharath & Cole Bailey, Delivery Hero, 50:19)
- [Feature Stores at Shopify and Skyscanner](https://mlopstalks.com/talks/feature-stores-at-shopify-and-skyscanner) (Matt Delacour, Shopify & Mike Moran, Skyscanner, 49:36)