# A Conversation Around Feature Stores

Venkata Pingali, Scribble Data | MLOps Coffee Sessions | Episode 4 | 1:03:18
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ajCd6k_OZO0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/a-conversation-around-feature-stores
Published: 2020-07-27
Tags: data-quality, feature-engineering, feature-stores, governance

## TL;DR
- A feature store gives data scientists a managed way to find, transform, reuse, and serve the data used by machine learning models.
- Feature stores become necessary as teams operate more models and need feature reuse, reproducibility, validation, observability, and governance.
- Trust depends on checking the data itself, including its quality, fairness, drift, provenance, privacy, and explainability.

## Summary
Jim Dowling and Venkata Pingali explain why feature stores emerged as machine learning teams grew beyond ad hoc feature engineering. Jim describes a feature store as a data platform and an API between data engineers and data scientists. It gives data scientists access to transformed data without requiring them to work directly with storage systems or write all the underlying pipelines. Venkata places the idea in the evolution of machine learning operations. As models and users multiply, teams need shared definitions, predictable computation, observability, and governance. The discussion covers offline and online storage, reuse across models, point-in-time data, real-time joins, and separate pipelines for feature generation, training, and deployment. The guests also connect feature stores to risk management. They describe validation, fairness checks, privacy enforcement, auditability, and feature-level trust as requirements for production systems. They expect feature stores to take different forms depending on a company's scale and data needs.

## Key ideas
### Features include everything from simple columns to model outputs
[01:49](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=109s)
Jim defines features as the data used to train models and make predictions in production. Venkata adds that production systems contain many forms of features. A feature may be a qualitative value, an image, logic over existing columns, or even the output of another model. He says this complexity has grown beyond what ordinary SQL and ETL systems can handle. David gives concrete examples such as restaurant demand, traffic, height, weight, and blood pressure. In each case, the feature describes an instance or example that the model uses.

### A feature store connects data scientists with data engineering systems
[04:54](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=294s)
Jim calls a feature store a data platform for AI. It gives data scientists an API for obtaining raw data and transforming it into features, instead of sending them directly to S3, Redshift, Hadoop, or another backend. He compares this with asking a business analyst using Tableau to fetch and prepare data from an S3 bucket. The interface should provide Python-based access rather than forcing users to write pure SQL. Data scientists can use existing features and create derivative features without taking on the underlying data engineering work.

### Feature stores appeared as machine learning teams began to scale
[07:18](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=438s)
Venkata describes feature stores as a natural evolution of feature engineering. When teams had a small number of models and users, data scientists or ETL engineers could manage features locally. As models, users, and feature complexity increased, teams found duplicated computations and hidden assumptions. They also needed predictable behavior, observability, and guarantees around features. Jim compares this progression with the rise of data warehouses, where organizations gradually centralized and governed analytics assets. He expects machine learning teams to centralize features, models, and training data over time.

### Good features still depend on human judgment
[16:50](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=1010s)
The guests say feature engineering can produce larger gains than changing an already efficient model implementation. Venkata divides the work into repeatable engineering that runs regularly and model-specific experimentation that requires creativity. Once a useful feature is found, a feature store can make it available to other teams. He says strong data scientists combine domain knowledge, statistical understanding, and experience. He rejects the idea that AutoML can replace judgment. Jim gives an anti-money-laundering example in which graph embeddings captured relationships between people and transactions that simple rules could miss.

### Feature stores reduce repeated work across models
[24:37](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=1477s)
Jim says teams need feature stores because data scientists otherwise spend time turning data lake and warehouse contents into clean training sets. Reusable features can support many models, including common bank data such as customer and credit information. The store can prepare outputs in formats such as TFRecord, NumPy, or CSV. Venkata says the economic pressure appears once several data scientists and models must coordinate. The system reduces friction, gives teams programmatic access to shared features, and makes feature development easier to repeat.

### Production feature systems need data governance and risk checks
[34:00](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=2040s)
Venkata says trust is becoming a main reason to use a feature store. Organizations need places to enforce privacy policies, conduct ethics reviews, and assess the risk of feature data. In one example, an ethics board approves features before they are made available to users. Jim argues that much of the risk in machine learning applications comes from data rather than algorithms. He describes validating distributions and checking whether groups are over- or underrepresented. Venkata adds requirements such as valid ranges, drift monitoring, explainability, and the ability to reconstruct point-in-time decisions.

### Offline and online feature serving require different designs
[41:10](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=2470s)
The conversation separates feature generation, management, and serving. Offline stores support large batch datasets used for training, while online stores provide low-latency values for operational predictions. Jim describes an application where a transaction supplies only a few current values, while hundreds of historical and contextual features are retrieved from the feature store. Reuse requires joining features by keys, including temporal keys. Venkata expects real-time and batch systems to become more integrated, although teams still use several design patterns, such as Kafka, stream processing, and Redis-style stores.

### Feature marketplaces may attach value and cost to features
[55:48](https://www.youtube.com/watch?v=ajCd6k_OZO0&t=3348s)
The hosts ask whether teams could measure feature usage, computation cost, and reuse. Jim says this depends on company culture, but feature sharing could work like a leaderboard for data engineers. Venkata says features carry different information value and maintenance costs. His team uses a marketplace concept and tracks whether features are used. He expects organizations eventually to prioritize features and ask data scientists or ML engineers to justify the cost and value of keeping them. Badges could provide incentives, but the main idea is making feature economics visible.

## Notable quotes
- Jim Dowling: "Feature features are the data that you use to train machine learning models but also when you're operating models, when you have models in production, features are the data that are used to make predictions with those models." (01:49)
- Jim Dowling: "The feature store is a platform that provides an API between the data scientists and the data engineers." (05:59)
- Venkata Pingali: "The best data scientists in my opinion are those who are able to combine deep insights into the domain with the statistical characteristics of the data and a bunch of experience." (16:50)
- Jim Dowling: "A lot of risk in productionizing machine learning applications derives from the data rather than the algorithm." (37:26)
- Jim Dowling: "Break up your monolithic end-to-end pipelines." (59:40)

## Tools & references mentioned
- Scribble Data
- Logical Clocks
- Hopsworks
- S3
- Redshift
- Hadoop
- Tableau
- Power BI
- ClickHouse
- TensorFlow Extended
- Apache Spark
- Apache Beam
- Apache Flink
- Kafka
- Redis
- MySQL Cluster
- Twitter
- Andrew Ng
- GDPR
- Jan LeCun

## Who should watch
- You are deciding whether your growing machine learning team needs a shared system for feature reuse and access to warehouse data.
- Your models need both historical training features and low-latency values at prediction time, and separate pipelines are becoming difficult to manage.
- You are responsible for data quality, fairness, privacy, or auditability and need those checks close to the feature data.

## Editor's note

Jim says teams need feature stores because data scientists otherwise spend time turning data lake and warehouse contents into clean training sets. ZenML lets teams define that work as Python pipeline steps and run it on different infrastructure without rewriting the pipeline. Each run records its inputs, outputs, and code version, so the resulting training data can be traced back to its source.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Machine Learning Feature Store Panel Discussion](https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion) (Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit, 1:05:16)
- [How Feature Stores Work](https://mlopstalks.com/talks/how-feature-stores-work) (Simba Khadder, Featureform, 30:33)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)
- [Global Feature Store: Optimizing Locally and Scaling Globally at Delivery Hero](https://mlopstalks.com/talks/global-feature-store-optimizing-locally-and-scaling-globally-at-delivery-hero) (Gottam Sai Bharath & Cole Bailey, Delivery Hero, 50:19)