# Unpacking 3 Types of Feature Stores

Simba Khadder, Featureform | MLOps Podcast | Episode 265 | 1:05:42
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Z3NtwdhvtNg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/unpacking-3-types-of-feature-stores
Published: 2024-10-01
Tags: embeddings, feature-engineering, feature-stores, rag, recommender-systems

## TL;DR
- Embeddings can capture relationships between users and items, and recommender systems use them for candidate generation and reranking.
- Vector stores index embeddings for fast nearest-neighbor lookups, while feature stores manage the pipelines, production data, versioning, lineage, and serving needed by ML systems.
- Simba Khadder divides feature stores into literal, physical, and virtual categories, with Featureform taking the virtual approach above existing data infrastructure.

## Summary
Simba Khadder explains how his work on recommender systems led him to embeddings and feature stores. Embeddings can represent users, items, images, and other objects, allowing systems to find related candidates and then rerank them with specialized models. He contrasts this with simpler models such as XGBoost, which may be easier to understand and debug. The conversation then separates vector stores from feature stores. Vector stores specialize in approximate nearest-neighbor search over embeddings. Feature stores handle the broader work of turning raw data into production features, including pipelines, versioning, lineage, monitoring, serving, and consistency between training and inference. Simba describes three feature store categories: literal stores that hold computed features, physical stores that own feature computation and storage, and virtual stores that orchestrate existing infrastructure. For LLM applications, he argues that retrieval should include structured user and business signals alongside document chunks. Personalization variables are features, and the harder problem is selecting and packing the right information into the prompt.

## Key ideas
### Recommender systems gave Simba an early view of embeddings
[02:28](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=148s)
Simba Khadder became interested in embeddings while working on recommender systems and two-tower architectures. These systems turn users and items into embeddings based on behavior and interactions. He describes an e-commerce example where the learned relationships captured analogies such as Coke, Diet Coke, Cherry Coke, and Coke Zero. The model inferred flavor relationships from sparse purchase data rather than from an explicit description of each product. Simba connects this to language models, where reading tokens one after another can lead a model to learn ideas and concepts that were never directly labeled.

### Embedding systems can improve recommendations while making failures harder to understand
[07:12](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=432s)
For recommendation, a model can create a user embedding, retrieve 1,000 or 10,000 nearby video or item embeddings as candidates, and then use specialized models to rerank them for different surfaces such as search, autoplay, or a side view. Embeddings can also be used as inputs to traditional models, or averaged across a user's recent items. Simba says the tradeoff is interpretability and reliability. Embedding-heavy systems may produce excellent predictions, but when they fail, the layered abstractions make debugging painful. Models such as XGBoost often remain useful because their behavior is easier to inspect and they perform consistently.

### Vector stores and feature stores solve overlapping problems with different purposes
[12:37](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=757s)
Simba defines a vector store as a specialized index for fast approximate nearest-neighbor lookups. Exact nearest-neighbor search is expensive, so systems use algorithms such as HNSW to trade some recall for speed, latency, scale, or usability. A feature store addresses a broader ML data problem: transforming raw data into useful signals and operating those transformations in production. That includes organization through versioning, lineage, and governance, along with production concerns such as uptime, monitoring, incremental processing, and stream processing. The overlap appears when a feature is itself an embedding. A vector store can hold or index that value, while a feature store manages how it is produced and used.

### A feature store becomes necessary when data work creates operational pain
[22:08](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=1328s)
Simba says every production model that takes processed data as input already has something that functions as a feature store. That system might be Bash scripts, a Python package, or processing embedded in a container. The decision to adopt a product should start with the pain experienced by data scientists and platform teams. He divides the ML lifecycle into data, training, model production or serving, and monitoring or evaluation. In his experience, data and monitoring tend to create the most pain. A feature store can let data scientists build and experiment with feature pipelines themselves while providing versioning, monitoring, scale, uptime, incremental processing, and stream processing.

### Featureform puts an operational layer above existing infrastructure
[30:57](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=1857s)
Featureform calls its approach a virtual feature store. It does not require a company to replace BigQuery, Spark, Snowflake, a vector store, or another existing system. Instead, it adds an application and metadata layer above those systems so they work together as a feature store. Simba describes this as a logical layer for features. A data scientist can provide a SQL or dataframe transformation, and Featureform can turn it into a production data pipeline with versioning, lineage, search, reuse, monitoring, and point-in-time correctness. The approach targets companies that have already built ad hoc systems on their existing data infrastructure.

### The three feature store categories differ in what they own
[34:10](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=2050s)
Simba describes a literal feature store, popularized by Feast, as a place where computed features are stored. Databricks and SageMaker use versions of this model. It is simple, but it does not by itself solve the productionization of feature pipelines, streaming, versioning, or lineage. A physical feature store owns feature computation and storage through its own systems. It can make pipelines production-ready and scalable, but it adds a proprietary compute and storage layer. A virtual feature store coordinates existing engines and storage systems through connectors. Featureform follows this third model, so organizations can keep their data gravity while giving data scientists a managed way to build and deploy features.

### RAG needs structured signals as well as document similarity
[53:23](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=3203s)
Simba describes basic RAG as chunking documents, embedding the chunks, finding nearby chunks for a user query, and placing them into a prompt template. He argues that this is only one form of context retrieval. If someone asks for investment advice, useful context may include their age, income, and risk profile. Those values are unlikely to come from a nearest-neighbor lookup in a vector database. They may live in a relational database or feature store instead. Simba expects vector databases to move toward broader context-retrieval systems that combine embeddings with metadata and other signals, then decide what information to put into the prompt.

### Personalization variables in LLM prompts are features
[57:56](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=3476s)
Simba reframes prompt variables as features, signals, or contextual data points. They can describe a person, a company, or another entity, and they can be numerical or textual. A prompt that only names a company gives the model limited context. A prompt that retrieves a richer description can give it more information for the task. He thinks this is especially useful for LLM applications that support people in their work, since LLMs are relatively slow and expensive and are often used as augmentation tools. He is more skeptical of fully automated uses that require very high accuracy, such as fraud detection.

### LLM evaluation should connect to user value
[1:03:44](https://www.youtube.com/watch?v=Z3NtwdhvtNg&t=3824s)
Simba compares LLM evaluation with recommender-system evaluation, where users rarely provide explicit thumbs-up or thumbs-down feedback. Recommendation systems instead use implicit signals such as whether someone clicked a result, how long they stayed, or whether they returned to search. He expects LLM products to use similar behavioral signals. Automated statistical evaluation and additional LLM judges can become confusing when the connection to the end user is lost. For applications used by people, the practical test is whether users found the response valuable. ChatGPT's side-by-side prompt experience is one example of collecting information that can help tune models.

## Notable quotes
- Simba Khadder: "The problem that feature stores are solving is the problem of a lot of ML is inherently a specialized form of data engineering." (16:16)
- Simba Khadder: "A vector database is a way to index your vectors through your embeddings such that you can do nearest neighbor lookups pretty quickly." (15:45)
- Simba Khadder: "Everyone has a feature store if you have a model in production that takes data as inputs." (19:44)
- Simba Khadder: "I don't think there's any reason that that is true other than one, that it does work quite well naively, and two, it just those companies came about first." (55:51)
- Simba Khadder: "All that matters is the end user and, like we talked about, most of the time the end user is literally a human." (1:04:38)

## Tools & references mentioned
- Featureform
- TritonML
- Google
- Feast
- Databricks
- SageMaker
- BigQuery
- Snowflake
- Spark
- HNSW
- Spotify
- YouTube
- Airbnb
- Uber
- Chalk
- LinkedIn
- Feather
- Mosaic
- Tabular
- ClickHouse
- RAG
- DSPy
- LlamaIndex
- ChatGPT
- XGBoost
- Weights & Biases
- MLflow

## Who should watch
- You are deciding whether your team needs a feature store or whether its existing scripts, warehouse, or ad hoc jobs are sufficient.
- Your ML system uses embeddings and a vector store, and you need to understand where feature computation, training data, and online serving fit.
- You are building an LLM application and want to retrieve personal or business context that cannot be found through document similarity alone.

## Editor's note

Simba Khadder says companies often have feature stores already, even when the system is only Bash scripts, a Python package, or processing inside a container. ZenML lets teams write workflows as Python steps and run the same pipeline on different infrastructure through configuration. Each run records its steps, inputs, outputs, and code version, so features can be traced back to how they were produced.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [How Feature Stores Work](https://mlopstalks.com/talks/how-feature-stores-work) (Simba Khadder, Featureform, 30:33)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [Machine Learning Feature Store Panel Discussion](https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion) (Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit, 1:05:16)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)
- [A Conversation Around Feature Stores](https://mlopstalks.com/talks/a-conversation-around-feature-stores) (Venkata Pingali, Scribble Data, 1:03:18)