# BigQuery Feature Store

Nicolas Mauti, Malt | MLOps Podcast | Episode 255 | 50:39
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=NtDKbGyRHXQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/bigquery-feature-store
Published: 2024-08-23
Tags: data-engineering, data-quality, feature-engineering, feature-stores, monitoring

## TL;DR
- Malt moved feature computation into daily, historized BigQuery tables so data scientists could reuse consistent features during model training.
- The design supports point-in-time retrieval, monitoring, alerting, and backfilling without adding a separate feature store product.
- The approach fits batch workloads, while live feature computation, very large serving data, and fresher-than-daily features would require other systems.

## Summary
Nicolas Mauti explains why Malt built a feature store with BigQuery for its recommendation, NLP, and matching models. Previously, data scientists computed features inside training code. Teams could reuse names through copy-paste, but the underlying calculations could differ, making results hard to reproduce and features hard to share. Point-in-time retrieval was also expensive because training required historical joins. Malt now computes features daily for freelancers and projects, adds timestamps, and stores the results in historized BigQuery tables. Training data needs the target, entity IDs, and observation date, followed by a join to retrieve the correct feature values. Airflow runs the workflows, while Great Expectations blocks training when feature checks fail and Grafana provides descriptive statistics. Nicolas is clear about the limits. The design is suited to batch computation and manageable online serving data. Live features, data that cannot fit in memory, or fresher updates would call for additional infrastructure.

## Key ideas
### Malt's original feature code made shared features inconsistent
[03:25](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=205s)
Malt trained recommendation, NLP, and matching models using information about freelancers, projects, and past interactions. Feature computation happened directly before model training. Two data scientists could use a feature with the same name while calculating it differently, which made results difficult to reproduce and confusing to explain to product teams. Sharing mainly meant copying code or queries. Features were also recomputed whenever a model was trained instead of being stored for reuse. Nicolas separates this training problem from serving: Malt already loaded its relatively small serving feature set into memory daily, so low-latency prediction was not the immediate issue.

### Point-in-time retrieval was expensive and easy to get wrong
[08:33](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=513s)
For a reliable training example, Malt needs the state of the freelancer and project at the time an interaction occurred. A freelancer's skills or willingness to take a type of project can change, so using today's values would distort historical training data. Reconstructing those values on the fly required large joins across historical tables. Nicolas says this process was complex, expensive, and prone to mistakes by data scientists. A feature store needed to preserve historical feature values so training could retrieve the state that existed on the observation date.

### BigQuery became a daily, historized feature table
[14:26](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=866s)
Malt created a feature table in BigQuery with one column per feature and rows computed daily for all freelancers and projects. Each daily result receives a timestamp and is stored historically. A data scientist builds a training dataset with the target, freelancer ID, project ID, and observation date, then performs one join against the feature table. That join supplies the feature values for the requested dates. The same daily computation can also be pushed into Redis when the serving data is too large for in-memory loading. Nicolas describes the basic design as simple because it uses an existing analytics system rather than adding a separate feature-store product.

### Backfilling new features uses existing historical ingestion data
[17:17](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=1037s)
Adding a feature creates a separate problem: the new column needs historical values, where the underlying data exists. Malt keeps historical data at the ingestion layer, followed by transformation layers and the feature tables. A script lets a data scientist provide a SQL query that computes the new feature across available dates and updates the feature table. The process is tracked with a change log and feature-table versions. The change is submitted through a GitLab project, then applied by a scheduled Airflow process after review. Airflow runs SQL files that have not yet been applied, so the process is controlled rather than an unrestricted change to the shared table.

### Feature monitoring checks both validity and change over time
[23:00](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=1380s)
Malt uses Grafana connected to BigQuery to inspect feature statistics, including means, averages, and category counts. The team also needs alerts for invalid training inputs, such as a freelancer daily-rate feature becoming null. Monitoring over time can reveal drift, such as rates rising or falling because of inflation or a product change. Great Expectations runs tests after the daily feature table is calculated. Nicolas compares these tests to unit tests for data. If a check fails, the workflow stops, model training does not continue, and a Slack alert identifies the failed expectation, such as an unexpected categorical value.

### Malt monitors models at business, data, and system levels
[30:13](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=1813s)
Malt retrains many models monthly, after feature checks pass, and deploys a new model automatically when its metrics are acceptable. Nicolas also monitors business measures such as matching conversion, model outputs and scores, feature data, and platform measures such as CPU use and latency. A latency increase can point to a system issue, but it can also reveal a model-quality problem, such as the model considering too many freelancers. Business metrics provide another view, although Nicolas prefers finding problems before they reach business results. Model accuracy and business performance can diverge, so the team compares metrics from different stages before deciding where to investigate.

### The design reduces repeated computation and separates team tasks
[39:51](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=2391s)
Nicolas does not give a precise current savings figure, but he says feature computation previously cost several hundred dollars per month when calculated repeatedly during experiments and training. The new design computes features once per day, after which data scientists can access them for training. Its larger benefit is how it separates feature engineering from model training. A data scientist can spend time creating and checking features in the shared table, then switch attention to model structure and hyperparameters. Nicolas says these tasks require different kinds of thinking, so keeping them separate makes the work clearer.

### Real-time requirements would push BigQuery beyond this design
[44:43](https://www.youtube.com/watch?v=NtDKbGyRHXQ&t=2683s)
Nicolas says the architecture works well for batch feature computation, but BigQuery would not be the best fit for live, on-the-fly features. Serving data that is too large for memory would need a low-latency database such as Redis. Features that must update more often than daily would create another challenge. Malt may use its streaming ingestion capability and consider microbatching, but Nicolas does not yet have a final design for writing those computations into the feature table. The future pressure points are therefore serving-data size and live feature computation.

## Notable quotes
- Nicolas Mauti: "We were not storing the feature, we were just computing on the fly and training the model." (06:57)
- Nicolas Mauti: "You have to get the state of the freelancer and also of the project at the time where the interaction was done." (09:15)
- Nicolas Mauti: "So we compute all our features and then we put a timestamp and store it also in BigQuery." (14:51)
- Nicolas Mauti: "At the end, you are evaluated on the business, not on the accuracy of your model." (36:27)
- Nicolas Mauti: "If you want to go with a full on-the-fly pipeline, maybe it's not the best way to go." (44:52)

## Tools & references mentioned
- BigQuery
- Redis
- Feast
- Grafana
- Great Expectations
- Airflow
- GitLab
- Flink
- Slack

## Who should watch
- You are considering a feature store for batch-trained models and already have BigQuery in your data stack.
- Your team has duplicated feature logic, inconsistent feature definitions, or expensive point-in-time training joins.
- You need real-time features or very large online-serving data, and want to understand where a BigQuery-based design stops fitting.

## Editor's note

Nicolas Mauti explains that Malt's shared feature names could hide different calculations, making training results hard to reproduce. ZenML records each pipeline step's inputs, outputs, and code version, so a feature computation can be traced to the data and code that produced it. Its pipelines also cache unchanged steps instead of recomputing them.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [How Feature Stores Work](https://mlopstalks.com/talks/how-feature-stores-work) (Simba Khadder, Featureform, 30:33)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [Building ML Blocks with Kubeflow Orchestration with Feature Store](https://mlopstalks.com/talks/building-ml-blocks-with-kubeflow-orchestration-with-feature-store) (Aniruddha Choudhury, Publicis Sapient, 1:26:03)
- [Machine Learning Feature Store Panel Discussion](https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion) (Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit, 1:05:16)
- [Real-time Feature Pipelines, A Personal History](https://mlopstalks.com/talks/real-time-feature-pipelines-a-personal-history) (Hendrik Brackmann, Tide, 58:07)