Malt moved feature computation into daily, historized BigQuery tables so data scientists could reuse consistent features during model training.
2
The design supports point-in-time retrieval, monitoring, alerting, and backfilling without adding a separate feature store product.
3
The approach fits batch workloads, while live feature computation, very large serving data, and fresher-than-daily features would require other systems.
Summary
Nicolas Mauti explains why Malt built a feature store with BigQuery for its recommendation, NLP, and matching models. Previously, data scientists computed features inside training code. Teams could reuse names through copy-paste, but the underlying calculations could differ, making results hard to reproduce and features hard to share. Point-in-time retrieval was also expensive because training required historical joins. Malt now computes features daily for freelancers and projects, adds timestamps, and stores the results in historized BigQuery tables. Training data needs the target, entity IDs, and observation date, followed by a join to retrieve the correct feature values. Airflow runs the workflows, while Great Expectations blocks training when feature checks fail and Grafana provides descriptive statistics. Nicolas is clear about the limits. The design is suited to batch computation and manageable online serving data. Live features, data that cannot fit in memory, or fresher updates would call for additional infrastructure.
Malt's original feature code made shared features inconsistent
Malt trained recommendation, NLP, and matching models using information about freelancers, projects, and past interactions. Feature computation happened directly before model training. Two data scientists could use a feature with the same name while calculating it differently, which made results difficult to reproduce and confusing to explain to product teams. Sharing mainly meant copying code or queries. Features were also recomputed whenever a model was trained instead of being stored for reuse. Nicolas separates this training problem from serving: Malt already loaded its relatively small serving feature set into memory daily, so low-latency prediction was not the immediate issue.
Point-in-time retrieval was expensive and easy to get wrong
For a reliable training example, Malt needs the state of the freelancer and project at the time an interaction occurred. A freelancer's skills or willingness to take a type of project can change, so using today's values would distort historical training data. Reconstructing those values on the fly required large joins across historical tables. Nicolas says this process was complex, expensive, and prone to mistakes by data scientists. A feature store needed to preserve historical feature values so training could retrieve the state that existed on the observation date.
Malt created a feature table in BigQuery with one column per feature and rows computed daily for all freelancers and projects. Each daily result receives a timestamp and is stored historically. A data scientist builds a training dataset with the target, freelancer ID, project ID, and observation date, then performs one join against the feature table. That join supplies the feature values for the requested dates. The same daily computation can also be pushed into Redis when the serving data is too large for in-memory loading. Nicolas describes the basic design as simple because it uses an existing analytics system rather than adding a separate feature-store product.
Backfilling new features uses existing historical ingestion data
Adding a feature creates a separate problem: the new column needs historical values, where the underlying data exists. Malt keeps historical data at the ingestion layer, followed by transformation layers and the feature tables. A script lets a data scientist provide a SQL query that computes the new feature across available dates and updates the feature table. The process is tracked with a change log and feature-table versions. The change is submitted through a GitLab project, then applied by a scheduled Airflow process after review. Airflow runs SQL files that have not yet been applied, so the process is controlled rather than an unrestricted change to the shared table.
Feature monitoring checks both validity and change over time
Malt uses Grafana connected to BigQuery to inspect feature statistics, including means, averages, and category counts. The team also needs alerts for invalid training inputs, such as a freelancer daily-rate feature becoming null. Monitoring over time can reveal drift, such as rates rising or falling because of inflation or a product change. Great Expectations runs tests after the daily feature table is calculated. Nicolas compares these tests to unit tests for data. If a check fails, the workflow stops, model training does not continue, and a Slack alert identifies the failed expectation, such as an unexpected categorical value.
Malt monitors models at business, data, and system levels
Malt retrains many models monthly, after feature checks pass, and deploys a new model automatically when its metrics are acceptable. Nicolas also monitors business measures such as matching conversion, model outputs and scores, feature data, and platform measures such as CPU use and latency. A latency increase can point to a system issue, but it can also reveal a model-quality problem, such as the model considering too many freelancers. Business metrics provide another view, although Nicolas prefers finding problems before they reach business results. Model accuracy and business performance can diverge, so the team compares metrics from different stages before deciding where to investigate.
The design reduces repeated computation and separates team tasks
Nicolas does not give a precise current savings figure, but he says feature computation previously cost several hundred dollars per month when calculated repeatedly during experiments and training. The new design computes features once per day, after which data scientists can access them for training. Its larger benefit is how it separates feature engineering from model training. A data scientist can spend time creating and checking features in the shared table, then switch attention to model structure and hyperparameters. Nicolas says these tasks require different kinds of thinking, so keeping them separate makes the work clearer.
Real-time requirements would push BigQuery beyond this design
Nicolas says the architecture works well for batch feature computation, but BigQuery would not be the best fit for live, on-the-fly features. Serving data that is too large for memory would need a low-latency database such as Redis. Features that must update more often than daily would create another challenge. Malt may use its streaming ingestion capability and consider microbatching, but Nicolas does not yet have a final design for writing those computations into the feature table. The future pressure points are therefore serving-data size and live feature computation.
