David Stein built Feathr to simplify feature preparation by letting teams define named features and use them consistently in training and inference.
2
Feathr grew inside LinkedIn through close work with machine learning teams, which gave its builders access to users, code, and varied production problems.
3
Real-time features can add value, but many machine learning problems do not change quickly enough to justify the cost and complexity of real-time processing.
Summary
David Stein explains how Feathr grew from an internal LinkedIn project into an open source feature store. LinkedIn had already been running recommendation, advertising, notification, and relevance systems for more than a decade, but teams struggled with large feature preparation workflows that had accumulated over time. Feathr lets engineers define named features, register them, and use those names in both training and inference instead of managing application-specific data workflows. Stein argues that clean concepts and user feedback matter early because unclear abstractions become expensive to fix later. He describes Feathr as more than a data store, with an expression language for feature transformations and support for real-time computation. He also explains why LinkedIn was a good place to incubate the project: the team could work directly with many internal users and see their code. Open sourcing required removing internal dependencies, improving documentation, and working with Microsoft Azure. Stein says he would have preferred to open source Feathr earlier, although LinkedIn's internal priorities delayed the release.
Feathr began as a response to LinkedIn's growing feature preparation burden
LinkedIn had been using machine learning in production for recommendation systems, advertising, feed ranking, and notification relevance for more than 10 years. Stein says that preparing and managing the features feeding those systems became one of the hardest parts of operating large machine learning applications. Workflows accumulated through team turnover, new features, joins, and successive generations of engineers. Feathr was created to reduce that burden. Its basic model is simple: define a named set of features, compute them from source data, and make them available in both training and production inference contexts.
Clean concepts help a platform avoid expensive redesigns
Stein says the most important design lesson was to establish clean concepts that make sense to machine learning engineers. Named things should match the work users need to do, and teams should ask users how they want to specify feature definitions and dependencies. He describes a configuration language or DSL as a way to test and shape those concepts before implementation hardens them. If a platform makes its core ideas too complicated, the cost of correcting them rises over time. Feathr's design work focused on making the system easier to reason about, rather than exposing every underlying processing detail.
A feature registry separates feature meaning from model code
Feathr treats a feature as a named entity that can be registered independently of the code that creates it and the models that consume it. This lets engineers use the same feature name in experimentation, offline training, and online production serving. Before this approach, feature names could be coupled to model code or data-processing workflows, with engineers responsible for keeping columns aligned across contexts. Stein compares importing registered features to importing dependencies at the top of a source file. The registry gives teams a shared name and meaning while the infrastructure handles how the feature is prepared and delivered.
Feathr's open source release required separating it from LinkedIn's internal systems
LinkedIn open sourced Feathr because the team believed its approach to organizing feature preparation could help companies outside LinkedIn. Stein says LinkedIn has a broader culture of open sourcing infrastructure, with Kafka, Pinot, and DataHub as examples he names. The release also created a reason to clean up the project, improve documentation, and remove internal dependencies that had accumulated over years. Microsoft Azure worked with the team on making Feathr easier to try and deploy in Azure. Stein describes that work as a substantial effort because a long-lived internal project carries many relationships with company-specific technology.
Feature stores differ widely in how much they do beyond storage
Stein says the meaning of feature store was still developing across the industry. Some products mainly store computed feature data for online access. Others let users define feature views, transform raw data, work with time windows, or build features from other features. Feathr aims to cover the common feature-engineering work through an expression language and APIs, which Stein compares to a Swiss army knife. He wants the standard capabilities to cover most needs, with user-defined functions for the remainder. In his view, a feature store needs to include the feature definition and preparation work, not only a place to put finished data.
Real-time features are limited by the difficulty of trying them
Stein expects feature infrastructure to improve through better visualization of feature trends, anomaly detection, and support for dynamic time-windowed features. He sees real-time signals as an underused area because stream processing remains difficult for machine learning engineers to pick up and apply. People are more likely to try an idea when they can build it reliably and learn its effect within their planning cycle. If testing a signal takes too long or requires too much specialist infrastructure, teams may avoid the experiment even when the signal could help. Feathr is intended to reduce that barrier by making real-time computation easier to define, backfill, simulate, and deploy.
Real-time processing depends on the prediction problem
Skylar Payne points out that some problems benefit greatly from immediate signals while others change slowly. He uses mental health as an example where a suicidal indication should be available quickly, while a person's broader mental health may not change much over an hour. Stein agrees that real-time value varies by use case. LinkedIn already has real-time processing for cases where its value is established, including situations involving recent user behavior and personalization. His infrastructure focus is on making experimentation cheap enough that teams can discover useful real-time applications instead of requiring them to commit to a difficult system before they know whether it will help.
LinkedIn gave Feathr better access to users than a separate startup would have
Stein rejects the idea that Feathr needs to become a separate company. LinkedIn has many machine learning teams working across different business areas and problem sizes, while employees can still build close relationships with users and see how they work. Stein knew Skylar Payne because Skylar had been an internal Feathr user, which gave the team direct feedback and access to real code. That environment let the project grow against varied production needs. He says he could build a better product at LinkedIn than by immediately seeking funding and working outside the company. He does say that open sourcing Feathr earlier would have been preferable.
"The idea that there should be a named abstract entity, a feature that is registered in a registry and has a name, was sort of an innovation really before feature stores and before Feathr."David Stein22:01
Who should watch
You are deciding whether a feature store should manage only storage or also feature definitions and transformations.
Your team has feature pipelines that have grown through years of additions, joins, and handoffs between engineers.
You are considering real-time features and need to judge whether the prediction problem changes quickly enough to justify the infrastructure cost.
