# Lightweight Feature Platform

Matt Bleifer & Mike Eastham, Tecton | MLOps Podcast | Episode 209 | 1:03:58
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=TU7HDABYjXs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/lightweight-feature-platform
Published: 2024-02-06
Tags: data-pipelines, data-quality, feature-engineering, feature-stores, platform-teams

## TL;DR
- Tecton expanded from a feature store into a feature platform that also handles feature pipelines, serving, monitoring, and collaboration.
- Rift gives teams a Python-native compute engine that can use pandas and connect to warehouses such as Snowflake without requiring Spark or external infrastructure.
- Matt Bleifer and Mike Eastham see feature platforms as part of the same production pattern as LLM applications, where relevant data is prepared, sent to a model, and returned to an application.

## Summary
Matt Bleifer and Mike Eastham describe how Tecton evolved from a feature store into a broader feature platform. The system helps teams define features, build and run pipelines, serve values for training and inference, monitor data, and share features across teams. They explain why Tecton added Rift, a Python-native compute engine built around Arrow and using DuckDB. Rift is intended for teams that work with Python, pandas, notebooks, or data warehouses and do not need the operational burden of Spark. Spark remains available for very large workloads. Mike explains the choice to keep stream ingestion stateless and perform aggregation at observation time, with compaction planned for larger time series. The conversation also connects feature platforms to LLM applications and retrieval-augmented generation. Matt expects Tecton to manage more of these real-time decision pipelines, including prompts and multi-stage recommendation ranking.

## Key ideas
### Tecton grew beyond storage because feature engineering creates most of the operational work
[13:25](https://www.youtube.com/watch?v=TU7HDABYjXs&t=805s)
Matt Bleifer explains that a feature store originally meant keeping feature values for offline training and low-latency online inference. Customer conversations showed that storage was only part of the problem. Teams also needed help building streaming and batch pipelines, orchestrating backfills, handling request-time logic, monitoring skew and data quality, and managing sharing and access controls. Tecton eventually called the broader product a feature platform, with the feature store as its storage and serving component. Matt says this framing better matches the work required to support real-time AI in an organization.

### Rift lets Python teams build features without taking on Spark
[21:14](https://www.youtube.com/watch?v=TU7HDABYjXs&t=1274s)
Matt says Tecton was historically Spark-centric because early customers had large, real-time workloads and already knew Spark. Market conversations showed that Spark was overkill or unfamiliar for roughly half of the organizations they spoke with. Rift addresses that by providing a built-in, Python-native compute engine. Teams can use Python and pandas transformations, work from a Hex, Deepnote, or local Jupyter notebook, and install Tecton with pip. Rift can also push initial transformations into a warehouse such as Snowflake, then apply further logic without requiring separate compute infrastructure.

### Rift uses Arrow to connect warehouses and processing components
[28:15](https://www.youtube.com/watch?v=TU7HDABYjXs&t=1695s)
Mike Eastham describes Rift as a modular system built around Apache Arrow for exchanging data between stages. A batch workflow might start with a Snowflake table, apply filters or projections in Snowflake, stream the result into DuckDB, run aggregations there, and upload the resulting Arrow data to Tecton's online database. Tecton also supports an HTTP interface for streams of raw rows, applying related transformations as data arrives. This design lets the platform work with different warehouses and data-processing libraries while keeping the internal interfaces consistent.

### Stateless stream ingestion reduces operational failure modes
[30:47](https://www.youtube.com/watch?v=TU7HDABYjXs&t=1847s)
Mike says Tecton chose a simplified stream architecture instead of relying on a stateful engine such as Flink for every aggregation. Ingestion can filter and project data, but aggregation happens when the feature is observed. That removes the need to maintain large streaming query states and avoids problems such as checkpoint storage, disk exhaustion, and checkpoint commit latency. Mike says the simpler critical path makes the managed service easier to operate for high-availability use cases such as fraud detection. The trade-off is that very large aggregations can hurt performance, which is why Tecton is working on compaction to aggregate older data later.

### Vertical scaling covers more workloads than teams often assume
[34:03](https://www.youtube.com/watch?v=TU7HDABYjXs&t=2043s)
Mike says the current Rift architecture can aggregate a single series with tens of thousands of data points, and sometimes low hundreds of thousands, depending on the latency budget. For larger series, compaction is intended to help. For batch processing, DuckDB and Arrow use memory more efficiently than pandas and can process data in chunks when it does not fit in memory. Matt also points to the argument that many companies have smaller datasets than the big-data narrative assumes. Tecton can split backfill ranges and scale the materialization machine vertically, which they say works for most use cases without distributed processing.

### The product keeps Spark as an escape hatch for extreme workloads
[43:24](https://www.youtube.com/watch?v=TU7HDABYjXs&t=2604s)
Matt describes a design principle of making simple work easy while keeping difficult workloads possible. Rift is meant to cover the common case for Python and warehouse-oriented teams, while Spark remains available when an organization needs much larger processing capacity. This option also made it easier for Tecton to simplify Rift without trying to optimize it for every extreme. Matt says a single company may contain Spark teams, warehouse teams, and Python users, so the tools need to work together under one Tecton product rather than forcing the organization into separate product versions.

### Feature platforms and LLM applications share the same production pattern
[52:09](https://www.youtube.com/watch?v=TU7HDABYjXs&t=3129s)
Matt compares traditional predictive ML with LLM-backed software. In both cases, raw or business data is turned into relevant information, passed into a model, and used to change application behavior. In retrieval-augmented generation, features become embeddings, a key-value store may become a vector database, and model output may be text rather than a prediction. He suggests that a feature platform could provide user context for prompts, store prompts as code, inject retrieved feature values, call an LLM, parse the response, and return it through one API. He also mentions the possibility of generating feature definitions from human-readable descriptions.

### Tecton plans to make real-time decisioning easier to try and manage
[57:28](https://www.youtube.com/watch?v=TU7HDABYjXs&t=3448s)
Matt says Tecton is investing in LLM use cases, continued Rift development, and lowering the barrier to entry for feature platforms. He expects the product to expand from managing features toward managing complete real-time decision pipelines, including LLM prompts and multi-stage ranking for recommendation systems. Mike adds that the team is working to make Tecton easier to test drive because access has historically required a more involved sales process. The simple pip-install experience for Rift supports that goal, although the team still has to address unexpected problems found by early users.

## Notable quotes
- Mike Eastham: "You just want to express here's semantically what the feature means, here's what my requirements are, and then you want the system to worry about the architecture." (20:01)
- Matt Bleifer: "You can literally hop into a Hex notebook or a Deepnote notebook or even a local Jupyter notebook, pip install Tecton and do all of your feature development there with Python as your only dependency." (22:50)
- Mike Eastham: "The critical path from data going in to being available at serving time is just dead simple." (32:15)
- Matt Bleifer: "The simple things should be simple, but the hard things possible." (46:29)
- Matt Bleifer: "There is no fundamental difference in taking data, giving it to a model, taking the results and changing the behavior of your application." (56:52)

## Tools & references mentioned
- Tecton
- Rift
- Spark
- Snowflake
- DuckDB
- Arrow
- pandas
- Flink
- Kafka
- Hex
- Deepnote
- Jupyter
- Twitter
- Google
- Uber
- Michelangelo
- MotherDuck
- LLMs
- retrieval-augmented generation

## Who should watch
- You are deciding whether a feature platform should hide compute and orchestration details from data scientists and ML engineers.
- Your team uses Spark for workloads that may fit better in Python, a warehouse, or a single scalable machine.
- You are designing production LLM or retrieval-augmented generation systems and want to compare their data flow with traditional feature-based ML.

## Editor's note

Matt Bleifer says teams needed help building streaming and batch pipelines, orchestrating backfills, and managing real-time feature work beyond storing values. ZenML lets teams write ML workflows as Python pipelines whose runs record steps, inputs, outputs, and code versions, so teams can trace what produced an artifact or model and reuse the same workflow across infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [AI Innovations: The Power of Feature Platforms](https://mlopstalks.com/talks/ai-innovations-the-power-of-feature-platforms) (Mahesh Murag, Tecton & Jose Navarro, Cleo & Nikhil Garg, Fennel, 1:05:28)
- [Building a Python-Centric Feature Platform to Power Production AI Applications](https://mlopstalks.com/talks/building-a-python-centric-feature-platform-to-power-production-ai-applications) (Matt Bleifer, Tecton, 27:11)
- [Machine Learning at Atlassian](https://mlopstalks.com/talks/machine-learning-at-atlassian) (Geoff Sims, Atlassian, 58:23)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)