# "Real-Time" ML: Features and Inference

Sasha Ovsankin & Rupesh Gupta, LinkedIn | MLOps Podcast | Episode 135 | 51:55
Hosted by Skylar Payne

Source: https://www.youtube.com/watch?v=cZpGgobIFxU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/real-time-ml-features-and-inference
Published: 2022-12-09
Tags: data-pipelines, feature-engineering, platform-teams, recommender-systems

## TL;DR
- Near-real-time features let recommender systems react to a user's current intent within seconds or minutes, while real-time inference usually refers to millisecond-level responses.
- Teams should add real-time features after improving their models and batch features, and should usually add real-time versions of important batch features alongside the existing historical features.
- Real-time feature systems are difficult because machine learning engineers must work with distributed streaming infrastructure, operational support, and consistency between batch training data and online inference data.

## Summary
Rupesh Gupta explains how recommender systems use recent user actions to infer changing intent and preferences. Batch pipelines can leave a delay of hours or days, while near-real-time feature generation can update recommendations within seconds or minutes. The guests distinguish this from real-time inference, which concerns millisecond responses. They advise teams to optimize models and batch features first, then add real-time versions of features that already matter. Historical and recent features should work together so a recommender can balance long-term interests with short-term needs. Sasha Ovsankin describes LinkedIn's difficulties with streaming processors, including the operational burden on machine learning engineers and the need for managed infrastructure. They discuss Apache Pinot, precomputed feature pipelines, ease of use, portability between batch and streaming, and the need for training-serving consistency. Rupesh gives examples of gains in job, feed, and search recommendation systems, with newer and less active members benefiting most.

## Key ideas
### Recent actions let recommenders respond to changing intent
[10:10](https://www.youtube.com/watch?v=cZpGgobIFxU&t=610s)
Rupesh Gupta says recommender systems infer intent and preference from actions because users rarely state them directly. Batch processing can create a delay of several hours or even days before an action becomes a feature. A person who previously searched for web developer jobs in San Francisco may begin searching in New York, while the recommender continues using the older preference. Near-real-time features reduce this delay, allowing the system to adapt recommendations to what the user is doing now.

### Feature values summarize recent actions into numbers or vectors
[14:37](https://www.youtube.com/watch?v=cZpGgobIFxU&t=877s)
The common approach is to turn recent actions into features that a model can use. Rupesh gives the example of collecting jobs a user applied to during the previous 24 hours, obtaining an embedding for each job, and averaging those embeddings. The resulting vector represents the kinds of jobs the user preferred recently. A simpler feature might count applications to jobs in San Francisco or New York during the same period. The summary makes the action history easier for a model to process.

### Near-real-time features usually mean seconds or minutes
[18:47](https://www.youtube.com/watch?v=cZpGgobIFxU&t=1127s)
Sasha Ovsankin separates several meanings of real time. In this conversation, the focus is near-real-time generation of feature values from recent events. That usually means a delay of seconds to minutes and often involves streaming systems. He distinguishes this from real-time inference, which he describes as operating at millisecond scale, and from real-time systems such as robotics and self-driving cars.

### Teams should add real-time features after fixing basic model problems
[20:28](https://www.youtube.com/watch?v=cZpGgobIFxU&t=1228s)
Rupesh argues that real-time features are useful for recommender systems that are already well optimized. Teams should first improve the model structure and the existing batch-computed features. Since adding streaming features takes real effort, they also need a way to decide which features deserve that effort. His practical rule is to start with real-time versions of batch features that are already important to the model.

### Recent and historical features need to work together
[22:21](https://www.youtube.com/watch?v=cZpGgobIFxU&t=1341s)
Rupesh says real-time features should complement batch features rather than replace them. A model might use the average embedding of jobs applied to today from a real-time pipeline, while still using the average embedding of jobs applied to during the previous day or month from a batch pipeline. These separate time windows help the recommender balance short-term intent with long-term intent. Historical data is especially limited for new users, so their current-session activity becomes more useful.

### Streaming infrastructure creates a large operational burden
[25:08](https://www.youtube.com/watch?v=cZpGgobIFxU&t=1508s)
Sasha describes the difficulty of making real-time features practical at LinkedIn. Existing streaming processors could support some use cases, but machine learning engineers found them hard to run. Apache Pinot helped with use cases where the system could aggregate a manageable amount of member activity. Larger aggregations over jobs or content required precomputed feature processors. Those processors introduce distributed-systems failures, hardware problems, and many separate services that need specialist support.

### Managed services and simple configuration improve adoption
[41:24](https://www.youtube.com/watch?v=cZpGgobIFxU&t=2484s)
Sasha says LinkedIn's streaming team developed a managed service so machine learning engineers could run feature pipelines without operating the underlying infrastructure. Rupesh adds that machine learning engineers are more likely to adopt real-time features when access is exposed through a simple configuration rather than requiring them to learn infrastructure work. The division of responsibility between application, infrastructure, and machine learning engineers is part of the system design.

### Batch and streaming definitions need portability and consistency
[48:59](https://www.youtube.com/watch?v=cZpGgobIFxU&t=2939s)
Sasha identifies portability between streaming and batch as an under-discussed problem. Ideally, a feature definition can work in both modes. The features used during training also need to match the features retrieved during inference. Without that consistency, the model is trained on data that does not match what it sees in production. He says progress exists, but the problem still needs more work.

## Notable quotes
- Rupesh Gupta: "The goal, the ultimate goal is to reduce the delay between when user takes an action and when it can be leveraged in the recommender systems to adapt recommendations." (12:39)
- Sasha Ovsankin: "Near real time, approximately you can think about like seconds to minutes." (19:18)
- Rupesh Gupta: "These real-time features should always complement the existing batch computed features. They should not replace them." (22:42)
- Sasha Ovsankin: "I don't care about your infra. Well, I do care about it, but I don't know that much about the infra and I don't want to spend my time responding to these on-call events in the middle of the night." (35:50)
- Sasha Ovsankin: "My personal pet peeve is portability, so in particular portability between streaming and batch." (49:39)

## Tools & references mentioned
- LinkedIn
- Feathr
- Apache Kafka
- Apache Pinot
- Samsa
- StarTree
- Benjamin Lee
- Alex Batty

## Who should watch
- You are deciding whether a recommender system needs near-real-time features and want a concrete rule for choosing which features to stream.
- Your team has batch pipelines and is worried about the operational cost of adding streaming processors.
- You need training-serving consistency across batch and streaming feature computation, especially for new users with little historical data.

## Editor's note

Sasha Ovsankin says LinkedIn's streaming feature systems create an operational burden for machine learning engineers, while training and inference also need consistent feature definitions. ZenML lets teams write pipelines as Python steps and choose the infrastructure stack by configuration, so the same workflow code can run across different environments. Each run records its inputs, outputs, and code version for tracing.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)
- [Real-time Machine Learning](https://mlopstalks.com/talks/real-time-machine-learning) (Chip Huyen, Claypot AI, 58:24)
- [Recommender System: Why They Update Models 100 Times a Day](https://mlopstalks.com/talks/recommender-system-why-they-update-models-100-times-a-day) (Gleb Abroskin, FunCorp, 49:01)
- [Real-time features, AI search, Agentic similarities](https://mlopstalks.com/talks/real-time-features-ai-search-agentic-similarities) (Varant Zanoyan & Nikhil Simha Raprolu, Zipline AI, 29:28)
- [Feathr: LinkedIn's High-performance Feature Store](https://mlopstalks.com/talks/feathr-linkedins-high-performance-feature-store) (David Stein, LinkedIn, 53:15)