Real-time AI/ML depends first on very low latency, then on scaling throughput, data volume, availability, accuracy, and cost together.
2
The online feature store is often the largest source of latency that teams can optimize, so its storage technology and data structures matter.
3
Feature stores such as Feast and Feathr help prevent training-serving skew and let data scientists reuse feature definitions across use cases.
Summary
Nava Levy explains why real-time machine learning becomes difficult when a prediction must use fresh data inside an interactive application. A 100-millisecond end-to-end target may leave only half a millisecond for the underlying data store after network, application, feature-serving, and model-scoring time are included. Once traffic reaches hundreds of thousands or millions of requests per second, teams also need to handle terabytes of data, high availability, accuracy, global distribution, and operating cost. Levy uses Redis customer examples, including Ekata, AT&T, DoorDash, Twitter, Uber, Tecton, and Wix, to discuss online-store choices and data structures. She describes the split between online and offline feature stores, along with a registry and orchestration layer that helps prevent training-serving skew. The talk closes with practical starting points using Feast and Feathr, including Redis integrations and Azure tutorials.
Real-time prediction has to fit inside the user's interaction
Levy defines real-time AI/ML as scoring a request with fresh, relevant data as soon as an operational system or application sends it. The prediction runs in the background of an interactive action, such as an online purchase, without the user noticing a delay. She calls latency the basic prerequisite for a real-time use case and gives a range of 50 to 200 milliseconds for online scoring. In an earlier fraud example, a company moved from batch scoring every 40 minutes to 12 seconds, which allowed fraud detection to become prevention, but that delay was still too long to sit inside a transaction flow.
The data store receives only a small part of the end-to-end latency budget
The total response time includes network latency, the application pipeline, feature serving, and model scoring. Levy gives network latency as roughly 50 to 70 milliseconds in one example. After a model works for one prediction, the harder problem is handling high throughput, potentially thousands or millions of requests per second, while the feature data reaches terabytes. The system must also preserve model accuracy between training and production. Levy describes latency as the real-time system constraint and accuracy as the data science constraint, with some skew expected but a need to reduce it.
Production real-time ML adds operational requirements as the use case becomes valuable
Levy compares machine learning engineering to Dr. Seuss's Cat in the Hat adding more objects to a juggling act. A successful low-latency and accurate model becomes part of a customer-facing, mission-critical flow. That raises the need for high availability, persistence, manageable scaling, cost control, and global distribution. She mentions disaster recovery, network distance, and regulation as reasons a deployment may need to be distributed. A failure is more serious than dropping an egg during a performance because lost data, downtime, or a slow prediction can affect a customer's experience or allow fraud.
Ekata's latency target makes the online store constraint visible
Levy uses Ekata's transaction risk API to show how an end-to-end target becomes a much smaller data-store target. The API requires 100 milliseconds end to end, while subgraph retrieval must take 20 milliseconds. That leaves the data store with a latency target below half a millisecond. The use case also handles hundreds of thousands of products per second and more than 10 terabytes of data, with high availability and stability requirements. Ekata implements its identity graph with a Redis sorted set, using a data structure suited to that access pattern.
Online-store technology and data structures affect both speed and cost
Levy presents benchmarks from Uber, Feast, Tecton, DoorDash, and Twitter to argue that an in-memory online store can be cost-effective when a workload has strict latency or throughput requirements. She says Tecton found Redis three times faster and 14 times less expensive than DynamoDB in its comparison. DoorDash improved latency, CPU efficiency, and memory use by replacing multiple flat key-value reads with Redis hashes and one HMGET command for a feature vector. Levy's point is that selecting Redis is only part of the decision. The data structure and access pattern also need to match the workload.
A feature store separates fast production serving from historical training data
Levy describes an online feature store for the latest values needed by production predictions and an offline feature store for static and historical data used in training and batch scoring. The offline side may use Snowflake, Redshift, S3, BigQuery, or Azure, while the online side needs low-latency access. A management layer handles the feature registry, transformation logic, and monitoring. She also describes streaming sources and time-to-live settings, which can expire stale features and free online storage. The architecture has to account for the different latency requirements of training and online serving.
Feature definitions can reduce training-serving skew and repeated engineering work
A feature store defines a feature and its transformation logic in a registry, then reuses that definition for training and production serving. Levy says this helps avoid training-serving skew, although some skew will always exist. Registered features can also be reused by other machine learning use cases and data scientists. She notes that data scientists spend more than half their time on feature engineering and data management, so sharing existing features can prevent repeated work. For real-time systems, she treats fast serving and avoiding skew as the two most important feature-store objectives.
Feast and Feathr provide practical entry points for experimenting with feature stores
Levy recommends trying the open-source feature stores Feast and Feathr. She describes Feast as lightweight enough to run locally, including for a data scientist working alone, and points to a tutorial using Redis and another for Azure. Feathr is presented as a recently open-sourced system that had been used inside LinkedIn for several years. Both can work with Azure Cache for Redis and support Redis Enterprise and Redis on Flash options. The examples are intended to let teams try the architecture without first deploying a large amount of infrastructure.
"The end-to-end latency includes the following key sources: there is the network latency, there is application pipeline, feature serving, and also the scoring."Nava Levy13:40
Who should watch
You are designing an online prediction path where fresh features must be retrieved within a tight latency budget.
Your team is deciding how to split online and offline feature storage, or is seeing the online store dominate cost.
You want a small, practical way to try Feast or Feathr with Redis before building a larger feature platform.
