Lyft processes streaming marketplace data into features for forecasting, surge pricing, and other models, usually with one-minute aggregation windows.
2
Geohash-based partitioning reduces hot shards caused by uneven traffic across cities, while early filtering and divided joins keep processing manageable.
3
Lyft compares real-time features with offline data, routes simple cases through self-service tools, and uses specialized pipelines for more complex workloads.
Summary
Rakesh Kumar explains how Lyft generates features from marketplace events for use cases such as demand and supply forecasting and surge pricing. The platform processes data asynchronously, most often in one-minute windows, then aggregates it by geohash so traffic from large cities does not overload individual workers. Lyft moved from cron jobs to Apache Beam running with Apache Flink, then refined the design by splitting operators, redistributing data, dropping irrelevant events early, and dividing large joins into smaller operations. The team supports both self-service pipelines and specialized implementations. Data scientists can develop and test features offline before Lyft converts them to real-time pipelines. An internal comparison framework checks real-time features against offline data and alerts on differences. Kumar also describes a geospatial feature store that supports region, sub-region, and geohash-level access. The discussion includes cost checks, technology selection, and a YAML-based pipeline system that hides much of the streaming implementation.
Lyft uses real-time features for marketplace decisions
Rakesh Kumar says Lyft processes data from user devices and services, transforms and aggregates it, then passes the results to machine learning models. He gives demand and supply forecasting as one example. Models forecast conditions for a region, city, or venue at a specific time. Surge pricing is another example. Lyft monitors the marketplace for an imbalance between demand and supply, then adjusts prices, such as raising them when too few drivers are available in an area. These are among the business cases that depend on current marketplace data.
Most Lyft real-time processing uses asynchronous one-minute windows
Kumar distinguishes offline feature generation from real-time generation. Offline jobs have more time to process data, while real-time jobs may need to process millions of data points with second or subsecond latency. For this discussion, he defines real time mainly as asynchronous processing rather than a synchronous model call. Some use cases use one-minute or five-minute windows, and most of Lyft's use cases use one-minute aggregated data. The required window depends on the model and the volume of data being processed.
Longer aggregation windows increase memory use and can stop scaling
Kumar says smaller windows are easier to aggregate because less data accumulates before emission. Longer windows keep data in memory for more time, which increases processing and memory use. Lyft had use cases where a 30-minute aggregation worked functionally but did not scale. He describes one-minute and five-minute windows as practical choices. The reason for keeping latency low is model performance. If the pipeline falls behind, the features may describe an older marketplace state and the model's predictions can become less accurate.
The first version of the system used cron jobs that started at the top of each minute. A job read data from Kinesis, processed it, and stored it in a temporary location, most likely Redis, before the next job continued the work. Even when one step finished in a second or two, the next step waited for its scheduled start. As Lyft grew, this built-in delay limited scalability and model performance. The team moved to streaming processing, where downstream operators receive data as soon as an upstream operator has processed it. Lyft used Apache Beam with Apache Flink underneath and selected the Apache Beam Python SDK because Lyft's services and models were already written in Python.
Geohash partitioning prevents busy cities from creating hot shards
The initial streaming design routed all traffic from a city to one shard or node. That created an uneven workload because a small number of major cities generated most of the traffic. Some nodes reached about 90 percent CPU utilization while others were around 10 percent. Lyft changed the partitioning unit from city to geohash, which Kumar describes as roughly the size of a city block. Events are routed by geohash, producing millions of more evenly distributed partitions. After the heavy filtering and aggregation work is complete, the smaller result set can be reshared to regions without recreating the same hot-shard problem.
Offline feature work becomes a real-time pipeline through a contract
Lyft supports simple aggregations through an off-the-shelf pipeline and helps with specialized features that contain more complicated business logic. Data scientists first build features from offline data, test their models, and then ask Lyft to productionize a feature in real time. Kumar says the team converts the offline query into a pipeline implementation. The contract is that the real-time feature should match the offline result at about 99.9 percent. Lyft samples real-time and offline features, compares them, and alerts when they differ. This gives the team a way to investigate missing or delayed data and other causes of mismatch.
Lyft's feature store is organized around geographic hierarchies
Kumar says traditional feature stores are suited to point features, such as a value associated with one user. Lyft's marketplace models often need aggregated values for a city or region, including driver counts and ride requests across many geohashes. Lyft built a geospatial feature store that can return aggregated values at a requested level, such as a region, GH4, or GH5. The underlying data has a hierarchical structure. Lyft stores the region-level view, stores other data at the GH6 level, and maintains mappings between levels. Different models can consume the same generated feature at region, sub-region, or geohash granularity.
Kumar says every request for a real-time feature is reviewed to determine whether real-time freshness is actually needed. If an offline feature is sufficient, Lyft directs the requester to offline data stores. Real-time processing costs more effort and money, so the platform team does not want its abstraction to make it appear free. This review is part of onboarding. Simple use cases can be handled through self-service SDKs, while specialized features receive support from engineers who understand the scaling and partitioning requirements.
Shared preprocessing and YAML definitions reduce pipeline work
After creating multiple pipelines, Lyft found that many repeated the same early processing. The team created a two-stage design. A shared preprocessing stage filters events and builds user state, while downstream pipelines apply feature-specific aggregation logic. This avoids repeating common work for every feature. Lyft also created a declarative YAML-based pipeline system. Users choose existing functions and declare windowing, such as a one-minute or two-minute window, without writing the streaming implementation. Kumar says an earlier design could require thousands of lines of code for one feature, while the YAML definition can be about 30 or 40 lines.
