# Real-Time Forecasting Faceoff: Time Series vs. DNNs

Josh Xi, Lyft | MLOps Podcast | Episode 305 | 53:42
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=zeTnQAKb_2U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/real-time-forecasting-faceoff-time-series-vs-dnns
Published: 2025-04-11
Tags: classical-ml, continual-learning, forecasting, monitoring

## TL;DR
- Lyft uses time series models for highly granular forecasts because recent history often captures short-term marketplace behavior and the models can be retrained quickly.
- Deep neural networks cost much more to train, need more historical data, and are harder to retrain online when demand patterns change.
- Forecasting teams handle unusual events with bias correction, manual adjustments in slower offline forecasts, and ensembles of small models weighted by recent performance.

## Summary
Josh Xi describes forecasting demand and supply for Lyft's marketplace at geohash level. The system may forecast every five minutes through the next hour across thousands of cells, so speed and adaptation matter as much as model complexity. Josh argues that autoregressive and other classical time series models often work better in this setting because recent history captures short-term repetition, the models are easy to interpret, and their weights can be refit online. Spatial-temporal DNNs can learn relationships between nearby cells, but they need more data and make training and retraining far more expensive. External events, weather, venue operations, and changing travel behavior remain difficult for either approach. Lyft responds with recent-error bias corrections, regional offline forecasts that include human adjustments, and weighted ensembles. Model comparisons rely on back testing, while production monitoring focuses on regional errors and downstream effects on pricing and driver incentives.

## Key ideas
### Lyft forecasts marketplace signals at geohash level
[01:19](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=79s)
Josh Xi works on Lyft's Marketplace team, which supplies signals for pricing and driver incentives. The team forecasts demand and supply so the marketplace can balance riders and drivers. Its real-time system works at geohash six, roughly a one-by-one-mile cell, although the exact aggregation depends on the use case. A city can contain a few thousand or even 10,000 geohashes. Lyft generates forecasts for every cell from five minutes ahead to an hour, producing millions of values at once. That spatial detail lets the system direct drivers toward specific busy areas instead of only forecasting demand for an entire region.

### External event data is useful but difficult to turn into reliable features
[04:58](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=298s)
Lyft uses event information, flight arrivals and departures, and weather as possible external inputs. Event start times are usually available, while end times are uncertain. A football game can finish in two minutes, take much longer, or go into overtime. Weather effects also differ from simple assumptions. Josh found that precipitation and temperature features had less correlation with demand than expected. People may respond to the difference between the forecast and the actual weather. A particularly bad winter also weakened the relationship between snow and taxi demand because roads stopped being usable and people stayed home.

### Classical time series models fit the real-time problem because they are fast and adjustable
[11:57](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=717s)
Josh says autoregressive models often work better in practice because they focus on recent history, such as the same time yesterday or last week. Short-term behavior can repeat even when the overall market is changing. These models are interpretable because the forecast is a weighted combination of historical values. A human can adjust the forecast when a known Super Bowl or local event is missing from the model. Decomposition models make this easier by separating trend and seasonality, allowing an event spike to be added to the appropriate part. DNNs can also be adjusted, but it is harder to know whether they already captured an event or which part of the prediction should change.

### Time series foundation models fit slower regional forecasts better than minute-level spatial forecasts
[16:11](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=971s)
Josh has limited experience with time series foundation models and has heard more from coworkers. His description is that they work better for regional or subregional forecasts made hourly or daily for the coming week or two. They have performed less well for Lyft's highly granular real-time use case, which refreshes every minute across thousands of cells. The models also appear too large to start and run quickly at that cadence. This creates a serving problem even before considering whether accuracy improves. The distinction is between a slower forecast for a region and a fast forecast for many spatial cells.

### DNN training costs far more than autoregressive training
[19:19](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=1159s)
Lyft's comparison uses autoregressive models and spatial-temporal DNNs. The DNN must learn relationships across many cells, while a single-series LSTM does not handle that spatial structure. Josh says autoregressive training runs on one CPU, while spatial DNN training uses a GPU. From Lyft's experience, training cost can differ by at least 100 times. Inference is less different because a model can load precomputed weights on a regular machine. Lyft's real-time cadence is one forecast each minute for roughly the next 30 minutes, and finishing within about 30 seconds is sufficient. Classical models have much smaller weights and remain faster.

### Online refitting gives classical models a practical adaptation advantage
[23:18](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=1398s)
Autoregressive models can be refit quickly because they contain only a small number of weights tied to historical features. When a new observation arrives, Lyft can update the model and change the balance between recent values and older history. That lets the forecast react to a spike in the marketplace. A DNN can also be retrained, but the process is expensive, so it can happen less often. Josh says that lower retraining frequency can mean lower accuracy when conditions are changing. Lyft currently retrains its real-time models every minute, accepting the machine cost because the machines remain available for this work.

### Recent forecast errors support automated bias correction
[27:26](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=1646s)
Josh says the online autoregressive models usually capture demand spikes, although they can react with a delay. The right learning-rate settings differ by region, and the team does not yet have a reliable way to tune every model perfectly. Instead of asking people to inspect thousands of cells, Lyft can examine recent forecast errors. If the forecast has been consistently below actual demand, a heuristic can apply a bias correction. Josh gives the example of dividing by 0.9 when forecasts have recently been about 90 percent of actual demand. The team also considers ensembles of small models, weighting the better recent performers more heavily than weaker ones.

### Offline regional forecasts can add event knowledge to real-time forecasts
[28:57](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=1737s)
Lyft's slower forecasting work operates at regional or subregional level and can include longer histories, explicit events, seasonality, and human review. Those models may use two years of data and run daily or weekly for the following week or two, so their compute costs are less restrictive. Josh describes a possible connection between the systems: aggregate the real-time cell forecasts for an area, compare that result with the adjusted offline regional forecast, and use the difference as a multiplier or other correction. This gives the real-time layer information about major events that are hard to learn from only the previous week or two. Events that occur only a few times per year still often need human intervention in the offline process.

### Back testing is the standard way to compare forecasting models
[38:53](https://www.youtube.com/watch?v=zeTnQAKb_2U&t=2333s)
Josh describes back testing as simulating a model from a point in the past. The team trains as if it were at that historical moment, moves forward through the known timeline, and compares each forecast with what actually happened. Since the outcomes are already known, the team can calculate bias and other accuracy metrics. Lyft uses this process when changing an existing model or evaluating a new one. The spatial part makes DNNs harder because the model must learn relationships among cells, often treating the region like an image with latitude and longitude axes. Convolutional layers can learn local patterns, but those spatial relationships may change because of construction, office moves, venue operations, pandemics, or economic changes.

## Notable quotes
- Josh Xi: "So every city you typically have like a couple thousand sometime to even like a 10,000 geohashes and so basically for every geohash we need to generate a forecast in real time for the next say five minute up to an hour." (03:09)
- Josh Xi: "So you can do online learning make the model sort of adaptive to whatever it's changing in the real real time marketplace." (24:18)
- Josh Xi: "So the adjustment can be like using ratio based on like how much it was." (29:33)
- Josh Xi: "For time series forecasting, the one of the most common technique is called back testing." (38:53)
- Josh Xi: "We know we can't model them correctly and so the best we can do is learning from our mistakes." (47:00)

## Tools & references mentioned
- Lyft
- Uber
- ARMA
- TimeGPT
- Microsoft
- long short-term memory
- convolutional neural network

## Who should watch
- You run forecasts at a high spatial and temporal resolution, and model startup time or retraining cost limits what you can deploy.
- You are deciding whether a DNN is worth supporting when a smaller autoregressive model may adapt faster and be easier to adjust.
- Your forecasts feed pricing, incentives, or other operational decisions and you need practical methods for handling events that the training data does not explain.

## Editor's note

From the pack [Feedback loops and retraining](https://mlopstalks.com/packs/feedback-loops-and-retraining):

Rizzoli returns production examples to versioned datasets, and Munro connects annotation speed to retraining cadence. ZenML records pipeline steps, inputs, outputs and code versions, so a resulting model can be traced to its producing run. That history helps investigate a dataset update while the team defines selection rules, reviews labels and decides whether the new model is useful.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- ["Real-Time" ML: Features and Inference](https://mlopstalks.com/talks/real-time-ml-features-and-inference) (Sasha Ovsankin & Rupesh Gupta, LinkedIn, 51:55)
- [Real-time Feature Generation at Lyft](https://mlopstalks.com/talks/real-time-feature-generation-at-lyft) (Rakesh Kumar, Lyft, 58:05)
- [Graph Neural Networks Just Solved Enterprise AI?](https://mlopstalks.com/talks/graph-neural-networks-just-solved-enterprise-ai) (Jure Leskovec, Stanford University and Kumo.AI, 49:01)
- [Real-time Machine Learning](https://mlopstalks.com/talks/real-time-machine-learning) (Chip Huyen, Claypot AI, 58:24)
- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)