# Scaling Real-time Machine Learning at Chime

Peeyush Agarwal, Chime | MLOps Lightning Sessions | Episode 1 | 24:22

Source: https://www.youtube.com/watch?v=6x7VQUOGRrA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime
Published: 2022-05-25
Tags: feature-engineering, feature-stores, model-serving, monitoring, orchestration

## TL;DR
- Chime separates machine learning workloads into batch, nearline, and real-time inference according to latency and the value of information available at decision time.
- Its first real-time model used about 200 features to detect fraudulent Pay Friends transactions, with feature lookups taking tens of milliseconds and the full decision completing in hundreds of milliseconds.
- Chime scaled deployment by making monitoring, model serving, orchestration, and infrastructure configuration reusable and available through platform code.

## Summary
Peeyush Agarwal describes how Chime built infrastructure for real-time machine learning, using fraud detection in its Pay Friends product as the first detailed example. Chime's teams distinguish batch, nearline, and real-time workloads based on latency requirements and whether the needed information can be precomputed. The fraud model combines transaction details with member history and about 200 features, then returns a decision within hundreds of milliseconds. Chime built a feature store with feature registration, computation, offline and online storage, ingestion, and serving components. It also records predictions, feature values, system data, and timing information for monitoring and auditability. To support more models, the ML platform team packaged shared serving and production code, provided orchestration templates, and managed deployment through code. Agarwal's advice is practical: invest in real-time features, instrument and load-test the full path, make monitoring part of the design, keep platform components replaceable, and define infrastructure as code.

## Key ideas
### Chime chooses the inference pattern according to latency and information needs
[04:02](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=242s)
Agarwal divides Chime's workloads into batch, real-time, and nearline inference. Batch pipelines work from historical data when minutes or hours of latency are acceptable, such as predicting member lifetime value or identifying likely churn. Real-time inference is needed when a person or system is waiting on the result, as with approving a card transaction or deciding whether a login needs two-factor authentication. The decision also depends on information that only exists at the moment, such as merchant, transaction amount, and location. Nearline inference sits between the two. A member viewing offers might trigger a follow-up email without requiring that email to be sent immediately. Agarwal says these cases can often be reframed as batch or real-time work.

### The first real-time model blocked fraudulent Pay Friends transfers
[08:18](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=498s)
Chime's first real-time model focused on Pay Friends, where members send money to one another instantly. Agarwal says bad actors used the feature for social engineering and account takeover, so Chime applied machine learning to identify and block fraudulent transactions. An engineering service sends the model details such as sender and recipient IDs, transfer amount, and IP addresses. The model adds historical features, including prior interactions, direct deposits, and previous Pay transactions for both people. It uses about 200 features and must make its decision in hundreds of milliseconds. Feature retrieval takes tens of milliseconds, after which the service tells the calling system whether the transaction is risky.

### The feature store separates registration, computation, storage, ingestion, and serving
[10:59](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=659s)
Chime built its feature store to remove a bottleneck in real-time feature lookup. It also reduced differences between training and inference pipelines and gave data scientists a shared way to define features. The platform has five parts. A feature registry holds feature-group schemas and definitions, such as transaction or direct-deposit history. A computation framework aggregates data from streams and the warehouse. Offline storage lives in S3, while online storage supports low-latency serving. An ingestion component handles asynchronous incoming data. A serving system reads online features and makes them available to models. This structure lets the same feature definitions support multiple model use cases.

### Instrumentation made latency problems visible and helped reduce them
[13:32](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=812s)
The inference service records predictions, feature values, system information, and timing data asynchronously for monitoring and logging. Agarwal says instrumentation showed how much time each step took, which helped the team optimize the complete inference path. Heartbeats and warm caches reduced P95 latency from three-digit milliseconds to two-digit milliseconds for feature-store calls. The team also used the Hummingbird framework with a tree-based model to reduce inference latency. This timing breakdown mattered because the service had to fetch features, run the model, and respond quickly enough for a transfer decision. Tracking the individual stages gave the team a way to find and address the slowest parts.

### Chime monitors system, service, platform, feature, and model behaviour
[14:44](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=884s)
Agarwal describes four monitoring categories. System metrics cover automatically generated data from the Kubernetes environment, including pod and container CPU and memory use. Operational metrics cover service latency, uptime, and service-level objectives. ML platform metrics include request throughput and traffic seasonality. Model-specific metrics include missing-feature frequency and the distribution of model probability scores. A change in score distribution can indicate that something is wrong with the model. Breaking missing values down by feature can show that one feature family is not being ingested correctly. Chime also tracks request counts, failed inferences, total invocation time, and the time spent retrieving features versus running the model.

### Monitoring and serving code became part of the platform instead of repeated project work
[16:16](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=976s)
As Chime added models, the team wanted monitoring and alerting to exist from the first deployment. It implemented the observability stack as code and used anomaly detection for monitoring metrics, with standard dashboards and alerts available to data scientists. The team also created a package shared by models in production. It handles model-compatible images, HTTP requests, logging, monitoring, error handling, and connections to the container registry and deployment system. Data scientists implement defined interfaces instead of having the platform team understand every model's code. The shared package reduces repeated work and sends standard errors and metrics to Datadog. This makes production behaviour more consistent across model use cases.

### Templates and continuous delivery let data scientists move models through environments
[20:18](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=1218s)
Chime's orchestration framework provides templates for training, batch inference, and real-time serving. When model code is merged into the main branch, continuous delivery deploys the related infrastructure to development and production environments. Pipelines can also run on cron schedules. Data scientists keep their model development in the model repository, define the orchestration pipelines when the work is ready, and test the end-to-end implementation in development before promotion. Agarwal describes this as Chime's approach to infrastructure as code. The framework is intended to handle different types of workloads while allowing the platform to manage the operational path around them.

### Reusable interfaces and code-managed infrastructure support more models
[21:45](https://www.youtube.com/watch?v=6x7VQUOGRrA&t=1305s)
Agarwal's lessons focus on making real-time ML repeatable. A real-time feature store provides the information needed for inference and helps teams move toward deployment faster. Instrumentation and load testing reveal where time is spent, how much traffic the system can handle, and how autoscaling behaves. Monitoring and auditability should be included in the design in a model-agnostic way, rather than rebuilt for every model. Pluggable components and open interfaces let the team evolve the platform incrementally. The orchestration layer can support models implemented in Python, C++, Java, or other environments when the interfaces remain clear. At larger scale, Chime manages monitors, alerts, and other infrastructure as code.

## Notable quotes
- "In the moment information is vital or necessary to make that decision." (05:33)
- "The model uses about 200 features to reach a decision and this decision needs to be computed in hundreds of millisecond latency." (09:49)
- "We wanted to have it built as part of the platform so that each new real-time model that we deploy we get the monitoring and alerting available out of the box." (16:20)
- "Having clear interfaces between different components that are part of ML platform can help keep them pluggable, be able to build on each of them incrementally and evolve the infrastructure over time." (23:14)

## Tools & references mentioned
- Chime
- Pay Friends
- Kubernetes
- AWS Kinesis
- S3
- Hummingbird
- Datadog
- Arize
- ECR
- Flask

## Who should watch
- You are building fraud, authorization, login, or other model services where a request must receive a decision in milliseconds.
- Your team is repeating model packaging, serving, monitoring, and deployment work for every production model.
- You need a practical design for connecting a feature store, inference service, monitoring system, and orchestration layer.

## Editor's note

Peeyush Agarwal describes how Chime turned monitoring, serving code, and orchestration into reusable platform components so data scientists could deploy models through code-managed environments. ZenML lets teams write workflows as Python steps and choose the infrastructure stack by configuration, so the same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud service.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Data Engineering: The Missing Piece of Your Data Science Puzzle](https://mlopstalks.com/talks/data-engineering-the-missing-piece-of-your-data-science-puzzle) (, 12:10)
- ["Real-Time" ML: Features and Inference](https://mlopstalks.com/talks/real-time-ml-features-and-inference) (Sasha Ovsankin & Rupesh Gupta, LinkedIn, 51:55)
- [Real-time features, AI search, Agentic similarities](https://mlopstalks.com/talks/real-time-features-ai-search-agentic-similarities) (Varant Zanoyan & Nikhil Simha Raprolu, Zipline AI, 29:28)
- [Real-time Machine Learning](https://mlopstalks.com/talks/real-time-machine-learning) (Chip Huyen, Claypot AI, 58:24)
- [AI Innovations: The Power of Feature Platforms](https://mlopstalks.com/talks/ai-innovations-the-power-of-feature-platforms) (Mahesh Murag, Tecton & Jose Navarro, Cleo & Nikhil Garg, Fennel, 1:05:28)