# Data Engineering: The Missing Piece of Your Data Science Puzzle

 | MLOps Community | 12:10

Source: https://www.youtube.com/watch?v=6ET-NwDnq7E
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/data-engineering-the-missing-piece-of-your-data-science-puzzle
Published: 2024-10-09
Tags: data-pipelines, data-quality, feature-engineering, feature-stores

## TL;DR
- Chime uses separate infrastructure for batch, near-real-time, and real-time features because their latency and cost requirements differ.
- A shared feature library lets data scientists define feature logic once and reuse it across training and inference, reducing training-serving skew.
- Point-in-time feature generation must account for source and processing delays so training does not use data that was unavailable at prediction time.

## Summary
Aishwarya explains how Chime supports fraud models that need near-real-time inference. She separates features into batch, near-real-time, and real-time categories, then maps each category to infrastructure with different latency and cost characteristics. Streaming pipelines write near-real-time features to an online store, while batch features can be computed on demand or read from offline storage. Data scientists use a feature library to define SQL logic, metadata, time windows, schedules, and data quality checks. CI jobs turn those definitions into ingestion and monitoring configuration, and the same logic is reused during model development and serving. The talk also explains why point-in-time correctness is difficult. Source availability lag and processing lag can make a feature appear available when it was not available at the prediction timestamp. Chime accounts for this by parameterizing feature definitions so training and inference use the same delay assumptions.

## Key ideas
### Fraud models need low-latency features because decisions happen during user activity
[00:50](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=50s)
Chime uses machine learning for payments, mobile check deposits, and unauthorized account logins. Some of these use cases need near-real-time inference because the system must respond before an attacker can access a user's funds. As the number of users and transactions grows, storing more data and serving predictions quickly become harder. The model infrastructure therefore has to support fast feature access as well as the volume of data behind the features.

### Batch, near-real-time, and real-time features have different operating requirements
[01:53](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=113s)
Aishwarya divides model inputs into three groups. Batch features use historical windows and are updated on a regular schedule, such as total withdrawals over the past week. Near-real-time features also use a time window, but streaming ingestion keeps them fresher, such as a user's transactions from the past five minutes. Real-time features are available immediately from the request or another service when the model is invoked. These categories determine which storage and processing paths are appropriate.

### An online store and offline computation path are both needed
[03:05](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=185s)
Real-time models need an online feature store that can return values with low latency. Offline computation remains useful for batch models and training pipelines, but it is not enough for inference that must complete in milliseconds. Chime also uses a feature library to keep feature definitions in one place. Reusing those definitions reduces ad hoc serving logic, feature mismatch, and training-serving skew between development and production.

### Chime matches infrastructure to latency and cost
[04:35](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=275s)
Chime uses Kinesis streams and Glue jobs with Spark SQL transformations for near-real-time features. The resulting data is written to an online store so models can read it efficiently. Batch features can be computed on demand or read from an offline store. Aishwarya points out that the streaming path costs more, so a model that produces predictions once a day does not need to use near-real-time ingestion unless latency affects its performance.

### The feature library carries logic and operational settings into generated jobs
[05:59](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=359s)
Data scientists define feature metadata, SQL query logic, time windows, and update cadence in the feature library. CI jobs use those definitions to generate Glue job configuration and data quality checks. Users can also configure alerts and specify checks in their SQL logic. The same feature definitions are used during model development and in the ingestion path, which keeps the computation consistent across environments.

### Data quality checks belong at several points in the ingestion flow
[06:52](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=412s)
Chime can check source data before transformations begin, which can prevent spending resources on a pipeline that will fail because of missing or corrupted input. It can check the transformed data for records lost or corrupted during processing. It can also check the data after it reaches the sink, before downstream consumers such as real-time models use it. Examples include null rates, duplicates, and freshness.

### Point-in-time features must exclude data that was not available yet
[07:45](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=465s)
Point-in-time feature generation joins records using only data available at or before the prediction timestamp. Including later records creates leakage from the future and can produce inaccurate training performance or a mismatch between training and deployment. The problem includes both source availability delay and processing delay, so the feature system has to model when data became available rather than relying only on the event's apparent timestamp.

### The same delay assumption must apply during training and inference
[08:59](https://www.youtube.com/watch?v=6ET-NwDnq7E&t=539s)
Aishwarya gives a five-minute lag example. If a feature is delayed by five minutes at inference time, it can use only data available up to five minutes before the prediction. Training must apply the same cutoff relative to the time the data became available for processing. Parameterizing this delay in the feature definition helps keep the training sample and production feature value aligned.

## Notable quotes
- "I want fast feature lookup to serve my real-time models." (02:46)
- "A feature Library will help us consolidate subject matter expertise and define features in one place." (03:23)
- "If you include data after then that constitutes data leakage from future data values." (08:00)
- "You need to build out monitoring capabilities for your models." (09:50)

## Tools & references mentioned
- Chime
- Kinesis
- Glue
- Spark SQL
- PagerDuty
- Bedrock
- DynamoDB

## Who should watch
- You are building fraud or risk models that need features at serving time and want to understand the infrastructure behind them.
- Your team has separate training and production feature logic and needs a way to reduce training-serving skew.
- You are designing feature pipelines and need practical guidance on data quality checks, latency tradeoffs, or point-in-time correctness.

## Editor's note

Aishwarya explains that the same feature definitions must keep training and production calculations consistent, including their assumptions about data delay. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a feature or model can be traced to the data and code that produced it. This gives teams a record when they need to check how training and inference aligned.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)
- [AI Innovations: The Power of Feature Platforms](https://mlopstalks.com/talks/ai-innovations-the-power-of-feature-platforms) (Mahesh Murag, Tecton & Jose Navarro, Cleo & Nikhil Garg, Fennel, 1:05:28)
- [Real-time features, AI search, Agentic similarities](https://mlopstalks.com/talks/real-time-features-ai-search-agentic-similarities) (Varant Zanoyan & Nikhil Simha Raprolu, Zipline AI, 29:28)
- [Mid-Scale Production Feature Engineering](https://mlopstalks.com/talks/mid-scale-production-feature-engineering) (Dr. Venkata Pingali, Scribble Data, 1:01:35)