# Mid-Scale Production Feature Engineering

Dr. Venkata Pingali, Scribble Data | MLOps Meetup | Episode 6 | 1:01:35
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=1CcYuVVwOGg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mid-scale-production-feature-engineering
Published: 2020-04-16
Tags: data-quality, debugging, feature-engineering, orchestration

## TL;DR
- Production data science needs trusted datasets because model outputs can affect customers, revenue, and legal risk.
- Feature engineering platforms need metadata, lineage, reproducibility, checkpointing, and data-quality checks to support debugging and accountability.
- The right feature engineering architecture depends on data shape, team size, model count, and how closely features are tied to modeling.

## Summary
Dr. Venkata Pingali explains why feature engineering needs the same discipline applied to software systems. His concern began with marketing projects where teams optimized for speed without checking whether the data and resulting claims were correct. In production, a model can affect inventory, pricing, customer decisions, and revenue, so teams need to know what data was used, when it was computed, and how it changed. Scribble Data built these capabilities in response to customer needs, including metadata, lineage, reproducibility, checkpointing, and quality monitoring. Pingali argues that feature engineering is not one linear step before model training. It touches resource planning, validation, workflow management, and parts of serving. The architecture also changes with data shape and organizational scale. Once an organization has around eight to ten models or use cases, he expects a shared feature platform to become necessary. He also describes the systems knowledge an ML engineer needs to work with data scientists and data engineers.

## Key ideas
### Production data science needs trust and accountability
[01:21](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=81s)
Pingali traces his focus on production systems to data projects where speed mattered more than correctness. Teams shared model-based claims with stakeholders without asking whether the claims were accurate. That becomes unacceptable when a CFO uses data to reject customers or when a system changes real-world decisions. He says data science is moving from a laboratory activity into a regular organizational function that finance, operations, and legal teams must be able to depend on. Explainability therefore reaches beyond the model. It also requires answers about the underlying data, the transformations that produced it, and the people responsible for each step.

### Lineage and reproducibility help teams debug their own systems
[17:45](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=1065s)
Pingali gives an inventory forecasting example where unusual model behavior was eventually traced to Java application logic that handled products differently by geography. The team first inspected modeling code and a Jupyter notebook, then followed the dataset back through the data pipeline. Scribble Data's metadata and lineage records showed which sources produced the dataset, allowing the team to trace the problem to the Java source and fix it. He says this kind of debugging need will drive reproducibility and explainability even without regulatory pressure. Teams cannot manage risk if they do not know what their data system did, when it did it, or why.

### Checkpointing turns data-service promises into operational capabilities
[21:25](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=1285s)
Scribble Data added reproducibility and checkpointing because customers wanted to treat the platform as a data service. If a bug enters a transformation, the team must identify when it began and how far its effects propagated. That requires knowing the state of a pipeline at a particular checkpoint and being able to backfill from the right point. Pingali connects these capabilities with metadata standardization and data-quality monitoring. The platform's promise of confidence in data required more than generating a final dataset. It required records of what happened throughout preparation, including the raw inputs and intermediate states.

### Data quality is part of model trust
[23:47](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=1427s)
Pingali says metadata, lineage, and data quality are connected. A team cannot claim confidence in a prepared dataset without confidence in the raw data from which it was built. Scribble Data initially encountered monitoring as an unexpected use case because customers wanted to check existing datasets, not only create new ones. He argues that data quality is needed for trust, and trust supports explainability and auditability. The model code is becoming more predictable through standard libraries, while the training data remains a major source of unpredictable behavior. For that reason, he expects trust in data to receive more attention as models affect business decisions.

### The surrounding infrastructure often takes more work than model building
[30:35](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=1835s)
Pingali discusses a Google paper from 2015 that showed machine learning systems require substantial infrastructure around the model. Scribble Data began with a simple view of feature engineering as a linear process that feeds data into a model. Deployments showed a wider set of needs, including data monitoring, quality checks, resource management, efficiency work, and parts of serving infrastructure. He describes four goals for the overall system: increase the speed of adding use cases, give stakeholders confidence, support continuous change and debugging, and scale across more customers and products. The exact distribution of effort differs by company size and system complexity.

### Feature engineering has several valid design points
[37:07](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=2227s)
Pingali rejects the assumption that every company should copy the architecture of Uber or Google. He says there are at least six or seven possible design points even within feature engineering. A system for data with ten columns and ten billion rows can look different from one for five million rows and two hundred columns. The relationship between feature preparation and deep learning also changes the architecture. As organizations grow, feature engineering often starts as one integrated activity and then develops clearer boundaries between data collection, preparation, and modeling. Pingali says a standalone shared platform is generally needed after an organization reaches about eight or ten models or use cases, because inconsistent feature definitions create confusion and slow development.

### Feature pipelines need careful planning because errors propagate
[42:03](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=2523s)
Feature preparation pipelines can form a chain in which one pipeline feeds another. If a leaf-level variable is defined incorrectly, the error can affect data science work over the following months. Pingali says productionization exposes hidden assumptions and bottlenecks that are easy to miss during modeling. He recommends thinking through these dependencies before the modeling phase is complete. The platform must give teams control over definitions, sources, quality checks, and downstream effects. He also notes that teams with roughly twenty or thirty people may struggle to assemble all these pieces, since they have limited resources and need to decide which processes and interfaces to standardize.

### Multi-step feature chains avoid rebuilding overlapping datasets
[44:29](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=2669s)
In response to a question about overlapping feature sets, Pingali describes a multi-step feature chain. Users can take a prepared feature set when it fits their use case, or move one level deeper when they need additional information. In Scribble Data deployments, partially computed information is available at several levels before the final output reaches the user. A use case may start at the fourth level, return to the third, or go back to raw data when another attribute must be extracted. He does not favor generating every possible feature combination. Data scientists usually know which features they want, and the computational cost can be too high for large feature matrices.

### Orchestration matters less than the structure of the development process
[49:02](https://www.youtube.com/watch?v=1CcYuVVwOGg&t=2942s)
Scribble Data integrates with Airflow and uses Prefect for cross-process workflows, while maintaining an internal workflow system for operations at data-frame granularity inside a process. Pingali says the orchestration tool itself is less important than how the team structures development, modules, and operating practices. He has mixed feelings about Kubeflow and Kubernetes for customers that cannot dedicate development and operations teams to running them. In those cases, Scribble Data has used simpler deployments. His answer places orchestration inside a broader system-design problem rather than treating it as the central solution.

## Notable quotes
- Dr. Venkata Pingali: "You cannot build defensible models until you have a robust underlying system." (27:41)
- Dr. Venkata Pingali: "Reproducibility and explainability will be driven not so much by the asks of third-party regulatory authorities but just because you have the need to understand and debug your own data science steps that you have built." (20:19)
- Dr. Venkata Pingali: "By the time you crossed about eight or ten models, ten different use cases, you'll start needing a standalone platform that is shared between all of these ten models." (39:32)
- Dr. Venkata Pingali: "The problem is not so much on the computer building blocks. Even if it is PySpark, you need to know what is going into Python, what is coming out of PySpark, how did you do it, and are you doing a quality check before it goes in and after it comes out." (27:24)

## Tools & references mentioned
- Scribble Data
- MLflow
- Google
- Airflow
- Prefect
- Kubeflow
- Kubernetes
- PySpark
- Jupyter notebook

## Who should watch
- You are building feature pipelines that need to be reproducible, auditable, or explainable when a model behaves unexpectedly.
- Your team is deciding whether feature engineering should remain inside individual projects or become a shared platform.
- You need to choose an architecture for a mid-sized data team and want guidance grounded in data shape, workflow complexity, and operational constraints.

## Editor's note

Dr. Venkata Pingali says teams cannot manage production risk when they cannot trace a dataset through its sources, transformations, and checkpoints. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it. This gives feature pipelines a reproducible record for debugging.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [A Conversation Around Feature Stores](https://mlopstalks.com/talks/a-conversation-around-feature-stores) (Venkata Pingali, Scribble Data, 1:03:18)
- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [The Future of Feature Stores and Platforms](https://mlopstalks.com/talks/the-future-of-feature-stores-and-platforms) (Mike Del Balso, Tecton & Josh Wills, Angel Investor, 1:11:15)